#1 — A migration's tale

A discussion with Nicolas Kowalski, from Criteo. Episode transcript.

Listen to the episode

Trailer

Nicolas: [00:00:00] We had actually, without noticing lost 95% of our metrics already, silently without realizing it. But those things happen. It would have been very easy for us to start pointing fingers. But you need to quickly realize that this is not going to solve anything and to have everyone on board saying, “okay, let’s behave like adults”. The goal here was to make sure that we would not impact our customers, internal customers, but still customers. And I think that keeping that in mind really helps you prioritize what you need to do.

Intro

François: [00:00:45] Hello and welcome to Post Mortem. My name is François. I’ll be your host for the show. In Post Mortem, we have software engineers coming to the mic and sharing with us their experience of an incident. That can be a bug, a cyber attack, or a project that did not go as expected. We’ll discuss what they did to mitigate the impact of the incident.

What worked, what did not work and ultimately the lessons learnt from those incidents.

So today in the show I have the pleasure to be with Nicolas from Criteo, hello Nicolas.

Nicolas: [00:01:21] Hello. And thank you for having me on the podcast, François.

François: [00:01:24] You’re very welcome. So could you tell me what you’d like to talk to us about today?

Nicolas: [00:01:29] Sure. We had about six months ago a problem that was not exactly a problem in prod, but more, a problem related to project management.

And we had actually a very, very big migration in Criteo that was planned to be executed for several years. And that was reaching the end. And there was an unexpected dependency on our team. We suddenly had to deal with it and it turned out to be much bigger than what we expected at the beginning.

So we had to deal with that and it was complicated more in terms of project management than in terms of technicalities.

François: [00:02:10] Okay. Thanks for the teaser and the intro, before going into the details, could you give us a quick intro / overview of what Criteo is and in which team you work in details?

Nicolas: [00:02:21] Sure. Criteo is an advertising company.

So we are trying to target products, to people which sounds like a curse word for a lot of people. The thing to remember is that this actually powers the entire internet in terms of having free websites. Our goal is really to provide advertising that provides value to the users while remaining completely privacy secure, and to provide alternatives to the likes of Google on Facebook that are very big and provide a lot of advertisement.

And we are trying to provide a more diverse environment. And specifically I work In the prediction team in the Criteo AI lab. In our team, we are providing services for the whole of Criteo to learn machine learning models and to serve them at scale. And when I’m saying at scale, it is really at the big scale because at Criteo we are seeing about two three or 4 million events per seconds that needs to be dealt with, analyzed. Do we want to actually provide an ad in that context, to that user? Which ad? There’s a lot of computation that needs to happen in about 10 milliseconds.

François: [00:03:50] So when you say event, what is an event for you typically?

Nicolas: [00:03:54] It is a user for which we have a hashed identifier.

So we don’t know exactly what the person is, but we know that we have already seen that person on the internet. We know that that person has already visited that and this website. And so we have an idea given that that may be, that person will be interested in video games or in a holiday in the South of France, for example.

So they come on the website. We know that they are on this given website, for example, a newspaper. We know that we could bid for a specific spot on the page. So we know where we will be placing that image. And we know who that person is in terms of the history that it has with us. Given those, we need to define if you want to bid on an auction for that spot.

François: [00:04:53] So yeah, I think one thing to keep in mind is that Criteo operate and maintain their own data centers worldwide, right? Compared to a lot of companies.

Nicolas: [00:05:02] Yes. We actually operate on bare metal, which means that we have data centers that we are renting. And we are operating from scratch and we are having our own apps directly running on them.

And we do have quite a bit of servers actually. We have thousands of servers all over the world. So we are operating at the scale that only very big GAFA companies are operating in terms of amount of compute and number of events that are seen on the internet.

ACT I — The context of the migration

François: [00:05:54] Now that we’ve set up the context of this discussion. Let’s zoom in on this particular migration that you teased us at the beginning of the episode. So what was this migration about?

Nicolas: [00:06:06] So the migration was about migrating a lot of our systems that are running on the servers that we rent from Windows to Linux, because for a variety of reasons, we wanted to actually make that move for a few years now, mainly because we have reached a scale where the amount of money spent on Windows licenses was quite impressive, given the amount of servers running, and also because simply it was possible now, it was not possible in the past to make that migration due to, for example, the .NET Core project.

François: [00:06:47] In a nutshell, what does .NET Core project change in this context for you to deploy to Linux?

Nicolas: [00:06:52] It’s actually a project that allows .NET projects to actually be run on Linux and not on Windows.

François: [00:07:00] Ok, because .NET is the Microsoft development framework typically. And you use this on Windows machines, right?

Nicolas: [00:07:07] Exactly. And .NET Core is an equivalent ported on Linux. So, at the beginning of the year, the migration of a lot of apps on, especially the biggest one, Arbitrage, to Linux was reaching the end.

François: [00:07:23] What is Arbitrage?

Nicolas: [00:07:25] Yes, Arbitrage is an app that is very important to us, which is actually the one that is seeing the event in a sense, and trying to test different possibilities of ads that we could show at this spot. And trying to arbitrate between them to define the best one and if it is good enough to actually buy the spot.

So it is really at the core of the Criteo systems. It’s actually one of our main components and it is an app or it is a team that depends on my team, the production team. They are sending different requests. And we as a team that is going to get all those requests, deal with them, with the models that we have learned and compute a value for each of them and give that back to Arbitrage.

So, as I was saying, Arbitrage has been migrated over the course of 2019. And at the beginning of 2020, they had plan to finish at the end of the first quarter, their migration. They had already migrated 95% of the servers to Linux. So only 5% remained on Windows and they encountered an issue, where they actually needed to migrate the logging part of our systems.

That relied before on a system called Graphite and to move it to another system called Prometheus.

François: [00:08:59] In a nutshell, can you summarize what those two systems, graphite and Prometheus are about?

Nicolas: [00:09:04] Both of them provide a way to record metrics and to send them to a frontend called Grafana where we have boards that can showcase the metrics that we are monitoring, so it’s pretty much a backend that is recording.

The thing is the graphite back end was not able to scale on Linux, for very technical reasons there was an impossibility to actually use that backend for Arbitrage under Linux. So we needed to actually migrate to this new backend Prometheus. One thing you may now realize is that since 95% of the system were already moved to Linux at this point, we had actually, without noticing, lost 95% off our metrics already, silently without recognizing it. Those metrics were mainly used by teams that are dependent on us, which are product teams that do AB tests and want to monitor them.

François: [00:10:08] Can you give us an example of a concrete metric that you record?

Nicolas: [00:10:12] Yes, of course, an example of a metric will be quite simply the Click-through rate (CTR).

So the probability that the user will click on a given webpage. So if, for example, you say, okay, now, I realized that we have this new data about the user, or we want to use this data with a heavier weight, for example, in a new model. You want to know how it’s going to impact your metrics, how it’s going to impact the probability that the user is going to click and then to buy things. So you have several metrics. What is the probability that the user is going to click on a page? Then what is the probability that given that it has clicked, this user is going to actually buy something. What is going to be on average, the quantity of stuff that is going to be bought. So you could have different ways of measuring the success of your campaign.

No matter what, that’s not exactly our problem. This is the product teams that are dependent on us that are going to define their own objectives, but they need to monitor it.

François: [00:11:15] So critical features I guess, the click-through rate is really something you need and you want to have to pilot your activity, right?

Nicolas: [00:11:21] Yes, it is really the thing that we rely on as the main metric. And so due to the fact that we were actually losing those metrics. Those teams did not realize necessarily, and we did not realize that we are losing those metrics because the main things that were used here during those AB tests were comparisons between two different metrics.

What this did mean was that the metrics were much more noisy since we only relied on 5% of all the data. And since in some cases you could define an AB test where the B portion of your test, the percent on which you would test the new feature […] (interrupted).

François: [00:12:02] Sorry to cut you, let’s take a quick moment to define what an AB test is ?

Nicolas: [00:12:06] Of course, an AB test is when you define two populations among your users.

A population A and a population B, and you’re going to test new features on the new population, on the population B. So the population A is, continuing running my system as previously. And the second one is going to be “What if I introduce this change?” and you’re going to monitor both concurrently and see, what is the impact of your change? So for example, someone that has a website could do an AB test where you change the banner at the top, and you can see if people stay or not. That’s a very simple example.

The main issue is that sometimes an AB test, depending on how certain you are, how confident you are in your new feature, you could run an AB test up to 50 / 50% for example, but in some cases you are introducing a change that could be really impactful and you are really not sure.

So what you’re going to do is to use 1% in your B population.

The thing is, since we only rely at this point on 5% of the data already, you can see that the two compounds. So, potentially, some AB tests that were running on a very low population already were relying on a very small amount of data compared to what we are used to, which meant that without us noticing, suddenly, some AB tests, were much more noisy in terms of the data that we were getting.

So that was a main issue at this point, but the thing is that the migration to 100% of Linux servers was expected to come two months after that. Two months later, the migration was supposed to be finished. And at that point, all AB tests would have become blind. So there was really, a very time sensitive impact.

François: [00:14:02] What do you tell yourself when you realize that this happened and you have two months to fix this critical deployment?

Nicolas: [00:14:09] At the beginning, we thought that it would be a matter of a couple of days to change his backend. So we were like, “okay, that was a miss, but no big deal. We’re going to deal with this unexpected thing and, we’re going to just get back on our feet and that’s not going to be the end of the world. For a month, our AB tests have been a bit more noisy in some cases. But stuff happens” and we were pretty confident.

The thing is, quite quickly, it became apparent that actually changing this backend in our code would require pretty significant stuff because a lot of other teams had to do similar changes, but did not have the metrics embedded in such a way. We were a team that was running a critical part of the infrastructure of Criteo too.

And so actually the changes were impacting us much more profoundly and in a way that neither the Arbitrage team, nor us did understand at the beginning.

So suddenly, this change that was supposed to take one or two days, we realized that it would probably take closer to two solid weeks.

François: [00:15:24] And how do we tell these to you or to your team or to management that “okay. I won’t be able to deliver those other features in time because we’ve got to deal with these deployments”?.

Nicolas: [00:15:34] Yeah, that’s a good point. And I think that it’s important at this point to recognize that the culture that you have in your company is going to really impact how you deal with that. We are actually very lucky in Criteo to have a culture that is relying mostly on people on the ground to define the priorities of projects.

Which means that the hierarchy is not that involved compared to a lot of other companies in defining what is important and what isn’t. That also means that in that case, we add a lot of room to prioritize this over other stuff. And to explain to our hierarchy that this would mean that potentially some of the things that we had planned for the quarter would not be delivered.

Not only did they accept this, but on top of that, it is actually encouraged in our culture to go out of your way to help another team, which means that we knew that we would be encouraged to actually prioritize this change over continuing our business as usual. Whereas in a lot of other companies, you could have a situation where you will say, “I’m sorry, but this is not planned. And due to this, it’s going to have to be planned for next quarter.” That would have meant that’s the whole migration, impacting a very large part of the Criteo infrastructure, would have been delayed by a full quarter, just because of this.

François: [00:17:05] I see. So it was pretty easy and natural within your company and the given culture to get this event acknowledged, accepted, as soon as you realized it and to deploy the appropriate resources internally to address this issue.

Nicolas: [00:17:21] Yes. I had no issue directly to have my direct manager and the people above green lights the fact that I will start spending all my time working on these things that was not planned at all.

ACT II — The resolution

François: [00:17:50] How did you actually solve this?

Nicolas: [00:17:54] So technically, it was a lot of small steps that needed to change in our code. It’s the most legacy of our code. So it’s not the most well-designed in terms of architecture, which meant that we were traversing a lot of, part of the code where we needed to actually route those new metrics.

But the main issues that we encountered was that, in the new backend, the C Sharp (C#) library did not contain all the features that we needed. In particular, we needed to handle floats and not just integers, but in the C# code that was responsible for those metrics, we only had histograms for integers.

So a simple fix will be to say that you use an integer by which you divide your integers. And that’s going to give you a float, but obviously that’s not a reliable solution in the long term, because you need to know how much you need to divide with. You’re going to have some times underflow and overflows, which are going to mess with your system without you necessarily realizing it. So it was quite dangerous.

The proper solution and the one that we knew we needed to do was to actually go into the metrics code and introduce those new classes, to handle properly double and floats histograms.

And that’s where things got a bit messier since we realized at this point, by talking to the observability team that they had developed this code quite a long time ago, three or four years ago, at least, which is a lot of time in a dynamic team in the tech environment nowadays.

And actually, all the people that were knowledgeable in C# had left the company since then. So they had at this point, no one that was comfortable developing in the C# code they owned. But did not have to act upon or modify for three years now.

François: [00:19:58] How does it work? Do you say, “okay, I’m going to do it myself”, or do you get a coffee with someone from the other team saying, “okay, can you include this in your roadmap for the next sprint?” How do you synchronize to address this ?

Nicolas: [00:20:11] Well, you start talking directly with the persons in those other teams. So that’s, first of all, not necessarily the case in all companies, in some companies I was in, in the past, you would have had to actually involve the managers of each team to talk to each other, have some things that go up and then down.

Fortunately in Criteo, we are encouraged to talk directly to each other. So we were able to openly talk about it. And the issue was due to them not having the knowledge. It would not have been a question of just a sprint. Similarly, we would have had to schedule for a whole quarter afterwards because they would have had to need to ramp up quite a bit.

So it was a possibility, but it would have definitely meant also delaying the migration that was already ongoing for a full year and that was scheduled to finish two months later. The other possibility, obviously, was for us to take care of that. It’s not something that we were exactly happy to do, obviously, you’re already being delayed for a few weeks in your schedule, you have your own goals. And you know that this is going to delay much further the goals of your team. But discussing with all the people involved, we all felt that this was definitely a top priority for Criteo and that we needed to find a solution one way or another.

François: [00:21:39] So you decided to take the ownership of implementing this feature?

Nicolas: [00:21:43] Yes. At this point we had an engineering program manager of the team observability that joined us. We created a Slack channel dedicated to solving this solution embedding people, both from the Arbitrage team, from our team, and from the observability team.

And we worked on a daily basis to move the topic forward. Having someone from the observability team provide knowledge about how the metrics were supposed to be used. So as to provide us with guidance on what needed to be implemented and what we needed to take care of

François: [00:22:23] It can often be a challenge to cooperate with different teams at the limit of each of each team’s scope because you don’t really know whose team ownership is it.

So do you have any advice based on this experience, a few tips that you can use when you have to synchronize with your team, the Arbitrage team, and the observability team?

What was the main takeaway in terms of coordination?

Nicolas: [00:22:51] I think the first thing is to have everyone be of goodwill. If you have someone that is dragging their feet, it would have been very easy for us to start pointing fingers saying either to the arbitrage team. Why did you miss your dependencies on us, say to the observability team, “how come guys, that you are not able to actually make modifications in a code that is your ownership?”

And you need to quickly realize that this is not going to solve anything and to have everyone on board saying, “okay, let’s behave like adults” and all think how can we do those modifications as soon as possible, have everything solved. And I think that this is both a question of the culture of the company, as I said, but also of hiring people that are not going to be capricious and that can see the big picture.

And to also encourage it on an ongoing basis to actually see the problems of your clients, of the people that depend on you, to think about them first, even if it is for a backend team, where your clients are actually internal people in the company. And to be confident in the fact that, when you’re going to have to show the work that you did for the year, for example, Not only are you not going to be punished for taking those initiatives, but you’re actually going to be encouraged and congratulated for actually having taken initiatives to solve problems beyond the scope just of your team.

François: [00:24:31] Now could you tell us about how it went once you develop this feature? This deployment process during the last two months of your project, did you manage to make these deliveries on time? And how did it happen?

Nicolas: [00:24:47] Yes, in the end it took close to six weeks from the point where we were alerted that we had an issue with the backend of our metrics to the point where we are able to say, “okay, we have completely finished”.

It was about six weeks. And we finished pretty much just the week before the planned migration. So it was kind of a just-in-time thing.. And at the end, obviously we were quite proud of ourselves to make it on time and be able to say that despite the challenges and the unknowns that we faced, we were able to deliver on time and not slow down the migrations that had involved dozens of people for over a year.

ACT III — Reflecting over the migration

François: [00:25:49] So looking back at this experience, now that it is passed, what would you have done differently to make it even more smooth to interact with different teams or to develop these features? Is there anything you’d ever made differently?

Nicolas: [00:26:06] So, as I said, it is easy to point fingers, but I don’t think that it is really productive.

And in that case, I’m not sure that there is something that you can really pinpoint as to we could have done better because obviously you can say, “try to get all your dependencies right” and “try to be sure that you can maintain ownership of your own code” but those things happen, They are not due to anyone messing up in particular.

They are just due to the fact that we are dealing with massive code bases with a lot of interactions involving a lot of teams. And some things are obviously going to slip between the cracks at some point. So the question is not, “how could we have prevented it”, it’s more, “how do you react once they happen” and to be sure that you can solve the issues as soon as possible.

And I think that we were all very satisfied with the ways that we handled it, in the end. And I think that we are all very grateful for the other assistance, because as I said, everyone was really doing their best. And that made it a very interesting experience for everyone involved. I think that it’s something also that can be missed in such a situation beyond the issues that were created for our schedule in terms of goals for our team.

That was also a very great opportunity to discover the code, for example, of the observability team. And it was quite interesting. Some things that I would not have had the opportunity to explore probably ever in my years at Criteo otherwise. And that gave me a lot of insight into how this part of our code base was working actually.

So I think that it is another important thing to keep in mind in those situations is to remain curious. And that’s actually one of the key values that we have in the Criteo AI lab, which is “always be learning”. And I think that it is something very positive and very encouraging on a day to day basis. We know that our hierarchy really does its best to allow us to continue to grow. And this was an opportunity to grow. And so it was quite natural to actually go forward for everyone involved.

François: [00:28:31] Okay. So in a nutshell, your main advice to deal with those situations is to nurture a sense of curiosity within your team and concretely, meaning being curious about other’s teams features, and code. Of course, you can’t be aware of the full details of implementation, but being at the first level comfortable with what they write about is something that made you gain time or that would have made you gain some time If you had prior knowledge of how their services work.

Nicolas: [00:29:02] Yes, I think that’s right. I think that it is something that is really important, both at the level of the company, but also at an individual level. I think that the tech scene is evolving at a quite rapid pace. Anyone involved in tech is aware of that.

And if you lose this natural curiosity, then it’s only a matter of time before you fall behind. And I think that it is really important for companies and for managers to understand this, to acknowledge it, and to really encourage the teams and the team members to continue trying to grow. And that’s, maybe this situation is not the first thing that would come to mind when you are thinking “what are going to be the benefits of me encouraging my team to continue learning?”. But actually it was. It was quite natural because it is encouraged in other ways in the company to actually go out of your comfort zone and to try to tackle this problem even so it was not what was planned.

François: [00:30:09] Okay. Thanks for those insights Nicolas, is there anything else you’d like to add on this experience? Other things you learned or hands-on advice fellow engineers could leverage in similar situations?

Nicolas: [00:30:25] Well, the one thing I might not have emphasized enough is also to really be customer-centric.

The goal here was to make sure that we would not impact our customers, internal customers, but still customers, in any adverse ways. We knew that they had more noisy AB tests and that they faced much more problems if we did not act quickly. And I think that keeping that in mind, really helps you prioritize what you need to do on a sprint to sprint basis.

And I think that it is very important to be able to separate a bit yourself from only the coolness of a project, the interest you may have in a given topic. And to think more often, how is it going to affect the people that depend on me? Whether they are external clients of the company, or even internal clients, to put yourself in their shoes and to do your best to facilitate their life.

Wrapping it up

François: [00:31:46] I think that’s a pretty neat conclusion of this project. Thanks. So I told you Nicolas before starting the interview that I’d be curious to hear about one recommendation you might have regarding this topic we just discussed, “how to handle incidents” or a project that encounters a challenge at any point.

Is there anything that comes to your mind that you’d like to recommend to our listeners?

Nicolas: [00:32:14] Yes. I will really recommend Clean Code from Robert Martin, which is nicknamed uncle Bob. It’s a book that is really impressive, that describes creating code as a craft, and encourages all programmers and people writing code to take more pride in what they are doing to think more thoroughly about what they are going to do and how it is going to affect other people coming back to the code.

Those other people could be you in a year. We often forget that a lot of time, other people needing to interact with our own code are future selves that have forgotten what we did. And it’s an excellent book that is really touching on all the parts of coding from naming the functions to How do you handle the responsibilities of classes and everything in between. And I think that taking those lessons to heart really help in reducing the chance that you face severe issues down the line.

François: [00:33:21] Good, thanks for this recommendation. Would you recommend it to any particular public starting engineers, mid's, seniors, or is it interesting for anyone?

Nicolas: [00:33:30] I think anyone seriously interested in coding should honestly try to pick that book and go through it. It’s quite a demanding book because there are a lot of examples where it asks you to think about the solution, but that makes it a much more effective book.

François: [00:33:47] Yeah. There is no free lunch. I mean, you need to invest some of your time.

Nicolas: [00:33:50] Yeah. Beyond that if you are interested in this migration project that we had at Criteo, I encourage you to actually go search. We wrote three blog posts, describing different parts of this project, which was, as I said, quite an ambitious one full of a lot of lessons.

François: [00:34:12] Good. So, we can find them on medium?

Nicolas: [00:34:16] Yes, they are all three on medium. And by Googling Criteo AI lab with a Mesos migration or Linux migration, you should find them quite easily. And also I’m guessing you’re going to put them in the show notes?

François: [00:34:30] Oh, absolutely. It’s going to be in the description of the show.

So make sure to check them out to learn more about this migration project, any last word, Nicolas?

Nicolas: [00:34:40]Yes, I’m actually, myself starting a project of my own. I’m starting a newsletter where I am discussing a lot of things related to tech and its impact on society. So please, if you think that this could be of interest to you, do not hesitate to go check it out.

The address is nicolaskowalski.substack.com

François: [00:35:05] Again, you’ll find it in the note show. Alright, thanks a lot Nicola for taking the time to share this experience with us. And I hope it will be very helpful to fellow engineers.

Nicolas: [00:35:18] Thanks a lot François for having me.

François: [00:35:26] Hi, this is François your host. I hope you enjoyed this first episode of Post Mortem. If so, please subscribe to the show and don’t hesitate to share it with a friend. Actually we really rely on word of mouth to get the podcast known.

Also, if you’d like to share your story in the show, don’t hesitate to shoot me an email at francois@postmortem.media. I repeat that’s francois@postmortem.media, M E D I A, and that’s it folks, bye bye.

Episode recorded in September 2020

Show notes

Nicolas walks us through a challenging service migration he experienced at Criteo, a personalized re-targeting company.

A Windows to Linux server migration was on the way for several months already when they realized that the logging backend they used was not working as expected under Linux and failed to collect metrics.

Problem: By that time, 95% of the machine park had already switched to Linux.

Product teams relying on those metrics to run their A/B tests were using small data sets subject to noise. As critical metrics, like the Click-Through Rate, were less abundant, business decisions were more challenging for Nicolas’ internal customers.

Nicolas shares with us the resolution process they undergo to restore the metrics collection. From an organization perspective, here are the main takeaways;

1. Be customer-centric in your resolution process; solving the client problem requires shifting the team priorities and adjusting its roadmap.
2. An organization with a culture that fosters curiosity reacts faster. Here, Nicola’s team’s ability to take ownership of a part of the code out of its initial scope was possible because curiosity about others’ team service, features, and how they work is encouraged within Criteo.

References

For more details about this migration project, check out Criteo’s blog post Moving .NET to Linux at Scale. Criteo’s engineers also published two additional articles to follow up on how they rolled out their .NET apps to Linux.

Nicolas recommends the book Clean Code from Robert Martin. He enjoys the specific examples developed in it, and the long-term thinking mentality the author invites developers to nurture to master their craft.

Last but not least, Nicolas is starting a newsletter, Memes Are Eating the World, where he explores tech and its impact on society; check it out at nicolaskowalski.substack.com