#7 Don’t knock the checklist

A conversation with Timothy Raymond

Listen to the episode

Episode transcript

François: Hello and welcome to Post Mortem. My name is François; I’m your host for the show. In postmortem, we look back at production issues and understand their root causes, and we reflect on lessons learned. For today’s episode, Timothy Raymond joins me. Hi, Tim. -How’s it going? -Doing good. So, Tim, today you’re going to talk to us about an experience you have been through quite a while ago now. As an introduction, can you give us some context about the industry you were working in and the job you had at that time?

Tim: Sure. So this is actually my first position out of college. This was probably about six or seven years ago now. So think back to an earlier time in the Internet when it was about 2013, and Web advertising was a little bit more, well, less maligned than it was these days and a little more prevalent in the publishing industry. I was working for a media company that was ad-supported. Being ad-supported was actually kind of a unique thing because its competitors were subscription-based. So that was actually kind of a differentiating factor. And it made it easier for customers to access high-quality material without paying for it every month when their need was more occasional.

François: Can you give us a few figures in terms of monthly visitors? Or do you have other elements that could help us understand the scale at which you were operating?

Tim: Well, it was it’s hard to remember, but it was definitely in either the like hundreds of thousands per month. Occasionally, with seasonality, It could crest over like a million. So it was a high traffic kind of site. Seasonality would coincide with things like the holidays where people were shopping and looking for information about things.

Tim: So it varied, and it also varied along with different kinds of events that we would cover.

François: And in terms of organization, what was your position at that time? In which team were you working?

Tim: It was kind of a unique team because there was the entire development team for this was only about six people, which seems very small. But the nice thing is that we had a huge latitude to work on all kinds of stuff. So coming out of college, it was a blast because I could work on everything.

François: OK, so now that we’ve got some context, Can you give us more details about the incident that happened?

Tim: Sure. With any kind of ad-supported Web site, ads are a huge deal. That is where the money comes from. Up to this point, I had worked with just about every kind of system that we had. Various front ends actually served the website, various databases, and caches that you would find in most media serving systems. I’d also worked quite a bit on the content management system that our editorial team used to edit articles and publish content to users. About six months in, I wanted to work on ads because it was close to the heart of the business. And it was something I hadn’t worked with before. So I decided to take a feature and work on the ad system. So to give you a little bit of context on how Web advertising works for those folks at home who haven’t worked with Web advertising in the past. Let’s imagine that you have a subway car and there are all kinds of different advertisements; there are places for ads within the subway car so that you might have a couple of ads by the door, some over the seats, maybe those are a little bit different size. So very similar to a website where you have different sizes in different places and all kinds of things like that. So broadly, how advertising works on the Web is that you would talk to a representative from Google, which in the webspace would be a piece of JavaScript. You would walk them into the subway car, and you would say, “well, here are all of the places where I would like ads.”

Tim: Here’s the size of them. There are two over here by the door. You describe all of them, and you also can add additional context about them. You can say these are on the right side of the train. These are on the left side of the train. These are in the front of the car, in the back of the car. And later on, you can use that to apply constraints on which ads actually get served. So once you’ve described all of this, you walk out of the car, and then Google is left to actually place the ads inside of the car. And this is done. So that way, they can combat things like ad fraud because otherwise, who’s to say that you pasted the ads that Google gave you? So you are removed from the process of actually serving the ads.

Tim: So the feature that I picked up was to add a new piece of targeting and to continue with our analogy, it would be like to say, I want to specify where in like the position within the train. Maybe we want to target ads in the very back of the train or something like that. So I added this piece of targeting. It wasn’t especially difficult once I understood the nature of how all of this worked. And we had a very good code review culture; I walked through it with another engineer who was very familiar with this code and had written quite a bit of it himself.

Tim: He approved it. We deployed it. The deployment process was very good. It was very much DevOps focused. We just push it, It gets deployed. And I did this early in the morning, one day like early in the week because; Who wants to deploy on a Friday? And I also picked the day when we were planning to be in the office. We were doing a part-time remote kind of thing at the time. So I figured it would be a good idea to do this when we were going to be in the office, just in case there were any issues, everybody would be there. But again, it was a pretty innocuous change. So, we did that and, yeah, so everything seemed fine, we had all kinds of robust monitoring around all of the components, so we had systems that would tell us that if we couldn’t ping the website, which would indicate that we were down and this was completely isolated from any code that we had. This is just a straight ping of the website to test if it shows up. If it doesn’t, it’s down. We also were using pager duty to notify us if anything went down. So we had a high degree of confidence in making changes like this. So I deployed this probably around 07:30 in the morning as I was commuting in on the train and got to the office. We had our usual morning meetings, and everything seemed fine. I mean, it was great. We started moving on to the next thing.

Tim: And then something weird happened.

Tim: We started to get emails saying that ads weren’t working. At this company, we defined three broad fires. One, the site is down, which is relatively obvious. If we can’t see our pages, if nobody can read it, that’s clearly a fire. The second one is that the editorial staff could not publish anymore, also a problem. And the third one was that ads weren’t working. So this was the third fire. So, OK, that was drop everything that you’re doing and see what’s going on. So I loaded the page, and everything appears fine. The ads are out; we could see plenty of ads on the page. This job taught me to disable every kind of ad blocker on my machine and have a separate browser instance specifically for viewing an untainted copy of the Internet. So we looked at it in all, you know, in a very, like, neutral browser and.

Tim: Everything seemed fine, so we responded with this like we, you know, don’t see anything wrong? Oh, no, no, no, there’s definitely something wrong. We knew that something was wrong because there was a lot of emotion in those emails, that like something was deeply wrong and we just plain didn’t understand what was wrong with it. We had to get on the phone with the sales team who managed these advertisements and understand a little bit more about what they saw so we could understand why they thought that ads weren’t working. Because we saw ads. And part of the problem with advertising is that on most websites, you look at an ad and you kind of look away, right? You see it, you acknowledge it, and then you go and look at content. It’s very difficult to develop a trained sense to look at that ad and acknowledge what it is. And they change all the time, too, which makes it tricky as well. So what we found when talking with the sales team is that they had different classifications of advertising.
Some were campaigns that they have specifically sold to an advertiser. And these are going back to our subway car again. Have you ever gone into a subway car and seen the same advertisement everywhere? Oh, yeah. So they call this road blocking because they’ve sold every single slot in the car to a single advertiser. And this is usually a premium product. And the issue was that these kinds of advertisements weren’t showing up. So the ones that really paid quite a bit were not being served. So once we understood it, we definitely understood that there was a huge problem and the issue was now understanding why that was a problem.

François: And what was your diagnostic at this point? Did you immediately think about some change you introduced in this morning’s commute?

Tim: Adding extra targeting to an ad didn’t seem like it had anything to do with what we were seeing. So it wasn’t even clear that the two are related, and we thought something else was broken, that maybe some other configuration changed because the sales team itself could make changes to the configuration on how ads were served. So we needed to determine whether it was the change that I had made or something else that had happened in the intervening hours between, say, seven-thirty and eleven-thirty. So it wasn’t clear to us we had to figure out that no other changes were made on the other side. And that allowed us to at least eliminate the chance that the sales team had made some some change. We basically narrowed it down to it had to be somehow someway might change, even though it didn’t make any sense. And we had to set aside from for a moment the fact that it didn’t make any sense and had to take on faith, that whatever this effect that we saw, the cause had to be my change. So any bug, you get more understanding about the codebase.

Tim: What we had to do at that point was dig in and see specifically how the description of ads that I talked about actually gets passed to Google. So in our analogy, that conversation happens between you and a Google representative. As I mentioned, this is a piece of JavaScript that we are interacting with and also we’re dealing with the Web. So it turned out that the description of all of those advertisements, their sizes, locations, and things like that, all of those things were appended into a large query string in the URL.

Tim: And we started thinking about that. And we thought there are a lot of advertisements on this page and there’s a lot of specific information about them. And we actually captured a packet and it turned out this query string was so long that it actually exceeded 2000 characters, which while there’s no defined limit for the length of a URL; in practice, most browsers, most HTTP clients will truncate it at some point, like 2000 characters. In this case, what was happening was it was getting truncated, and we were losing a lot of the targeting information that was necessary to set these premium campaigns.

François: I see. So what was a fix you deployed to address this issue?

Tim: So the immediate fix was just to roll back. We had a good rollback system. We were using Heroku that let us just have like a one-click rollback. So we rolled back the change and things worked. And that’s kind of the nicest answer to any kind of incident, is if you have a good, good rollback strategy in place, you just employ it and everything comes back up and is working again. Longer-term, it became more about figuring out which pieces of targeting were actually useful. Some of them are nice to have, but we had to sit down and say, OK, which ones of these do we really need? And that was the longer-term solution to this once we revisited the issue and actually worked on it again.

François: And what were your main takeaways once you solve this issue? Was anything about, for instance, the system testing to make sure that ads were showing up?

Tim: We never found a way to test those things automatically. Once this happened, we evaluated the monitoring that we had in place because this was one of the three fires.

Tim: And we should have a way of detecting that a fire is occurring, ideally without somebody emailing us. Once we looked at everything, we thought we could add some automated, like scraping to see the ad campaigns. But it was also very difficult because ads change all the time and we’re not sure how they change. We didn’t necessarily want to know when they would change because there would be a lot more communication overhead with the sales team all the time. And we wanted to give them the independence to set their own campaigns and basically get out of their way. So what we ended up settling on was just a simple checklist and having a checklist. The changes to the advertising codebase wasn’t that often. So we could have a little bit more rigorous checklist for those kinds of changes. And one of those things was making sure that these premium campaigns were functional. So sometimes the simplest solution doesn’t require a lot of extra automated scraping and monitoring like alerts and Prometheus and things like that. Just a simple checklist with an extra item on it to make sure that one thing works is the best solution. And it served us pretty well. And it wasn’t necessarily the case that we didn’t have the capability to add. We could add all all kinds of custom metrics and things like that to these things, but we just decided that the overhead wasn’t worth it.

Tim: So and with the infrequency of the changes, this made the most sense.

François: Now, today, when you look back at this experience you had in this company, is there any particular take away that you’re taking with you and that you’re bringing along with you in your current jobs and your new role as a software engineer?

Tim: I think the biggest takeaway that I had was that solution over the long term, again, wasn’t some kind of big monitoring solution, I’ve worked for monitoring companies since then that have built time-series databases and companies that have built all kinds of amazing things on Kubernetes.

Tim: And I mean, these things are valuable, certainly. And it is amazing some of the things that you can monitor. But don’t knock the checklist like it is a great tool for just making sure that certain things are reviewed on your pull request, and you can make sure that even complicated changes things like that.

Tim: It is the lowest overhead in terms of development costs and things like that. That’s what I continually think about. And again, the simplest thing that could possibly work is sometimes the best.

François: Well, that’s a very good line to conclude our discussion. As closing words, do you have any recommendations for our audience, maybe books, blogs, or any exciting resources for software engineers?

Tim: Well, I have a nascent blog that I am trying to work on more in this New Year.

Tim: So I would invite people to check that out. It is at TimRaymond.com, and I’m counting on all of you to keep me accountable. And I’m going to create some great posts this year. And also, I will announce those posts on my Twitter, and that’s just at tim_raymond.

François: Thanks a lot, Tim. So, of course, I’ll put that info in the note show, so make sure to check out Tim’s blog posts and Twitter.

François: Thanks for paying attention to today’s episode of postmortem. If you want to support the show, the best way you can do this is by actually sharing this episode with a friend of yours. I wish you the best and see you in the next episode.

Show Notes

Action items resulting from a postmortem analysis are often to implement more advanced monitoring or deploy new tools to track usage at a fine-grain level.

Today, Timothy Raymond shows us that a good old checklist can sometimes be the most efficient way to prevent regressions in production.

Some concepts mentioned in the episode:

• Roadblocking in advertising: when a single ad or product is displayed all over the place to capture maximum attention.
• Beware of URL capping. As a rule of thumb, keep your URL below 2000 characters.

About our guest, Timothy Raymond (@tim_raymond on Twitter) is a seasoned Go programmer and currently a Principal Software Engineer at Microsoft. You can check his blog at timraymond.com