Approaching Debugging

The first (and most important) aspect of tackling a thorny problem is having the right mindset.

I personally like to approach bugs calmly, with a keen sense of curiosity and (possibly undue) optimism, because frustration, panic and pessimism can all end up causing tunnel vision, making the problem much, much harder to fix.

It's just code. You can fix it.

Understanding the problem is obviously important, but there's also a cost/benefit analysis to do first: how important is it? How much time (and how many engineers) are you willing to spend to fix it?

Is there a threshold at which it would be better to patch over the problem instead? Or, is it better to leave it unfixed²? ²: Leaving it unfixed is not a decision to be taken lightly: bugs, particularly heisenbugs tend to compound rapidly, and can render a code base impossible to work with or improve. With this in mind, incurring technical debt should be seen as a very explicit, conscious decision.

The process of debugging often involves switching between extremely involved depth-first and wide breadth-first searches when diagnosing the issue. In these cases, writing things down acts as an additional form of memory, and helps a lot when you need to be able to quickly bring additional people up to speed.

Most importantly, though, it can also help highlight what hasn't been tried yet.

Written notes also come in handy after the problem has been solved, for the purposes of retrospectives and spreading knowledge, as well as building a nice collection of war stories.

Make sure you clearly understand and are able to describe the issue: what's the actual behavior and what's the expected behavior?

Oftentimes, I'll find something I had considered a bug to be the result of complex systems interacting in unexpected but intended ways.

All through the process, you're going to constantly need to evaluate if/when you need to mitigate the problem (even if you don't quite know the right solution yet).

A common solution is to revert to a known-good state; patch around the problem, or disable a specific feature while the behavior is being fixed.

Some things you should consider when making this decision are your confidence in the mitigation, the ease and speed of deployment, the urgency of the problem and the likelihood of finding a "better" fix.

Bugs can have several distinctive characteristics: quickly identifying them can serve to speed up your debugging process significantly.

Some common tactics to apply:

• Are there any hints in logs or telemetry?
• Where does it happen? In specific devices, only in production, only in the development environment, everywhere?
• Who does it occur for? Are there any specific characteristics for affected users/identifying units for the bug? Instrumentation can be extremely valuable here.
• Does it reproduce consistently? Is it (shudder) a race condition?
• When did it start happening? What changed around that time?
• Visualize any available data so that you can look for patterns.

Ideally, you'll have enough instrumentation, telemetry and samples to be able to answer each of these questions quickly. Alternatively, though, collecting a small set of examples can also significantly serve to reduce the search space.

Based on the clues you've carefully written down, you might already have some intuition of where the problem is potentially happening: this step relies on your understanding of the system.

Another way to build this intuition is to look for the first point at which reality diverges from your mental model of the system: you need to understand why.

Make a hypothesis first, predict what will happen, and then go and apply the change. Change only one thing at a time to make sure you are able to clearly reason around cause and effect.

Figure out how to test changes cheaply and have a tight feedback loop. Consistently look for reasons to disprove your hypothesis.

There will be times when there are multiple tiny bugs coming together in wonderfully imaginative ways, which will then require you to apply multiple fixes. Remember to maintain a record of everything that's been tried!

Sometimes things just don't fit together, or there's just no time to play "scientist" and experiment. In these kinds of scenarios, you can always try and brute force the solution by treating the system like a black box and looking at it from the outside.

By looking at when the problem started happening, you can then begin to understand what changed. Look through the commit history; look at the changes in the surrounding systems; look through updates to hosts and other related applications. Perhaps something interesting happened in the world, causing load on the service to spike and the servers to fall over.

If nothing in the change logs or release notes looks out of the ordinary, then it's time to fall back to binary search. Simply revert the system to a known good state, and keep walking it forward till it starts failing; alternatively, you can also go backwards till it starts working.

If absolutely nothing else is working, then simply taking a break from the problem and letting your subconscious deal with it s something that can oftentimes turn out to produce really good results.

Distract yourself with other (preferably nontechnical) things: go for a walk, move away from the computer, doodle and recover.

Once you have an acceptable fix, now it's time for you to release it. I hope you have established some ways to quickly and smoothly release your code (or have plans to make it that way). Validate that the both actual and expected behaviors match up.

Then, celebrate! It's finally time to set aside a few moments to savor your victory, recover from the stress and take a well-earned break.

Things are all right with the world again.

You aren't done yet, though.

There's still a lot to learn from a complex bug, and a lot to do in order to to make dealing with the next one simpler.