EE > Do the Work: Mechanical Tactics

A.k.a try to swallow the elephant in a single bite. I realize this sounds counter-intuitive (and impossible), but trying to quickly prototype a working end-to-end solution can generally pay outsized rewards. There are a couple of important parts I'd like to highlight:

• The hack must be end-to-end; to really get the value from this process you need to touch every part that will be involved in a real solution.
• You must timebox yourself: one of the points of this exercise is to identify the rabbit holes, not fall down every single one of them.
• Focus on making something that works: it doesn't need to be well designed, most of it can be hardcoded – you could simply write everything into a single 5,000 line file to avoid imports if you want to – free yourself from any “good practices”.
• You're not aiming to commit – or ship – any of this; remind others looking over your shoulder as well just in case. I'll often hard-code in names like “Foo” and “bar” to remind myself and others that this is in no way meant for real world use.

There are a lot of valuable outcomes from this approach:

• You'll get a much better appreciation for the actual work involved, and will have a surprisingly good idea of what “shape” the solution could take.
• There's a far lower likelihood of running into unknown unknowns and magical surprises as you learn to navigate the full system.
• You'll identify the riskiest parts that need to be done, as well as the easiest ones; this can also help you both delegate and estimate work with much more confidence.
• You'll be able to empirically test your mental model and confirm that it applies.
• There are few things more powerful than a working demo to align people: giving them something tangible to play with makes sure you're all working towards a common goal.

This is the part where you start nibbling at the edges of your (virtual) elephant to build up confidence that you'll actually be able to eat it all some day. Learn to work in your new project, the build and release process, how tests work, simply getting through code review and other processes to get your changes deployed.

I recommend making small changes and simply tidying up your workplace. Clean up existing lints, add more test coverage, improve the quality of the tests that do exist; fix spellings; add instrumentation; improve the documentation – particularly by adding examples. Make simple, easy to understand – preferably non functional – changes that will run in production. Make sure you can find out if they work correctly once they reach customers.

Along the way, you'll gain goodwill from your team, have something to show for all the time you've been spending ramping up on the system, and get that glow of satisfaction at actually landing something instead of rotating in place for months.

This part is a reminder to take small bites that you can chew (in complete contrast to the very first piece of advice I gave you). Because there are so many unknowns in this system, once you're actually working on a solution – move carefully. Make sure the system runs as you expect it to; make a small change, validate that it works again (similar to – but not necessarily the same as – test driven development).

If the results diverge from your mental model at any point you can immediately backtrack and identify what changed in the small step you took, instead of having to bisect a large swathe of changes (and possibly having to determine that two changes overlapped to cause issues). This simple mechanical behavior can save you hours of debugging and working backwards to identify unintentional consequences.

At times, it can take a little bit of humility to constrain yourself to small changes instead of giant leaps; you're most likely to learn the same way I did – which is to fail often enough till you learn to make small but confident steps. (Hat tip to Kent, who helped me work through this several times at the start of my career.)

(Credit for this section goes to Ivan Savov who pointed out that I never covered the importance of looking at the data.)

Your project almost certainly interacts with several other systems, some of which may be significantly easier to learn from. Look at the inputs, outputs, and persisted state of your program to see what's going on.

From a tactical perspective, simply reading code is not empirical evidence: you need to run it, and see exactly what it's doing. Simply reading code can be very misleading – you can never be sure what's overriding behavior in production. It could be something as cartoonish as an “#define false true” to something more realistic like compiler optimizations eliding code paths you care about.

Reaching for an example from something I've been working on recently: PyTorch supports several transforms to make your easy-to-modify but slow-to-run model into something that's extremely fast. But if you didn't know that it's going to be traced, you're going to have completely broken assumptions on what runs. Here's an example from torch.fx where my beloved debugging mechanism – a print statement – gets elided:

import torch

class MyModule(torch.nn.Module):

    def __init__(self):
        super().__init__()
        self.param = torch.nn.Parameter(torch.rand(3, 4))
        self.linear = torch.nn.Linear(4, 5)

    def forward(self, x):
        print(f"A wild print appeared! {x=}")
        return self.linear(x + self.param).clamp(min=0.0, max=1.0)

The traced equivalent that will run (notebook for proof):

def forward(self, x):
    param = self.param
    add = x + param; x = param = None
    linear = self.linear(add); add = None
    clamp = linear.clamp(min = 0.0, max = 1.0); linear = None
    return clamp

It's not just PyTorch: I've run into similar surprises across ART (while running the debugger), C optimizations, and other brilliant magic.

While ideally you'd be able to navigate a project simply based on a good folder structure, consistent naming schemes and obvious class naming patterns, reality tends to be significantly messier. It can be much easier to find an entry point by looking up fingerprints of other engineers and pulling on a thread starting from that point.

Some common fingerprints I use to quickly find a starting point to navigate:

• Unminified CSS class names
• User facing messages, particularly if they include rare phrases
• Specific combinations of words in log messages, making them easier to grep for

Looking for these generally takes me directly to a string or a constant that I can then follow through all the way.

Of course, this is more an art than a science: any of these strings could be generated dynamically; or you might be extremely unlucky and the phrase might end up being split up across multiple lines instead.

Be extremely conscientious while reading logs: it's too easy to miss something obvious right in front of you. Most of the logs correspond to something someone else thought was interesting about the program's behavior; you're likely to benefit by paying a little bit of attention.

Sadly, more often than not logs tend to be unfortunately spammy, occasionally out of order (if parallelism or concurrency is involved): don't hesitate to pull them together into an editor for quick manipulation. You'd be surprised at just how readable logs can be when formatted and highlighted correctly.

One of my favorite tactics is to collect all logs or compiler output into a file that I then manipulate in Vim: I delete everything extraneous, and reformat the bits I care about. Versioning these abbreviated log files makes change much easier to observe and dissect. In extreme cases, consider post processing the logs and visualizing them manually – a few lines of python in a notebook could save you hours of spelunking.

Any live instrumentation that captures the system's execution is the last set of data I'd recommend pulling up. Instrumentation should help you build empathy with how the system executes: do requests take milliseconds, seconds, or minutes? What does p99, p99.9, p99.99 latency look like? How many resources are consumed? Is this code CPU bound, I/O bound, or neither?

I almost never trust instrumentation and metrics that are handed to me: at the very least, I'd like to trace a single data point (that I generated manually) through the pipelines, and then I can accept that it's at least partially working. If you can triangulate metrics through multiple different sources, it's worth the investment. If you run across complex metrics that are hard to explain, don't rely on them for anything but sanity checks: look for simpler, direct metrics to actually guide your decisions as you explore the code.

Even though relying solely on reading code is misleading, you still need to be able to skim it really quickly. The same rules for reading books fast also apply to reading code: focus on the interfaces instead of digging deeply into the implementations, spend your time on critical pieces.

Incrementally building your mental model (as described above) and taking thorough notes along the way (as described near the end) should help you move faster and remember what's going on.

You should figure out ways to quickly navigate your code base for this to work really well. Some editors also offer bookmarks to be able to jump back and forth between interesting points quickly.

As you ramp up, don't ignore the tools available: you'll probably bring some along based on your past experience, but also take the time to learn the ones built for your current problem. For example, Android Studio encodes (and automates away) a tremendous amount of knowledge and conventions you'd otherwise need to manually look up and follow to build anything with Android – I couldn't recommend using Emacs for it.

Ideally, you should understand the strengths and weaknesses of the tools you have available, and build some intuition on when to use what – this knowledge inevitably compounds over time. One path is to use them on a trivial problem and identify how the tool works. Understanding how – and why – they work will help you be that much more effective with them.

I optimistically started writing this section and later realized that there was no realistic way to cover all the tools (or types of tools) available; I'm just covering some of the tools you might find useful to apply the tactics covered in the previous sections.