How many toolsmiths should your team have?

This notebook explores how much of your team you should allocate to toolsmiths, and the corresponding tradeoffs. The assumptions are explicitly laid out and varied over different sets of ranges to give an envelope of potentially useful values: jump to the end of this notebook for a visualization that summarizes the envelope. THere are several different ways to interpret this data as well, depending on what you consider the job of a toolsmith to be.

Before you spend too much time reading -- or making extremely important decisions -- based off this notebook, you should remember that everything is made up, and the numbers don't matter. The point of this notebook is to help build intuition, and not to be a precise recommendation. All the graphs are low-fidelity and XKCD-style to reinforce that fact.

There's also a tl;dr; near the bottom: you can jump to it by clicking here.

This is part of a series of notes on building developer tools that ended up being much longer than I ever anticipated, available here.

Utility functions

First, starting with a collection of functions that will be useful in building up the actual model, with a few simple tests next to each utility: following my style guide this notebook should be executable from top to bottom. (If you'd like to run it locally, remember to install humor sans for the pretty fonts.)

Set some nice defaults for plotting

Modeling the impact of toolsmiths

Using sigmoid cuvers as a way to model the effect of toolsmiths: if all the toolsmiths together put in $x$ person-days of work, the "productivity curve" returns the new productivity for engineers. (A "person-day" is the amount of work we expect the average engineer to complete in a day.)

The utility function is a heavily parameterized sigmoid curve to allow testing against different shapes and assumptions around how the effect toolsmiths can have.

Brooke's law

A confounding factor while dealing with large teams is the slow-down triggered because of the coordination overhead, among other things. Assumptions for this particular exploration:

(Presumably, some sanity prevails at that point, and engineers are organized into teams to cap the communication overhead.)

Estimating work done

For simple calculations, assuming

For these numbers, it'll take 200 days to complete the project (.5 productivity per engineer, 10,000 / (100 * .5)) without any toolsmiths.

To evaluate the effect of toolsmiths, it's interesting to determine:

This utility function simply walks through each day, adjusting the productivity multiplier and total work done along the way. The interesting data is returned as a tuple, along with a line demonstrating the work done per day.

Finding an optimal allocation

Finally building up to something interesting: a function to figure out the optimal allocation between toolsmiths and engineers by simply brute-forcing all options.

The example checks the optimum allocation with a linear productivity curve where we can double productivity by spending 10,000 person-days of work: both to maximize total work completed, and to minimize the time to finish the main project.

Then it checks these values by graphing out all the intermediate values.

Varying the optimal points

Last but not least, it's most interesting to identify how the maximum work, minimum time, and optimal allocation change with different productivity curves. Abstracting this out to avoid duplicate code.

Testing this out with the simple linear model of productivity that we used above.

Exploring with a linear productivity curve, while varying the maximum increase in productivity.

Different productivity curves

With all the utility functions in place, it's now possible to build an envelope of optimal toolsmith allocations depending on your assumptions around how effective the toolsmiths can be, and how hard their work is.

A complex domain that needs a lot of work

Assuming that the work the toolsmiths need to do is very complex, and much larger in scope than the actual project (say, 10x the size of the project). As in any new field, there are different stages -- exploration, expansion and extraction.

Looking at how total potential work / minimum time vary by moving along the curve: depending on different starting points along the curve, the toolsmiths might be very effective -- or completely useless.

There's a lot to unpack from these 2 curves:

Obviously,

Not so obviously,

And of course, it's always interesting to sanity check how work done progresses over time -- just to see when (or if) your toolsmith assisted team does better than the baseline.

It's interesting to note that the effect of the optimal allocation of toolsmiths only shows up around 3/4th of the way in: before that the raw output is lower than simply executing without toolsmiths. That's a pattern that'll reoccur throughout the different productivity curves.

Smaller, tightly scoped projects

On the other end of the spectrum, consider tooling projects that are the same order of magnitude of work as your actual project, and correspondingly have a much smaller win. In this case, it's interesting to figure out how much of a win toolsmiths need to add before they start adding value to your team.

From the curves: if your toolsmiths can get you more than 2.5x productivity multipliers by putting in as much effort as the size of your actual project, you want ~40% of your team to be toolsmiths. I'm actually surprised at the consistency of this number. As the productivity multiplier increases, total work done increases dramatically and work comes down.

Tiny projects

Considering tiny toolsmith projects which take a fraction of the time of the actual project.

Irrespective of the productivity multiplier, the optimum number of toolsmiths hovers around ~15 -- presumably because that number of engineers is sufficient to extract the value from the tooling.

For completeness, plotting out work done vs time for different sets of toolsmiths:

Varying the size of the project

Again, it's worth investing in toolsmiths as long as they can pull off a meaningful multiplier.

The point at which wins show up varies according to the size of the project

tl;dr;

Putting it all together: we're making a decision between directly doing the work, or spending part of our time/energy speeding up how we do the work. As someone who spends his days building developer tools, that's the subset of poeple I've considered -- but you can change the definition to be all infra engineers, or people working on software for collaboration instead. Set some bounds on how much of an impact you can realistically expect the meta-work to have; and then look up potential allocations of engineers to choose your team.

For example, if tools/infrastructure work can cut your build times in half: you could estimate that your engineers can become twice as effective with half the build time. Choosing a multiplier of 2, you should allocate a minimum of 10% of your team's time towards improving the build speed -- ideally enough to actually be able to hit the build speed improvements. Depending on just how much work it takes to speed up builds, you could potentially halve the time it takes to work on your project.

Of course, if it's too hard to get improve productivity: in relation to the actual work you're trying to do, you're better off not trying. The other caveat you should remember is that adding toolsmiths will initially slow down your team's output compared to a baseline team without toolsmiths; but over time this will invert as the effect of the toolsmiths' work starts showing up. On the other hand, if there's a lot of opportunity for toolsmiths -- it's not irrational to spend more than half of your resources towards speeding things up.

Finally, I'll leave you with a reminder that everything is made up, and the numbers don't matter: please only use this notebook to build your intuition around how toolsmiths can help, and not as a precise manual.

Acknowledgements

This notebook -- and the ideas inside -- were constantly refined with feedback from Kent Beck.