How many toolsmiths should your team have?

This notebook explores how much of your team you should allocate to toolsmiths, and the corresponding tradeoffs. The assumptions are explicitly laid out and varied over different sets of ranges to give an envelope of potentially useful values: jump to the end of this notebook for a visualization that summarizes the envelope. THere are several different ways to interpret this data as well, depending on what you consider the job of a toolsmith to be.

Before you spend too much time reading -- or making extremely important decisions -- based off this notebook, you should remember that everything is made up, and the numbers don't matter. The point of this notebook is to help build intuition, and not to be a precise recommendation. All the graphs are low-fidelity and XKCD-style to reinforce that fact.

There's also a tl;dr; near the bottom: you can jump to it by clicking here.

This is part of a series of notes on building developer tools that ended up being much longer than I ever anticipated, available here.

Utility functions

First, starting with a collection of functions that will be useful in building up the actual model, with a few simple tests next to each utility: following my style guide this notebook should be executable from top to bottom. (If you'd like to run it locally, remember to install humor sans for the pretty fonts.)

Set some nice defaults for plotting

import matplotlib.pyplot as plt
import matplotlib as mpl
import seaborn as sns

mpl.rcParams['figure.figsize'] = [16.0, 5.0]
plt.xkcd(length=200, randomness=1)
palette = sns.color_palette("deep")

from functools import partial

colors = {}
for i, label in enumerate(["productivity", "work", "toolsmiths", "time"]):
    colors[label] = palette[i]

Modeling the impact of toolsmiths

Using sigmoid cuvers as a way to model the effect of toolsmiths: if all the toolsmiths together put in $x$ person-days of work, the "productivity curve" returns the new productivity for engineers. (A "person-day" is the amount of work we expect the average engineer to complete in a day.)

The utility function is a heavily parameterized sigmoid curve to allow testing against different shapes and assumptions around how the effect toolsmiths can have.

import math

def sigmoid(x, a=1, b=1, c=1, d=1, e=0, f=0):
    return a / (b + c * math.exp(-d * (x + e))) + f

def display_curve(title, curve, xs=None, show_project_size=True):
    if xs is None:
        xs = list(range(100))
    plt.title(title)
    plt.plot(xs, [curve(x) for x in xs], color=colors["productivity"])
    plt.xlabel("Person-days of work")
    plt.ylabel("Productivity multiplier")
    if show_project_size:
        plt.axvline(x=100 * 100, color="black", linestyle="--", label="Project size")
        plt.legend()
    plt.show()
    
xs = [x/10 for x in range(-100, 100)]
display_curve("The default shape of sigmoid", sigmoid, xs, show_project_size=False)

Brooke's law

A confounding factor while dealing with large teams is the slow-down triggered because of the coordination overhead, among other things. Assumptions for this particular exploration:

• each additional team-mate costs 1% person-day of time
• the total overhead is capped at a maximum of half a person day

(Presumably, some sanity prevails at that point, and engineers are organized into teams to cap the communication overhead.)

def brookes_law(current_productivity, number_of_engineers):
    return current_productivity - min(.5, number_of_engineers * .01)

xs = list(range(0, 101))
plt.title("Work done per day per engineer with increasing engineers")
plt.xlabel("Number of engineers in the team")
plt.ylabel("Productivity Multiplier")
plt.plot(xs, [brookes_law(1, x) for x in xs], color=colors["productivity"])
plt.ylim(bottom=0)
plt.show()

Estimating work done

For simple calculations, assuming

• there are a 100 total engineers working on the project
• the total size of the project is 100 * 100 = 10,000 person-days

For these numbers, it'll take 200 days to complete the project (.5 productivity per engineer, 10,000 / (100 * .5)) without any toolsmiths.

To evaluate the effect of toolsmiths, it's interesting to determine:

• what's the maximum amount of work we could do in 200 days
• how fast can we complete the project

This utility function simply walks through each day, adjusting the productivity multiplier and total work done along the way. The interesting data is returned as a tuple, along with a line demonstrating the work done per day.

def calculate_work(*, toolsmiths, engineers, productivity_curve, brookes_law=brookes_law, project_size=10_000, default_days=200):
    engineer_work = 0
    toolsmith_work = 0
    productivity = 1
    days = 0
    days_to_complete = None
    work_in_default_days = None
    ys = []
    
    while engineer_work < project_size or days < default_days:
        productivity = productivity_curve(toolsmith_work)
        toolsmith_work += toolsmiths * brookes_law(productivity, toolsmiths)
        engineer_work += engineers * brookes_law(productivity, engineers)
        days += 1
        
        ys.append(engineer_work)
        
        if engineer_work >= project_size and days_to_complete is None:
            days_to_complete = days
        if days == default_days:
            work_in_default_days = engineer_work
    
    return (work_in_default_days, days_to_complete, ys)

baseline_work, baseline_time, ys = calculate_work(toolsmiths=0, engineers=100, productivity_curve=lambda x: 1)
print(f"{baseline_work = }, {baseline_time = }")

baseline_work = 10000.0, baseline_time = 200

def compare_work(productivity_curve, toolsmiths_counts, title):
    plt.title(title)
    
    for toolsmiths in toolsmiths_counts:
        _, _, ys = calculate_work(toolsmiths=toolsmiths, 
                                  engineers=100-toolsmiths, 
                                  productivity_curve=productivity_curve)
        plt.plot(list(range(0, 200)), ys[:200], label=f"{toolsmiths} toolsmiths")
    
    plt.xlabel("Total days")
    plt.ylabel("Work done")
    plt.ylim(bottom=0)
    plt.xlim(left=0)
    plt.legend()
    plt.show()

compare_work(
    productivity_curve=lambda x: 1 + x / 10_000,
    toolsmiths_counts=list(range(0, 100, 20)),
    title="Linear productivity curve")
           

Finding an optimal allocation

Finally building up to something interesting: a function to figure out the optimal allocation between toolsmiths and engineers by simply brute-forcing all options.

def optimum_work(productivity_curve, brookes_law=brookes_law, total_engineers=100, project_size=10_000, default_days=200):
    max_work = None
    min_duration = None
    
    for toolsmiths in range(total_engineers):
        work, duration, _ = calculate_work(toolsmiths=toolsmiths, 
                                           engineers=total_engineers - toolsmiths, 
                                           productivity_curve=productivity_curve, 
                                           brookes_law=brookes_law,
                                           project_size=project_size,
                                           default_days=default_days)
        if max_work is None or max_work[1] < work:
            max_work = (toolsmiths, work)
        if min_duration is None or min_duration[1] > duration:
            min_duration = (toolsmiths, duration)
            
    return max_work, min_duration

The example checks the optimum allocation with a linear productivity curve where we can double productivity by spending 10,000 person-days of work: both to maximize total work completed, and to minimize the time to finish the main project.

Then it checks these values by graphing out all the intermediate values.

max_work, min_time = optimum_work(lambda x: 1 + x / 10_000, brookes_law)
print(f"{max_work = }, {min_time = }")

max_work = (19, 10923.214441128348), min_time = (14, 187)

xs = []
ys = []
zs = []
for toolsmiths in range(100):
    work, duration, _ = calculate_work(toolsmiths=toolsmiths, 
                                       engineers=100 - toolsmiths, 
                                       productivity_curve=lambda x: 1 + x / 10_000) 
    xs.append(toolsmiths)
    ys.append(work)
    zs.append(duration)

fig, ax1 = plt.subplots()
ax1.set_title("Exploring toolsmith allocation")

ax2 = ax1.twinx()
ax1.plot(xs, ys, label="Total work done in 200 days (person_days)", color=colors["work"])
ax1.set_ylabel("Work done (person-days)")
ax1.set_xlabel("# of toolsmiths (out of a team of 100)")

ax2.plot(xs, zs, label="Time to complete 10_000 person-days project (days)", color=colors["time"])
ax2.set_ylabel("Time to completion (days)")

ax1.hlines([max_work[1]], 0, max_work[0], linestyle="dotted", color=colors["work"])
ax1.vlines([max_work[0]], 0, max_work[1], linestyle="dotted", color=colors["work"])
ax2.hlines([min_time[1]], min_time[0], 100, linestyle="dotted", color=colors["time"])
ax2.vlines([min_time[0]], 0, min_time[1], linestyle="dotted", color=colors["time"])

ax1.set_xlim(0, 100)
ax1.set_ylim(bottom=0, top=12000)
ax2.set_ylim(bottom=0, top=600)

ax1.legend(loc="lower right")
ax2.legend(loc="lower left")

plt.show()

Varying the optimal points

Last but not least, it's most interesting to identify how the maximum work, minimum time, and optimal allocation change with different productivity curves. Abstracting this out to avoid duplicate code.

Testing this out with the simple linear model of productivity that we used above.

def explore_optimum_work(*, title, xlabel, xs, ys, zs, ws, vs):
    """
        xs: value along x-axis
        ys: work completed / baseline
        zs: toolsmith allocation for optimum
        ws: time to completion / baseline
        vs: toolsmith allocation for optimum
    """
    fig, axs = plt.subplots(2, 1)
    fig.set_size_inches(16.0, 10.0)
    
    axw = axs[0]
    axw.set_title(title)
    axw.plot(xs, ys, label="Total work done / baseline", color=colors["work"])
    axw.set_ylabel("Total work done / baseline")
    axw.set_xlabel(xlabel)
    
    axw.set_ylim(bottom=0)
    axw.legend(loc='upper left')
    
    axw2 = axw.twinx()
    axw2.plot(xs, zs, label="Toolsmiths", color=colors["toolsmiths"])
    axw2.set_ylabel("Toolsmiths")
    axw2.set_ylim(bottom=0, top=100)
    axw2.legend(loc='center right')
    
    axd = axs[1]
    axd.plot(xs, ws, label="Total time (days) / baseline", color=colors["time"])
    axd.set_ylabel("Total time (days) / baseline")
    axd.set_xlabel(xlabel)
    axd.set_ylim(bottom=0)
    axd.legend(loc='lower left')
    
    axd2 = axd.twinx()
    axd2.plot(xs, zs, label="Toolsmiths", color=colors["toolsmiths"])
    axd2.set_ylabel("Toolsmiths")
    axd2.set_ylim(bottom=0, top=100)
    axd2.legend(loc='center right')
    
    plt.tight_layout()
    plt.show()

Exploring with a linear productivity curve, while varying the maximum increase in productivity.

linear_productivity = lambda x, multiplier=1: 1 + multiplier * x / 10_000

xs = []
ys = []
zs = []
ws = []
vs = []

base_work, base_days, _ = calculate_work(toolsmiths=0, engineers=100, productivity_curve=linear_productivity, brookes_law=lambda x, _: x)
for multiplier10 in range(0, 41):
    multiplier = multiplier10 / 10
    xs.append(multiplier + 1)
    work, duration = optimum_work(partial(linear_productivity, multiplier=multiplier), brookes_law=lambda x, _: x)
    ys.append(work[1] / base_work)
    zs.append(work[0])
    ws.append(duration[1] / base_days)
    vs.append(duration[0])
    
explore_optimum_work(
    title="Linear productivity curve without brooke's law", 
    xlabel="Final productivity at the end",
    xs=xs,
    ys=ys,
    zs=zs,
    ws=ws,
    vs=vs)

Different productivity curves

With all the utility functions in place, it's now possible to build an envelope of optimal toolsmith allocations depending on your assumptions around how effective the toolsmiths can be, and how hard their work is.

A complex domain that needs a lot of work

Assuming that the work the toolsmiths need to do is very complex, and much larger in scope than the actual project (say, 10x the size of the project). As in any new field, there are different stages -- exploration, expansion and extraction.

def exploratory_tooling_curve(x, offset=0):
    a = 49
    d = 1 / 12000
    e = -40000 + offset
    return 1 + sigmoid(x, a=a, d=d, e=e) - sigmoid(0, a=a, d=d, e=e)
    
display_curve("A complex domain for toolsmiths", exploratory_tooling_curve, list(range(0, 100_000, 10)))

Looking at how total potential work / minimum time vary by moving along the curve: depending on different starting points along the curve, the toolsmiths might be very effective -- or completely useless.

base_work, base_days, _ = calculate_work(toolsmiths=0, engineers=100, productivity_curve=exploratory_tooling_curve)

xs = []
ys = []
zs = []
ws = []
vs = []
for offset in range(0, 100_000, 3000):
    xs.append(offset)
    work, duration = optimum_work(lambda x: exploratory_tooling_curve(x, offset=offset), brookes_law)
    ys.append(work[1] / base_work)
    zs.append(work[0])
    ws.append(duration[1] / base_days)
    vs.append(duration[0])

explore_optimum_work(
    title="A complex domain", 
    xlabel="Existing progress around the productivity curve",
    xs=xs,
    ys=ys,
    zs=zs,
    ws=ws,
    vs=vs)

There's a lot to unpack from these 2 curves:

Obviously,

• once you hit diminishing returns, there isn't much point in investing in toolsmiths; simply push to get more work done

Not so obviously,

• if there's a lot of potential to be had with better tools, you want more than half of your team working as force multipliers
  • particularly if the toolsmiths need to do a lot of exploration
• having toolsmiths can lead to an order of magnitude more work, with significantly lower time to completion depending on the potential of the tools

And of course, it's always interesting to sanity check how work done progresses over time -- just to see when (or if) your toolsmith assisted team does better than the baseline.

compare_work(productivity_curve=exploratory_tooling_curve, 
             toolsmiths_counts=[0, 40, 80, 99], 
             title="Exploratory tooling curve")

It's interesting to note that the effect of the optimal allocation of toolsmiths only shows up around 3/4th of the way in: before that the raw output is lower than simply executing without toolsmiths. That's a pattern that'll reoccur throughout the different productivity curves.

Smaller, tightly scoped projects

On the other end of the spectrum, consider tooling projects that are the same order of magnitude of work as your actual project, and correspondingly have a much smaller win. In this case, it's interesting to figure out how much of a win toolsmiths need to add before they start adding value to your team.

def defined_tooling_curve(x, bonus=1):
    a = bonus
    d = 1 / 1000
    e = -5000
    return 1 + sigmoid(x, a=a, d=d, e=e) - sigmoid(0, a=a, d=d, e=e)
    
display_curve(f"A defined problem: max multiplier {1 + 1}", lambda x: defined_tooling_curve(x, bonus=1), list(range(0, 10_000)))

base_work, base_days, _ = calculate_work(toolsmiths=0, engineers=100, productivity_curve=defined_tooling_curve)

xs = []
ys = []
zs = []
ws = []
vs = []
for bonus10 in range(0, 100):
    bonus = bonus10/10
    xs.append(bonus + 1)
    work, duration = optimum_work(lambda x: defined_tooling_curve(x, bonus=bonus), brookes_law)
    ys.append(work[1] / base_work)
    zs.append(work[0])
    ws.append(duration[1] / base_days)
    vs.append(duration[0])
    
explore_optimum_work(
    title="A defined problem", 
    xlabel="Final productivity multiplier with 10,000 person-days of effort",
    xs=xs,
    ys=ys,
    zs=zs,
    ws=ws,
    vs=vs)

From the curves: if your toolsmiths can get you more than 2.5x productivity multipliers by putting in as much effort as the size of your actual project, you want ~40% of your team to be toolsmiths. I'm actually surprised at the consistency of this number. As the productivity multiplier increases, total work done increases dramatically and work comes down.

compare_work(productivity_curve=defined_tooling_curve, 
             toolsmiths_counts=[0, 40, 80, 99], 
             title="Defined tooling curve")

Tiny projects

Considering tiny toolsmith projects which take a fraction of the time of the actual project.

def tiny_tooling_curve(x, bonus=1):
    a = bonus
    d = 1 / 100
    e = -500
    return 1 + sigmoid(x, a=a, d=d, e=e) - sigmoid(0, a=a, d=d, e=e)
    
display_curve(f"A defined area: max multiplier {1 + 1}", lambda x: tiny_tooling_curve(x, bonus=1), list(range(0, 10_000)))

base_work, base_days, _ = calculate_work(toolsmiths=0, engineers=100, productivity_curve=tiny_tooling_curve)

xs = []
ys = []
zs = []
ws = []
vs = []
for bonus10 in range(0, 30):
    bonus = bonus10/10
    xs.append(bonus + 1)
    work, duration = optimum_work(lambda x: tiny_tooling_curve(x, bonus=bonus), brookes_law)
    ys.append(work[1] / base_work)
    zs.append(work[0])
    ws.append(duration[1] / base_days)
    vs.append(duration[0])
    
explore_optimum_work(
    title="A tiny problem", 
    xlabel="Final productivity multiplier with 10,000 person-days of effort",
    xs=xs,
    ys=ys,
    zs=zs,
    ws=ws,
    vs=vs)

Irrespective of the productivity multiplier, the optimum number of toolsmiths hovers around ~15 -- presumably because that number of engineers is sufficient to extract the value from the tooling.

For completeness, plotting out work done vs time for different sets of toolsmiths:

compare_work(productivity_curve=tiny_tooling_curve, 
             toolsmiths_counts=[0, 40, 80, 99], 
             title="Tiny tooling curve")

Varying the size of the project

def tooling_curve(x, size=1):
    a = 1
    d = 1 / (100 * size)
    e = -500 * size
    return 1 + sigmoid(x, a=a, d=d, e=e) - sigmoid(0, a=a, d=d, e=e)

display_curve(f"Varying the size: {5} of the curve", lambda x: tooling_curve(x, size=5), list(range(0, 10_000)))

base_work, base_days, _ = calculate_work(toolsmiths=0, engineers=100, productivity_curve=tooling_curve)

xs = []
ys = []
zs = []
ws = []
vs = []
for size in range(1, 100):
    xs.append(size / 10)
    work, duration = optimum_work(partial(tooling_curve, size=size/10), brookes_law)
    ys.append(work[1] / base_work)
    zs.append(work[0])
    ws.append(duration[1] / base_days)
    vs.append(duration[0])
    
explore_optimum_work(
    title="Differently sized problems", 
    xlabel="Size of problem compared to the actual project",
    xs=xs,
    ys=ys,
    zs=zs,
    ws=ws,
    vs=vs)

Again, it's worth investing in toolsmiths as long as they can pull off a meaningful multiplier.

compare_work(productivity_curve=partial(tooling_curve, size=5), 
             toolsmiths_counts=range(0, 100, 20),
             title="Tooling curve with size 5")

The point at which wins show up varies according to the size of the project

tl;dr;

Putting it all together: we're making a decision between directly doing the work, or spending part of our time/energy speeding up how we do the work. As someone who spends his days building developer tools, that's the subset of poeple I've considered -- but you can change the definition to be all infra engineers, or people working on software for collaboration instead. Set some bounds on how much of an impact you can realistically expect the meta-work to have; and then look up potential allocations of engineers to choose your team.

For example, if tools/infrastructure work can cut your build times in half: you could estimate that your engineers can become twice as effective with half the build time. Choosing a multiplier of 2, you should allocate a minimum of 10% of your team's time towards improving the build speed -- ideally enough to actually be able to hit the build speed improvements. Depending on just how much work it takes to speed up builds, you could potentially halve the time it takes to work on your project.

Of course, if it's too hard to get improve productivity: in relation to the actual work you're trying to do, you're better off not trying. The other caveat you should remember is that adding toolsmiths will initially slow down your team's output compared to a baseline team without toolsmiths; but over time this will invert as the effect of the toolsmiths' work starts showing up. On the other hand, if there's a lot of opportunity for toolsmiths -- it's not irrational to spend more than half of your resources towards speeding things up.

Finally, I'll leave you with a reminder that everything is made up, and the numbers don't matter: please only use this notebook to build your intuition around how toolsmiths can help, and not as a precise manual.

from mpl_toolkits import mplot3d
import numpy as np

baseline_work, baseline_time, _ = calculate_work(toolsmiths=0, engineers=100, productivity_curve=lambda x: 1)

def final_productivity_curve(x, y):
    a = x - 1
    d = 1 / (1000 * y)
    e = -5000 * y
    adj = sigmoid(0, a=a, d=d, e=e)
    
    def curve(x):
        return 1 + sigmoid(x, a=a, d=d, e=e) - adj
    
    work, duration = optimum_work(curve)
    return (work[0], work[1] / baseline_work, duration[0], duration[1] / baseline_time)

samples = 20
x = np.linspace(1, 10, samples)
y = np.linspace(0.1, 3, samples)

X, Y = np.meshgrid(x, y)
Z = np.vectorize(final_productivity_curve)(X, Y)

def add_3d_plot(ax, z, title, label=None, cmap=mpl.cm.coolwarm):
    label = label or title
    ax.set_title(title)
    ax.set_xlabel("Final productivity multiplier")
    ax.set_ylabel("Size of the toolsmiths project")
    ax.set_zlabel(label)
    ax.plot_surface(X, Y, z, label=label, cmap=cmap)
    
elevation=20
    
fig = plt.figure(figsize=(20, 20))
ax1 = fig.add_subplot(2, 2, 1, projection='3d')
ax1.view_init(elev=elevation, azim=240)
add_3d_plot(ax1, Z[0], "Toolsmiths to maximize work", "Ideal number of toolsmiths", cmap=mpl.cm.copper)

ax2 = fig.add_subplot(2, 2, 2, projection='3d')
ax2.view_init(elev=elevation, azim=180)
add_3d_plot(ax2, Z[1], "Maximum possible work / baseline", cmap=mpl.cm.coolwarm_r)

ax3 = fig.add_subplot(2, 2, 3, projection='3d')
ax3.view_init(elev=elevation, azim=240)
add_3d_plot(ax3, Z[2], "Toolsmiths to minimize time", "Ideal number of toolsmiths", cmap=mpl.cm.copper)

ax4 = fig.add_subplot(2, 2, 4, projection='3d')
ax4.view_init(elev=elevation, azim=-45)
add_3d_plot(ax4, Z[3], "Minimimum possible time / baseline", cmap=mpl.cm.coolwarm)

plt.tight_layout()
plt.show()

Acknowledgements

This notebook -- and the ideas inside -- were constantly refined with feedback from Kent Beck.