I first read Accelerate by Nicole Forsgren, Jez Humble, Gene Kim back in 2018 when it was a hot topic in the office; only an hour or two would go by between overhearing somebody talk about it, internal blog posts were being written, and we’d receive emails about following the findings.

One of the goals behind the book was to run a series of surveys, gathering 23000 responses across 2000 organizations, then using statistical methods on the results in order to find a way to measure software delivery performance, and to find out what drives it. One of the most impressive things about the book is that it not only presents the research findings, but also goes into detail about why the findings are trustworthy, meaning that they not only tell us what they learned, but how they learned it.

I recently read through the book again and took a bunch of notes which cover the main points.

The authors wanted to find a set of metrics that could be used to measure delivery teams, with two key characteristics:

1. A focus on global outcomes. The goal of this is to ensure teams aren’t pitted against each other i.e. rewarding developers for throughput and operations for stability.
2. A focus on outcomes, not output. The goal of this is that people are rewarded for meeting organisational goals, rather than busywork.

The authors settled on four key metrics to measure the effectiveness of delivery teams:

1. Delivery lead time
2. Deployment frequency
3. Change failure rate
4. Time to restore service

1. Delivery Lead Time

In the product development context, there are two types of delivery lead time:

1. The time it takes to design and validate a product and/or feature, aka “the fuzzy front-end”
2. The time it takes to go from code committed to code successfully run in production

One of the challenges with measuring the first type of lead time is that it has a high level of variability, and it’s difficult to know when to start the clock.

For this reason, the authors prefer measuring the second type of delivery lead time; the time it takes to go from code committed to code successfully run in production.

The book emphasizes that shorter product delivery lead times are better because it enables:

1. Faster feedback on what is being built, which in turn enables earlier and more rapid course-correction
2. A fix to be delivered much more rapidly in the case of a defect or outage

2. Deployment frequency

Deployment frequency was used as an alternative measure to batch size, and combined with delivery lead time, a kind of delivery tempo can be measured.

Unlike in a factory assembly line setting (and aside from the fact that software development is not manufacturing or assembly line work), due to lack of visible inventory in software development, batch size is hard to measure and communicate.

The authors wanted to determine whether increasing this “tempo” resulted in a negative impact on the stability of the systems that were being worked on, and were surprised to find that for higher performing organisations, there was no trade-off. By building in quality, it was possible for them to have both high deployment frequency, and low change failure percentage.

In contrast, the “medium performers” group had a worse change fail percentage and time to restore service than the low performers in 2016, whereas in 2017, the change failure rate matched that of the “high performers” group (15%) - the reason for this wasn’t explained in the research, however authors suspect that this was likely because they were working through their journey to becoming higher performers, and running into challenges such as transitioning legacy code bases.

3. Change Fail Percentage

This is measured as the percentage of changes to production which subsequently require some sort of remediation such as a hotfix, rollback, fix-forward, or patch. For example:

• Software deployments or releases which break functionality
• Infrastructure releases which cause an outage

Change failure percentage maps well to the Lean concept of “percentage complete and accurate”, which is a key quality metric.

4. Time to restore service

Mean time between failure has been a traditional reliability measure, however in modern and complex systems, failure is inevitable. Some part of you system will always be broken for some people. Because of this, the authors decided to measure the average time between a change failure, and the subsequent remediation being completed.

Findings from the survey

The authors performed some statistical analysis on the survey results, and here are some highlights:

The authors found no correlation between particular types of systems and software delivery performance. This surprised me, as I would have expected that systems built in highly regulated environments and/or safety-critical/life-critical systems would have had a slower delivery tempo, and because of this, be categorised as “low performers”.

The authors also found evidence that organisations which were categorised as high performers were twice as likely to exceed their organisational goals such as profitability, productivity, market share, and number of customers. This might also be fairly intuitive for many, but there has historically been a distinct lack of evidence that there was a link between the two.

Unsurprisingly, teams that reported no approval process achieved higher software delivery performance, and a lightweight approval process predicts software delivery performance.

Those were just a few of the highlights. Appendix B has a nice wrap-up, summarising a bunch of findings. If you’re short on time, and just want to get the gist of the book, I recommend reading this section first.

Continuous delivery performance

Continuous delivery performance is a predictor of higher delivery performance, and there are multiple factors that support it.

Whether the teams can:

• Make large-scale changes to the design of their system without the permission of somebody outside the team.
• Make large-scale to the design of their system without depending on other teams to make changes in their systems or creating significant work for other teams.
• Complete their work without communicating and coordinating with people outside their team.
• Deploy and release their product or service on demand, regardless of other services that it depends upon.
• Do most of their testing on demand, without requiring an integrated testing environment.
• Perform deployments during normal business hours with negligible downtime.
• Perform pain-free deployments and releases.

and whether:

• System architecture is loosely coupled and well-encapsulated
• The organisation itself is loosely coupled and well-encapsulated. This relates to Melvin Conway’s observation that “organisations which design systems are constrained to produce designs which are copies of the communication structure within these organisations”
• Delivery teams are cross-functional, with all the skills necessary to design, develop, test, deploy, and operate the system on the same team.
• Security teams make pre-approved, easy-to-consume libraries, packages, and toolchains available to developers and IT operations to use in their work.
• Teams “shift left” on security. That is, performing security reviews throughout the development process, rather than as a separate phase at the end of the delivery process.
• Employees have the authority to make decisions that affect their work and their jobs, particularly in areas where they are responsible for the outcomes.

All this doesn’t mean that teams shouldn’t work together; the goal is to help ensure that communication bandwidth isn’t overwhelmed by fine grained decision making at the implementation level, so that teams can instead focus on discussing higher-level shared goals, and how to achieve them.

Lean product development practices

There were four product delivery practices which were examined and found to be statistically significant predictors of higher software delivery performance, along with organisational performance, culture, and reduction in burnout. Implementing these practices will likely enable a virtuous cycle::

1. Slicing up products and features so that they can be completed in small batches of less than a week, and released frequently.
2. Teams having a good understanding of the flow of work from the business all the way through to the customers, and having visibility of this flow including the status of products and features.
3. Customer feedback is actively and regularly gathered, and incorporated back into the design of the products.
4. Development teams have the authority to create and change specifications as part of the development process, without requiring approval.

Interestingly, I didn’t find any mention of the #1 core concept of Lean, which is muda, a Japanese word meaning “futility; uselessness; wastefulness”

Thanks for reading!

Grab a copy of the book here

Enjoyed this article? Follow me on Twitter

Any questions, feedback, or anything that you think I’d be interested in? Please leave a comment below, or get in touch with me directly.