Impact helps show how much teams are delivering and how complex their work is. Like Velocity, it requires context to interpret. Baselines for each team are different, but it's typically an incredibly useful indicator of a team's actual capacity.
Impact attempts to answer the question: “Roughly how much cognitive load did the engineer carry when implementing these changes?”
Understanding Impact calculations
To illustrate, consider the following example:
One engineer makes a contribution of 100 new lines of code to a single file.
Compare that to another engineer’s contribution, which touches three files, at multiple insertion points, where they add 16 lines, while removing 24 lines.
The significance of each contribution can’t be boiled down to just the amount of code being checked in. Even without knowing specifics, it’s likely that the second set of changes were more difficult to implement, given that they involved several spot-edits to old code.
On the left we have someone adding 100 lines of new code to a single file. On the right is an example that represents only 24 lines of new code written, but there’s a bit more going on here:
- this change required modifying previous work
- the edits happened in four different locations
- three different files were affected
Even without knowing the severity of changes or comparing to historical changes, it’s probably safe to assume that the second contribution involved a higher cognitive load and therefore carries a higher impact score.
Although Change set #1 is technically more code, the added complexity of the work happening in Change set #2 could arguably make that change set at least as much work, and possibly more. Even in the simplistic representation above, it’s clear that there’s more than just lines of code at play.
Impact is a measure of work size that takes this into account. Impact considers the following:
- The amount of code in the change
- What percentage of the work is edits to old code
- The surface area of the change, like the number of edit locations
- The number of files affected
- The severity of changes when old code is modified
- How this change compares to others from the project history
We roll all of this up to create a score that more accurately lets us understand how “big” a particular piece of work is. The example above is based on data from a Flow repository, and when we evaluate the impact of these two changes we got the following results:
- Change set #1: 24 impact score
- Change set #2: 36 impact score
The more involved nature of Change set #2 yielded a higher impact score, even though only about a quarter of the amount of raw code was written.
Tip: Impact and Commit complexity can seem very similar. The two metrics serve two different purposes and are not interchangeable. While Impact considers the cognitive load of the work being done, Commit complexity looks at the riskiness of a commit.
Understanding Impact in context
At its core, Impact is a contextual metric. Its value comes from monitoring the metric and using it as a signal to detect changes in behavior and understand the effect of changes in your organization. Every team’s situation is different, so understanding your team and their Impact in context will help you address any issues and celebrate wins.
For aggregate team views over long periods of time, you can expect a sense of normalcy for Impact. This is because any short-term or team-specific fluctuations should even out when viewing multiple teams over longer time periods. These time and team selections help account for variations in project phases and languages used. These levels of normalcy can even be used for planning and setting expectations for what future work might look like.
For team-level granular views and shorter time periods, like week over week, it would be counterproductive to use Impact as an absolute comparative metric of normalcy or compare it across languages or stages of projects. Variation of Impact in those contexts is expected, given the delivery rhythm at which teams operate.
In many cases, the best thing you can do is understand what normal Impact looks like for your team and organization over long periods of time, then take note when the Impact shifts dramatically—for example, by more than 50 points. Then use it as a chance to start conversations with your team to understand the shift and whether more support is needed.
Consider these use cases for Impact:
-
Detecting unexpected behavior and starting conversations
-
As an example, say your team has just started a code freeze and are about to do a major release. Any changes to the code should be fit and finish only. There shouldn’t be major changes or new code—so your Impact shouldn’t be spiking. If it is, that’s an opportunity to have a conversation with your team about what prompted the Impact to understand if there are other issues in the project that you should discuss and resolve.
-
You can also expect to see a lower Impact at the beginning of projects where research and design are the focus and you may have fewer commits. This Impact should steadily increase through your project. If you notice a deviation from this pattern, talk with your team.
-
-
Measuring change in your organization
-
Say a senior engineer with a lot of codebase knowledge recently left your team. You can keep an eye on Impact, as well as other metrics, to understand how that loss affects your team’s coding.
-
Like with most Flow metrics, asking questions and having conversations about changes you see in Impact will drive the most positive changes in your team and organization.
Aggregate Impact calculations in reports
In reports like Team health insights, Impact is shown in aggregate and calculated as weekly Impact per developer.
Weekly Impact per developer is sum(Impact of users) / sum(total weeks of users)
Impact is the sum of the Impact of individual commits authored by a user.
Total weeks for a user is the total number of weeks in the period, minus weeks in which a developer committed no code.
For organization and team calculations, Impact is the sum of Impact for the individuals for the selected time period. Total weeks is the sum of the total weeks for each user in the time period.
Example Impact calculation
Let's say we have two engineers on a team. One engineer has an Impact of 100 on a particular week. The other engineer has an Impact of 50 on that same week. The following week, the first engineer takes some much-needed PTO and therefore has an Impact of zero. The second engineer has an Impact of 80.
Here's how Flow calculates the weekly Impact per developer for that team:
- Calculate Impact. Flow sums the total Impact of each team member during the time period. Here the total Impact would be
100 + 50 + 80 = 230. - Calculate total weeks. The first engineer committed during two weeks. The second committed during three weeks. This means total weeks for the team is three.
- Calculate weekly Impact per developer. Divide the total team Impact by the total number of weeks. Here, we get
230 / 7 = 77in weekly Impact per developer.