How to Measure Developer Experience: The Danger of Metrics

Whenever I talk with people about developer experience or developer productivity, one question is never far: How do we measure it?

And this is a valid question. We are used to making data-driven decisions and need to be sure that we invest our time and money into the things that actually matter. Therefore, the question is how you can measure developer experience and the impact of an investment on your engineers’ perceived experience.

What is the status quo in numbers? Where should those numbers be? And what kind of investment will bring our numbers to the target state?

When people do not have a great answer to that (great — think: dashboard full of numbers, graphs, and gauges), they often cannot generate the buy-in for developer experience investments. Quickly, some other initiative with more hard data to back it up becomes the priority. When a customer or prospect is behind a request, it is much easier to put a price tag on it (even if that tag is wrong).

And this is problematic.

Bias at Work: What we can measure seems more important

We live in a (business) world with a lot of data. Sometimes too much. We quickly become obsessed with our data and forget the aspects we cannot measure. — The things that we cannot explain with data.

Nothing becomes more important just because you can measure it. It becomes more measurable, that’s all. — W. Edwards Deming

When it comes to developer experience and developer productivity, for sure, we can measure things like the DORA metrics (change lead time, change failure rate, mean time to recover, deployment frequency). We can also count the number of merge requests, commits, story points per period, and other irrelevant aspects.

All those metrics might give us the confidence that we know how things are going. But this is often false confidence. First, all these metrics are output metrics, not outcome metrics. Outcome: What is the business value of all these deployments?

Furthermore, two companies can end up with the same numbers but in entirely different conditions. These metrics are trailing metrics, at best. They can stay at the same level for weeks or months, even when the company has fundamental issues.

Thus, judging developer productivity solely from hard numbers will yield a minimal and skewed insight. On the other hand, the satisfaction and well-being of engineers are leading indicators of developer productivity. But happiness and well-being are also much harder to measure.

Does Data Even Matter?

• When eight out of ten engineers complain about the software development process, do you really need a number to prove it needs improvement?
• When an increased number of people are frustrated about the structure and processes in the company, do you really need a written down business case before you take action?
• When people start to work long hours, burn out, and leave, do you really need a graph visualizing the issues?

I hope not. The tragedy with all those things is that they fundamentally damage your team’s morale, job satisfaction, performance, and productivity. However, employees often can compensate for inferior experience by working more and harder. This has disastrous long-term consequences, while short-term developer productivity metrics remain the same.

Thus, focusing on data alone will likely lead you in the wrong direction and will only alert you when it is almost too late.

How to Overcome This Issue?

In all companies I have worked with, engineers knew where the issues were and how they would rate their experience and productivity. Furthermore, no single company had no opportunities to improve the developer experience (and productivity).

Given the human tendency to prefer measurable things, it is safe to assume that your developer experience would benefit from more attention and investment than it currently receives.

If you are unsure where to start, surveying engineers and getting information on their perceived experience is an excellent place to start. It will give you a lot of ideas on potential issues. Furthermore, thoughtful and consistent survey design can help you to get as objective data as possible from this process. (Our Fractional DevEx Team is happy to help you with this.)

The key to measuring DevEx is to focus on developers and their lived experiences in delivering software. — “DevEx: What Actually Drives Productivity”

We can measure the subjective perception of engineers using surveys to understand their perceived satisfaction, well-being, and productivity. And, of course, we can and should augment those insights with hard data.

Measure What You Can Measure

While I believe we often need to focus more on the things we cannot measure (easily), this is not an excuse to not measure what we can measure (Our team can help with that, too.).

Whenever we lack data to back decisions and understand the state of our company, it is wise to think about ways to measure these aspects. However, if we cannot come up with good objective metrics, we especially need to be careful not to ignore the matter in favor of something that is more data-backed.

Conclusion

As we steer through the data-centric business landscape, it’s crucial to remember that not every valuable element can be quantified, and not every quantified element is valuable.

While hard metrics provide a tangible means to trace developer productivity, they fall short of encompassing the qualitative nuances of the developer experience.

Despite being challenging to measure, developer satisfaction, well-being, and morale are essential productivity indicators. Therefore, balancing what can be measured and what cannot is critical. Whether through carefully crafted surveys, open conversations with engineers, or nurturing an understanding culture, we need to measure developer experience with an open mind and readiness to look beyond the hard numbers.

The key to efficient developer experience lies in blending quantitative and qualitative measures to sketch a comprehensive picture of your team’s actual productivity.

Want to discuss how to improve and measure developer experience and productivity in your specific context? — Schedule a free 30-minute call with me where we will get to know each other and discuss exactly this!