Three things I disagree with the DevEx Paper
Great paper, but I don’t entirely agree with everything…
Introduction
The industry is catching up to the wisdom that developer experience (DevEx) is paramount to increase productivity for companies that produce software. So then the paper “DevEx: What Actually Drives Productivity?” by Abi Noda, Margaret-Anne Storey, Nicole Forsgren, and Michaela Greiler was released a few months ago, I wasn’t too surprised to see it go viral.
Yet, as much as I loved the paper and the ideas behind it and I am an admirer of the authors’ work, there are a few points where I don’t entirely agree.
I tried to reach out to Abi Noda for comment on all the points below but he is understandably very busy, so I thought I would just post this and hopefully spark some discussion around it.
My motivations behind writing this are purely for my own edification, I am not being paid to write this by anyone -I am just passionate about this topic.
I am not implying by writing this post that I know better than the authors or that I could write a better paper. It’s a lot easier to poke holes at something than creating it yourself. I am just contrasting what the paper says with my experience and what I’ve seen working in various DevOps roles and with other developers.
Quick summary of the paper
(If you have read the paper, you can skip this part)
The paper “DevEx: What Actually Drives Productivity?” by Abi Noda, Margaret-Anne Storey, Nicole Forsgren, and Michaela Greiler discusses the challenges of measuring and improving developer productivity in engineering organizations. The authors argue that traditional approaches to measuring productivity, such as measuring output or task completion time, are not enough to capture the complex activities of developers. They propose a new framework for understanding and improving developer productivity, called DevEx.
The paper goes on to present a framework with three core areas where companies need to create metrics in order to improve DevEx:
- Feedback loops: Developers need to be able to get quick and accurate feedback on their work. This helps them to identify and fix problems early, and to stay on track. When feedback loops are slow or inaccurate, it can lead to frustration and decreased productivity.
- Cognitive load: Developers need to be able to focus on their work without being distracted. This means having the right tools and resources at their disposal, and being able to work in an environment that is free from distractions. When cognitive load is high, it can lead to mistakes and decreased productivity.
- Flow state: Flow state is a state of intense focus and concentration where developers are fully immersed in their work. When developers are in flow state, they are able to work for long periods of time without getting tired or bored. They are also more productive and creative.
The authors mention that metrics can be captured as perceptual metrics (subjective, survey based) and workflows (objective, time or step based) in order to get a holistic picture of the developer experience at your organization.
1- It trivialises the importance of tooling in the developer experience
My biggest objection of the paper is what it has to say about tooling:
A common misconception is that DevEx is primarily affected by tools. Our research, however, shows that human factors such as having clear goals for projects and feeling psychologically safe on a team have a substantial impact on developers’ performance
I don’t disagree that psychological safety and clear goals have a substantial impact on developers’ performance. I also agree that culture comes before tooling, but I disagree with the insinuation here that tools are not one of the primary causes of bad DevEx.
In fact as you read on, the paper seems to contradict itself. For example when you look under the definitions of their three core areas (Cognitive load, feedback loops and flow state), tools get mentioned extensively in two of them.
On Feedback loops:
To improve DevEx, organizations should shorten feedback loops by identifying areas where development tools can be accelerated (e.g., build and test processes, development environment setup) or human hand-off processes improved
On Cognitive Load:
Dedicated DevEx and platform teams should provide easy-to-use, self-service tools to help developers streamline the steps for development and release.
Tools are the abstractions you put in place to make your developers’ life easy.
A developer doesn’t need to know all the technical details of how a server gets provisioned, or how an ip is whitelisted, if a tool can make any of these tasks easy so the developer can focus on their craft, that’s good tooling.
Whether it is implementing your own abstractions in the form of APIs, portals or applications, or picking open source and commercial tooling for your organization, the choices you make will have a dramatic impact on the developer experience and productivity of your company.
User Experience on the tooling matters more than number of features
Companies may be tempted to pick or create tools that can do everything, but if not done well this will likely impact the developer experience negatively because there is a tradeoff between usability and flexibility:
You can give tools to your developers that are very feature rich, but are very difficult to use, resulting in too high a learning curve and cognitive load and this will negatively impact DevEx.
A good example here is Jenkins vs Gitlab CI. Jenkins is so much more powerful than Gitlab CI, you can do so much more with it and it is infinitely extensible with plugins. And yet, most developer teams I’ve met prefer to use Gitlab CI. Simply because Gitlab CI provides a far superior user experience, the syntax is cleaner and simpler to use, it is better integrated and far more intuitive, and easier to get started with.
My own experience with GCP vs AWS is another example. You could argue that AWS gives you the moon in terms of features and products, but AWS suffers from a terrible user experience. It is confusing to use, navigate and very hard to learn. Whereas GCP is far more intuitive and better put together, even if it has fewer features and products than AWS. Most people I know who have used both overwhelmingly prefer GCP because they can get things done quicker.
The bottomline here is that tools matter a lot, and while DevEx is definitely not entirely driven by tooling, it certainly is one of the core areas that impacts it. If you want to provide a better DevEx you need to take into consideration the following factors when assessing your tooling:
- How intuitive is it? Can you get something done without having to refer to the documentation constantly?
- How easy is it to navigate? Do people often get lost with it?
- Do the designers of the tool take most of the responsibility to make it easy, or is it outsourced to the user? If the tool involves a lot of toil and steps, then that’s a good indicator that the design is not thorough.
- Does it solve most of the common problems it aims to solve?
A good tool makes a user happy like this:
Tooling abstractions in the form of platforms and developer portals are often associated with DevEx for this very reason, because developers are bogged down by too much work outside of their purview that prevents them from focusing on delivering code.
2- Core dimensions are overlapping
The areas chosen as the core dimensions for developer experience in the paper are cognitive load, flow state and feedback loops. The paper alludes to “25 sociotechnical factors” that inspired the three core dimensions but it doesn’t state what these are, which is a shame, because it would be helpful information to devise metrics and surveys. To be fair, a footnote points to where these factors are listed and talked about, but you need to sign up to read the paper, so not the most useful source.
For someone who is trying to improve DevEx in their organization if you tell them they need to measure Cognitive Load may be helpful, but it would be far more helpful if narrow down a list of concrete areas where they can tackle, for example:
- Infrastructure provisioning — often not the expertise of developers or directly related to their job but foisted on them.
- Configuration management — again, developers configuring servers or other infrastructure in ways that’s complicated and not standard.
- Tooling with terrible user experience — see my previous section in this article.
- Tedious processes — in banks and big corporations this can be soul drenching aside from giving you cognitive overload.
- Operational experience — how easy it is for platform engineers to deliver self-service solutions for developers. Sometimes the desire is there, but the means are missing.
And for each of those areas you could arguably have a dozen example metrics. The paper offers some good examples that cover some of the above, but it’s far from comprehensive.
3- Flow state should be the goal of DevEx — not productivity
The paper has flow state as one of the core areas to improve, but when you look at examples on what to measure there, they are mostly things related to planning, scheduling and vision, which are also very relevant to achieve flow, but you can’t independently tackle them because cognitive load and lack of feedback loops are also a burden that prevents engineers from achieving flow state.
Flow state shouldn’t be a core area to improve, because flow is what you get when you fix all the other areas that impede it, in other words, good DevEx and flow are intertwined. Flow always yields more creativity and productivity.
If your organization has all the processes, culture and tooling in place so that it is easy to reach flow when developers need to do some work, you are off the races — this is good DevEx. And while you may have cases where flow state is achieved by pursuing the wrong goals, generally speaking it goes hand in hand with great vision and planning.
If instead of productivity, the aim is to make conditions so good that flow comes easy, productivity and creativity will follow. A good example here are video games. If you make games that are engaging to play, sales naturally follow, so videogames designers would do better in focusing on achieving that, not selling more games.
I get why you see all these studies hammering on productivity and not flow. Productivity would definitely sell to an executive team, flow state… not so much, especially if the leadership doesn’t have an engineering background. The irony is that the authors know full well how difficult it is to measure productivity for developers because they rightly define it as more of an art than bricklaying, but the mere act of focusing on productivity and not flow, keeps linking it more closely to a production line than to Michelangelo painting the sistine chapel.
Defining Good DevEx
I want to finish this post by defining what good DevEx looks like to me
The authors define DevEx as follows:
Developer experience encompasses how developers feel about, think about, and value their work.
Which works for me. But I think it’s even more helpful to define what good DevEx is, because that’s the objective:
Good DevEx is what you get when developers can easily enter and maintain a flow state during most of their productive hours at work.
That very simple definition works because when developers are in the flow they are generally happy, and productivity and creativity skyrockets. All factors that affect developer experience negatively, are usually factors that harm flow, such as:
- Meetings peppered throughout the day that don’t add value
- Tooling with awful user experience
- Toxic, pathological culture that distracts from creative endeavours.
- Onerous tasks outside of their purview that have not been properly abstracted with tooling.
- Etc.
You can deliver a good DevEx by simplifying or eliminating all areas that don’t contribute to the creative output and the delivery of applications for developers. Excellent DevEx increases the amount of time developers spend in a flow state, eliminates burnout, and mitigates developer churn, which in turn improves productivity — productivity being the net result, not the objective.
So when in doubt ask yourself, does this help the developers be in the flow in their most productive hours? If the answer is yes, do it — that’s good DevEx.
Conclusion
The DevEx paper and many other articles written by the authors are a source of a wealth of good information and advice, but in order to have an unambiguous and narrower focus on the developer well-being (and as a side-effect increase productivity), flow state is a much better compass and overall aim than productivity.
Focusing on productivity makes it too easy to create an illusion that we can easily find global measures of productivity, and creates opportunities for other consultancies to write misguiding and maligned posts at the expense of these papers. And while this wasn’t the intention of the DevEx and SPACE paper, the productivity focus may have encouraged these confident assertions that we can measure productivity.
Focusing on flow state is much harder to game and it has a sharper focus on the developer and the creative process, not upper management, and it will naturally lead to higher productivity. Optimizing flow state also means improving your engineering organization’s processes, culture, vision, tooling, working environment, hiring, culture, communication, planning, etc so even though the aim is simple, the scope for improvement is large. It is the art of reducing friction by making the number of steps and time spent on drudgery as close to zero as possible anywhere and everywhere.
It’s worth noting that pursuing flow state does not mean everyone needs to be in the zone all the time, it just means that when work needs to get done, doing deep work is easy.
