Accelerate Digest

The essential elements of a high performing organisation

This is a loose outline of the first part of Accelerate book which I totally recommend you read, especially if you are going through a DevOps transformation or starting a new company.

This book explores twenty four different capabilities that were thought to be key elements of high performing IT organisations and it aims to prove their effectiveness by running a study across multiple companies and engineering teams.

Up until this research was done by the authors, there was mostly anecdotal evidence that these practices were the reason for high performance. So yes, it was quite obvious that Google, Netflix, Flickr and many of this companies were practising these and achieving great results, however it was just as easy for the naysayers to claim that doing DevOps only works for greenfield projects, unicorns or rare cases like Google.

This book presents abundant amount of evidence that this is not the case, and it also shows how the evidence was gathered. The method presented in this book is what is used to present the yearly State of Devops Report.

The capabilities are spread over five different areas, if your company is doing all of this well, it very likely you are a high performance organisation. Use them as reference to improve your overall performance:

Continuous Delivery Capabilities

1. Version Control
2. Deployment Automation
3. Continuous Integration
4. Trunk Based Development
5. Test Automation
6. Test Data Management
7. Shift Left on Security
8. Continuous Delivery

Architecture Capabilities

9. Loosely Coupled Architecture

10. Empowered Teams

Product and Process Capabilities

11. Customer Feedback

12. Value Stream

13. Working in Small Batches

14. Team Experimentation

Lean Management and Monitoring Capabilities

15. Change Approval Processes

16. Monitoring

17. Proactive Notification

18. WIP Limits

19. Visualising Work

Cultural Capabilities

20. Westrum Organizational Culture

21. Supporting Learning

22. Collaboration among Teams

23. Job Satisfaction

24. Transformational Leadership

Following the above practices will help your company to:

• Delivering goods and services to their customers
• Engagement with the market to detect and understand customer demand.
• Anticipation of compliance and regulatory changes that impact systems.
• Respond to potential risks, such as security threats or changes in the economy.

Maturity Models Vs Capability Model

The focus of you company should be to continuously improve these capabilities, not reach and zenith and maintain it. Stale processes decrease in effectiveness by entropy. Hence why there is a focus on a capabilities model, rather a maturity model

Avoid maturity models because they seek to achieve a state and then abandon efforts to continuous improve that state.

Maturity models are often prescribed formulas to reach a state. Capability models on the other hand focus on outcomes.

Adopting DevOps culture and capabilities works

In the 2017 state of DevOps report the comparison was stark, the difference between low performers and high performers was the following:

• 46 times more frequent code deployments.
• 440 times faster lead time from commit to deploy
• 170 times faster mean time to recover from downtime.
• 5 times lower change failure rate

Measuring Software Delivery Performance

A successful measure of performance should have two characteristics

1. Focus on global outcomes to ensure teams aren’t pitted against each other. For example, rewarding developers for throughput and operations for stability would not be a global outcome reward.
2. Focus on outcomes, not output. Being busy for the sake of being busy does not help achieve organisational goals.

The measures of delivering performance that were decided upon the above criteria were the following four:

1. Delivery lead time — Shorter is better because they enable faster feedback and easier to correct smaller changes. This is measured as the time from code committed to code successfully deployed to production.
2. Deployment Frequency — It’s an easy way to measure batch size of a change.
3. Time to Restore Service — The shorter, the better, it measures the resilience of architecture and infrastructure
4. Change fail rate — it measures how good the testing framework is.

Culture

Modeling and Measuring Culture

Three levels of organisational culture:

1. Basic assumptions — these are formed over time as members of the group make sense of the relationships, events and activities. Visible but difficult to articulate.
2. Values — visible to group members as these can be discussed and debated by those aware of them. Netflix provides a good example of this.
3. Artifacts — this include written mission statements or creeds, formal procedures, or even heroes or rituals.

The book focuses on understanding the values of an organisation by using the Westrum model:

The above model was very accurate in describing high, creative performers, those who felt on the right:

1. Collaborated more effectively with a higher level of trust, up and down the hierarchy.
2. Generative culture emphasises the mission and hence people will put aside personal and departmental issues to achieve the company’s goal.
3. Encourages a level playing field in which hierarchy plays less of a role.

A generative culture improves software delivery performance and organisational performance. It’s also key to find root solutions to problems, rather than punishing someone and throwing the issue under the rug.

A study performed by Google found that these characteristics that are common in generative organisations create the best teams:

source: https://rework.withgoogle.com/blog/five-keys-to-a-successful-google-team/

How to change culture?

John Shook, Lean manufacturing expert found that the best way to change culture is not by trying to shape their thinking, but instead change how they behave — what they do.

Implementing technical practices like continuous delivery or Lean Management would effectively force everyone to change the way they think.

Technical Practices

At the time the Agile manifesto was published, Extreme Programming (XP) was the most popular implementation of it.

XP describes a set of technical practices, such as TDD and continuous integration. The Continuous Delivery book (by Humble and Farley, 2010) also emphasises this technical practices as an enabler of more frequent, higher quality, lower risk frequent releases.

Other Agile adoptions, most prominently Scrum, treat technical practices as secondary compared to the management practices they emphasise. The research showed that technical practices played a vital role in achieving high performance outcomes.

Defining Continuous Delivery (CD)

It’s a set of capabilities that enables to get all kinds of changes into production safely, quickly and sustainably. There are five principles at the heart of CD:

1. Build quality in — enabling detect defects quickly so they can be fixed straight away when they are cheap to detect and resolve.
2. Work in small batches — small changes are faster to deploy and revert. It’s a lot less likely to break things if you release small. Also the customer gets the benefits quicker.
3. Computers perform repetitive tasks; people solve problems — automation reduces the possibility to make manual mistakes and it delivers results a lot faster.
4. Relentlessly pursue continuous improvement — constantly improving your process reduces the chance that it will become worse by entropy. It also keeps you on the cutting edge.
5. Everyone is responsible — By sharing responsibility all hands are on deck and we don’t throw the blame on others and create bottlenecks.

Required foundations for CD

• Comprehensive configuration management: environments should be easy to recreate at a push of a button and all this information should be in version control. Builds can recreate this environments for testing.
• Continuous Integration (CI): Keep branches short lived, a day or less, and integrate them into trunk frequently.
• Continuous Testing: Automated tests should be run against every step of the release process.

Impact of CD

Teams that did well in continuous delivery achieved the following outcomes:

• Strong identification with the organisation they worked for.
• Higher levels of software delivery performance (lead time, deploy frequency, time to restore service)
• Lower levels of deployment pain.
• Reduced team burnout. Effects improve culture.

Impacts of Continuous Delivery (source)

Driving Factors of CD

• Version control — Interestingly, keeping configuration management in version control helped performance more than keeping code in version control.

Test Automation

Having automated tests that are reliable is a key factor in performance, rather obviously.

What’s less obvious is that having automated tests that are created and maintained by a QA or an outsourced party do NOT improve performance, a high predictor of company performance is when developers themselves are thorough creating automated tests for their code. The main two reasons for that are that:

1. The code becomes more testable when developers write their own tests. Test driven development (TDD) helps a lot because it forces developers to create more testable designs.
2. When developers are responsible for automated tests, they care more about them and will invest more effort into maintaining them and fixing them.

None of this means that testers are redundant, QA engineers are of paramount importance in performing and designing manual exploratory, usability and acceptance testing, and helping evolve the suite of automated tests with the developers.

Test Data Management

Successful teams had adequate test data to run their fully automated test suites and could acquire it on demand. As opposed to having to copy entire real data sets.

Trunk Based Development (TBD)

Teams who practiced TBD correlated with much higher delivery performance these teams had:

• Fewer than three active branches at a time.
• Branches had very short lifetimes (less than a day) before being merged into trunk.
• No code freezes or stabilisation periods.

Results were independent of team size, organisation size or industry.

Information Security

High performing teams were more likely to incorporate information security into the delivery process.

Drivers of Continuous Delivery (source)

Architecture

Bad architecture, or architecture that’s not loosely coupled will hinder your capacity to delivery software quickly.

Types of Systems and Delivery Performance

• Greenfield: new systems that have not yet been released
• Systems of engagement: used directly by end users.
• Systems of record: used to store business critical information where data consistency and integrity is of paramount importance.
• Custom software developed by another company
• Custom software developed in-house.
• Packaged, commercial off-the-shelf software.
• Embedded software that runs on a manufactured hardware device.
• Software with a user-installed component (including mobile apps)
• Non-mainframe software that runs on servers operated by another company.
• Non-mainframe software that runs on our own servers.
• Mainframe software.

Low performers are most likely using custom software by other company. Also likely to use mainframe systems. In the rest of the cases there was no correlation between system type and performance.

Hence why it is important to focus on the architecture, rather than the type of systems used.

Focus on deployability and testability

The following architectural characteristics are conducing to high performance:

• To be able to do most of the testing without requiring an integrated environment.
• To be able to deploy or release the application independently of the other services and applications it depends on

The greatest contributors to achieving CD based on the architectural model was based on whether teams could:

• Make large-scale changes to the design of their system without the permission of somebody outside the team.
• Make large-scale changes to the design of their system without depending on other teams to make changes in their systems or creating significant work for other teams.
• Complete their work without communicating and coordinating with people outside the team.
• Deploy and release their service on demand, regardless of other services it depends upon.
• Do most of the testing on demand, without requiring an integrated environment.
• Perform deployments during normal business hours.

Architectural approaches that may aid achieve this strategy include bounded contexts and APIs as a way to decouple large domains into smaller, more loosely coupled units.

A loosely coupled architecture enables scaling

Allow teams to choose their own tools

Teams should choose the best tools for their particular needs, this allows them to experiment with new approaches and paradigms to solve problems.

Teams should also be encouraged and allow to pick and perfect their own security solutions.

However there are some areas where standardisation is best, for example around architecture and configuration of infrastructure.

Architects should focus on engineers and outcomes, not tools or technologies

Architects should focus on the team’s needs and their requirements when choosing tools, not pick something cause it’s new and flashy.

Management Practices for Software

Lean Management Practices

1. Limiting Work in Progress (WIP)
2. Creating and maintaining visual displays showing key quality and productivity metrics and the current status of work, making them visible to both engineers and leaders and aligning these metrics with operational goals.
3. Using data from application performance and infrastructure monitoring tools to make business decisions on a daily basis.

Limiting WIP on its own does not strongly predict delivery performance, it’s only when they are combined with the use of visual displays and have a feedback loop from production monitoring tools back to delivery teams or the business that there is a strong effect.

Key takeaways: improve visibility through kanban, storyboards, failures, defect rates, etc.

Implement a lightweight change management process

Teams that had no approval process or used peer review achieved higher delivery performance. Teams that required external approvals exhibited lowest performance.

In addition external approvals do NOT improve the stability of a system and they slow things down a lot.

Peer approval and deployment pipeline to detect and reject bad changes is the best.

Lean Product Development Practices

Ask whether:

1. The teams slice up products and features into small batches that can be completed and released frequently, including the use of MVPs
2. Teams have a good understanding and visibility of the flow of work from business to customer, including the status.
3. Organisations actively seek customer feedback and incorporate it into the design of their products.
4. Teams have the authority to create and change specifications as part of the development process without requiring approval.

Leaders vs Managers

Transformational Leadership

Transformational leadership means leaders inspiring and motivating followers to achieve higher performance by appealing to their values and sense of purpose, facilitating wide-scale organisational change.

These leaders encourage their teams to work towards a common goal through their vision, values, communication, example-setting and their evident caring about their follower’s personal needs.

Transformational leadership is essential for:

• Establishing and supporting generative and high trust cultural norms
• Creating technologies that enable developer productivity, reducing code deployment lead times and supporting more reliable infrastructures.
• Supporting team experimentation, innovation and quality engineering.
• Working across organisational silos to achieve strategic alignment.

The Five characteristics of transformational leadership:

1. Vision: clear understanding of where the organisation is going and where it should be in five years.
2. Inspirational communication: communicates in a way that inspires and motivates, even in uncertain or changing environment.
3. Intellectual communication: Challenges followers to think about problems in new ways.
4. Supportive leadership: Demonstrates care and consideration of followers personal needs and feelings.
5. Personal recognition: Genuinely praises and acknowledges achievements of goals and improvement in work quality, personally compliments others on their work.

Questions to ask about leaders:

Vision

• Do they have a clear understanding of there we are going?
• Do they have a clear sense of where he wants the team to be in five years?
• Do they have a clear idea of where the organisation is going?

Inspirational Communication

• Do they say things that make employees proud to be part of the organisation?
• Do they say positive things about the work unit?
• Do they encourage people to see changing environments as situations full of opportunities?

Intellectual Stimulation

• Do they challenge me to think about old problems in new ways?
• Do they have ideas that force me to rethink some things that I have never questioned before?
• Have they challenged me to rethink some of my basic assumptions about my work?

Supportive Leadership

• Do they consider my personal feelings before acting?
• Do they behave in a manner which is thoughtful of my personal needs?
• Do they see the interests of employees are given due consideration?

Personal Recognition

• Do they commend me when I do a better than average job?
• Do they acknowledge improvement in my quality of work?
• Do they personally compliment me when I do outstanding work?

The characteristics above were found to be highly correlated of software delivery performance.

High performing teams reported having leaders with the strongest behaviours across all dimensions and vice versa. These companies also were most likely to have leaders that promoted the company.

The Role of Managers

Managers play a critical role in connecting the strategic objectives of the business to the work their teams do.

Managers should create a safe environment, invest in the capabilities of their people and remove obstacles.

Managers should enable DevOps Specific practices and should invest in the following initiatives:

• Ensure existing resources are made available and accessible to everyone in the organisation. Create space for learning and improving.
• Have a dedicated training budget and make sure people know about it. Provide the resources for training and give people latitude to chose what they wish.
• Encourage staff to attend technical conferences at least once a year and summarise what they learned for the entire team.
• Setup internal hack days, where cross-functional teams can get together to work on a project.
• Encourage teams to organise internal “yak days” to address technical debt.
• Hold internal DevOps mini-conferences.
• Give staff dedicated time to experiment with new tools and technologies. Allocate budge for special projects.

Tips to improve culture and support your teams

Enable cross functional collaboration by:

• Build support with your counterparts on other teams.
• Encouraging practitioners to move between departments.
• Actively seeking, encouraging and rewarding work that facilitates collaboration.

Help create a climate of learning by:

• Creating a training budget and advocating for it internally.
• Ensuring that your team has the resources to engage in informal learning and the space to explore ideas.
• Make it safe to fail with blameless postmortems and don’t punish people for failing.
• Create opportunities and spaces to share information.
• Encourage learning and innovation by having demo days and forums.

Make Effective use of tools:

• Make sure your teams can choose their tools.
• Make monitoring a priority.

Below is a chart of transformation practices you can use as cheatsheet:

https://devops-research.com/assets/transformation_practices.pdf