The State of DevOps Report 2020 is out very recently, after a long wait. This year it is presented by Puppet and CircleCI and sponsored by Servicenow, Sysdig, Armory, New Relic, and Splunk.
It has some interesting observations and very relevant to the current day. The following are the key points mentioned in the report:
- Scaling DevOps practices with internal platforms
- Change Management in the DevOps Era
- Security Integration
Scaling DevOps practices with internal platforms
Internal Platform usage is widespread:
- Sixty-three percent of respondents say their company has at least one self-service internal platform.
- Of those with internal platforms, 60 percent have between two and four.
- Almost a third of respondents have 25 to 50 percent of developers using an internal platform
It mentions that high DevOps evolution correlates strongly with the high use of internal platforms. Highly evolved firms are six times as likely to report high use of internal platforms as firms at a low level of DevOps evolution.
It also mentions that a product mindset is a key to scaling DevOps and the platform. Highly evolved firms are nearly twice as likely to be highly product-oriented as firms in the middle of their DevOps evolution.
Higher levels of DevOps evolution mean more self-service offerings for developers. Highly evolved firms offer a wide range of self-service capabilities, including:
- CI/CD workflows
- Internal Infrastructure
- Public Cloud Infrastructure
- Development environments
- Monitoring and alerting
- Deployment patterns
- Database provisioning
- Audit logging
Top Challenges to providing and Internal Platform are:
- Lack of time
- Lack of Standardization
- Lack of technical skills
It is interesting to see the Product approach and the use of a platform to improve DevOps adoption.
Change Management in the DevOps Era
- Change management effectiveness increases as organizations evolve their DevOps practices
- The most effective change management is achieved by firms that emphasize:
- A high degree of testing and deployment automation
- A high degree of automated risk mitigation
- Less rigid and much less manual approval processes
- Writing changes in code
- Allowing employees more scope to influence change management
- DevOps Process and culture
- Highly orthodox approval processes make change management process inefficient
- Automation makes people more confident their change management is effective
- Firms that give people a say in the change management process have better change management
Top Challenges to automating the change management process are:
- Incomplete test coverage
- Organizational mindset
- Tightly coupled application architecture
Integrating security fully into the software delivery process improves your ability to quickly remediate critical vulnerabilities. The survey shows:
- Among companies with full security integration, 45% can remediate critical vulnerabilities within a day
- Just 25% of those with low-security integration can remediate within a day
Integrating security fully into the software delivery process leads to providing self-service for security and compliance validation.
The report also talks about the DevOps Evolution model as below:
Platform Model – A newish approach to Scaling DevOps
It is found that many organizations are not able to scale after a certain point, mostly when they are in their mid-way through the journey. The platform approach works well when multiple products need to be managed.
While having multiple end-to-end product teams doesn’t scale well across large complex environments, a platform model is defined by a clear purpose, boundaries, and responsibilities do. A platform, built with the customer in mind, can significantly reduce the toil and overhead of individual product teams.
Broadly, the platform team provides the infrastructure, environments, deployment pipelines, and other internal services that enable internal customers, who usually are the application development teams to build, deploy, and run their application.
The platform model is best suited to cloud-native environments but is also appropriate for many other types of architecture – modern to legacy. The primary benefits are:
- Application teams can be more efficient
- Improved Governance
- An end to context-switching
- Continuous Infrastructure improvement
Platform use and DevOps Evolution
The survey found a strong relationship between DevOps evolution and the use of internal platforms. Highly evolved firms are almost twice as likely as mid-level organizations to report high usage of internal platforms and are six times more likely to report high usage than low-level organizations.
The survey talks about looking at Platform as a Product. “The platform team isn’t meant to be an ivory-tower-siloed cadre that prescribes all architectures, functionality, tooling, and more. The platform team’s job is to provide core capabilities that make it easier for their customers — that is, other teams — to get work done and achieve their goals, as well as those of the overall business. You have to make the platform a compelling option. Application teams should want to use the platform because it’s easier and more cost-efficient than building and maintaining their own”.
The Project to Product Approach concept is being extended here to Platform. The Platform team is there as long as Platform is there and has a continual improvement to the platform.
What makes Platform a Product instead of a Project?
- Think Self-Service and API first
- Start locally but build globally
- Focus on developer experience and flow
- Continuously invest in the product
As per the survey, DevOps evolution and platform evolution go together. The graph shows how platform offerings change as organizations progress through their DevOps evolution.
The following graph also shows the Platform Team responsibilities:
Challenges for internal platforms as per the respondents:
A case study by CircleCI is also there. Interesting to note how they use Platform and how they measure the success of the Platform.
Here are some metrics we make available for our internal teams at CircleCI:
- 28-day rolling and/or seven-day rolling uptime
- Incident frequency
- Mean investigation time per incident (e.g., how long it takes to find out what went wrong)
- Percentage of services or capabilities with vulnerability-patching SLAs
- Cost per work unit
- Developer throughput rate
- Deployment rate
- Rollback rate
- Conformance metrics (how close service is to using “paved roads,” i.e., standards provided by the platform team)
Change Management in the new way of working is the next important thing that this survey talks about. Here is a list of changes that happened from earlier way to DevOps ways:
The survey tried to find the correlation between change management effectiveness with DevOps evolution. 3 Dimensions were looked into:
- Implementation Success
- Level of Efficiency
- Performance Sentiment
The survey respondents were mapped to 4 clusters:
- Operationally Mature
- Engineering Driven
- Governance Focused
- Ad Hoc
The following graph shows the result:
The survey talked about 3 types of Change Approval Processes:
Orthodox change approval is based on strict adherence to established practices:
- Changes are approved by a committee (e.g., a change approval board).
- Approval is required from multiple levels of management.
- Changes can be made only in predefined windows.
- The person requesting the change cannot implement the change
(separation of duties).
Adaptive change approval is based on input from teams that are close to the work. We call this “responsible autonomy.”
- Changes are approved by the team implementing the change.
- Post-implementation reviews identify opportunities for improvement.
- Pre-approval is based on proof from the delivery team that the change can be made safely.
Evasion of the change approval process happens in a couple of ways:
- Changes are approved without proper consideration (i.e. rubber-stamped).
- Team members explicitly bypass the change management process and there are no consequences.
An interesting example is given on the sidebar about how Regulations are misinterpreted. The example shows the SOX 2 compliance, how it is misinterpreted, and how it can be handled having the same control objective with different controls.
Change written in Code is the way to go. Change as Code is explained as below:
When your changes are written in code, they may have the same properties as changes written in prose. But because they follow coding practices, your changes can be authored, tested, reviewed, and deployed like code.
The real power here is that your changes can be subjected to any validation techniques that are available for code. Plus, you can roll forward to a previously stable implementation if the change does not go as planned.
Changes implemented in code also have the advantage of sometimes escaping the traditional — and slow — change management processes that most companies use. That’s because they follow development deployment guidelines rather than an ITIL-based review process.
Going through the development process instead of the change-review process usually results in greater velocity, and often more consistency.
Common changes produced in code include changes through infrastructure automation tools, delivering new capabilities in software, or updating software packages.
It mentions “TOIL” from Site Reliability Engineering and also focuses on automation and TOIL reduction.
The report talks about 4 approaches to change management. The report plotted the four change management approaches on a matrix (see below). The x-axis represents orthodox approvals, and the y-axis represents automation, meaning a combination of automated tests, automated deployment, and automated risk mitigation.
The following is mentioned to answer “What drives Change Management Effectiveness?”
- Orthodox approvals make you less efficient
- Automation gives teams confidence in change management
- Giving people agency over the process results in the higher effectiveness
The graphs below show the results for each.
The top challenges reported by all groups for implementing change management are:
- Incomplete Test Coverage
- Organizational Mindset
- Tightly coupled application architecture
Top challenges by change management approach:
- Operationally mature organizations have a lower risk tolerance
- Both engineering-driven and operationally mature organizations feel constrained by their application architecture
- Governance-focused organizations lack trust across functional teams
- Ad hoc organizations cited a lack of skills as an inhibitor
principles to be applied to change management are:
- Break down silos and build empathy
- Create feedback loops
- Measure the impact of your new approach
This year 14% of respondents reported working in an infosec department. It was seen that security integration strongly correlated with the ability to quickly remediate critical vulnerabilities.
- Those with low integration, 25% can remediate vulnerabilities within 1 day compared to 45% of those with full security integration
- Self-service offering of security and compliance validation is positively related to the level of security integration. Those with full security integration are over twice as likely as those with no security integration to offer security and compliance validation as a self-service capability.
My take – Points that interest me
The following points strike the most:
- Trust and Empowerment of the team. The self-organized team, given full autonomy and working collaboratively, breaking down the silos of the whole enterprise is important for doing better. This is clear in the points related to change management. Reference to ITIL4 on how it has adapted is there. ITIL4 making Change authority decentralized and removing the need for CAB to be mandatory shows which way it is going. Even Security being incorporated as part of the whole lifecycle shows the trust, empowerment, and collaboration
- Project to Product Approach being the way to go along with the Platform approach is an interesting one. We need to carry on with the team to get the benefit of trust that is built and not start over again once released from a project. This also helps in cross-skilling and also better delivery and outcome
- The mention of misalignment of incentives is important. We need to look at value creation and that happens when users use and get the intended benefit. The whole organization design needs to be worked out including performance appraisal to make it individual to team based and output to outcome-based. Everyone works towards one single shared objective. Project to product approach also helps in achieving it better
- What I also see is the focus towards looking at a holistic approach both from the IT System side and from the Domain side. Everyone must understand the IT System and Domain holistically. People need to be experienced to be in the team
- Finally, Security is a primary consideration. DevSecOps seems to be the way to go. Security is everyone’s responsibility. Security professionals and GRC SMEs need to come as advisors to the team and train them on control objectives and other knowledge needed. They will specify the control objectives and let the team implement the controls. Security is part of the flow at every stage and not a separate project/program
- This report brings in a lot of interesting points mainly focusing on Scaling DevOps. Many have already started the DevOps journey. Now scaling is the need of the hour. The other point that strikes is that it is talking about “Evolution”. This is an important point. We need to implement DevOps, SRE as evolution, smaller improvement done repetitively and in shorter time intervals, adapting to change instead of “Big Bang Transformation”
~ Dr. Niladri Choudhuri