How to Measure the Effectiveness of Your DevOps Program

In the wake of the age of mobility, and increasing demands for high-quality web and mobile applications, DevOps is fast becoming the most reliable and preferred strategy for most organizations. It’s cross-functional collaboration and speed are making it more like a go-to strategy, as it enables quick delivery of software solutions, which is precisely what is needed in the current business scenario. Moreover, as businesses are in the process of digital transformation around the world, an agile environment is only a necessity, as a lot of business aspects linked to the transformation process like growth, customer loyalty, and satisfaction, competition differentiation etc., need to be taken care of well. DevOps helps in creating such a responsive IT environment, enabling organizations in rapid development and deployment of high-end software solutions.

But how far is DevOps successful? How can an organization judge for itself whether it’s DevOps initiative was, or will be successful? Sure, a lot of organizations are following it, but given the transformational scenario that most organizations are in, it could be difficult to measure its success, because DevOps by itself is not exactly a formal framework and it doesn’t provide a lot of guidance. Organizations simply have to learn into it.

There are some metrics or elements that accurately help in measuring the success of DevOps. Most organizations make the mistake of measuring a number of elements, which often may not be necessary and retreat to ones that can be easily collected. But, the issue is that some of these metrics that matter for DevOps may not be as familiar to organizations. For example, the speed of deployment, rate of change and the like are metrics that are only applicable to DevOps, which in turn is a comparatively new concept for organizations.
So what are the metrics that should be considered for DevOps?

The elements that matter

Importantly, we need to consider people-centric metrics, and process and technology centric metrics as well for DevOps. Out of these, people-centric metrics are probably the most difficult to collect, but often prove to be the most useful. They can actually be one of the most powerful influencers on a DevOps program. Hence, internal metrics like staff training and retention rates should be strongly considered.
When it comes to

When it comes to process metrics, we need elements that help to measure the effectiveness of interlinked processes throughout the delivery process. It helps to see if the collaboration is effective. It also helps to identify deficiencies within the processes that need more work.

Technology metrics are those such as uptime and capacity to support expected web traffic, which basically help in reviewing the technologies used in the DevOps process. It also includes insights derived from failures or errors like failed releases, code defects and the like.

Another important thing to note while determining metrics is to sustain a comprehensive or holistic approach. Sticking to just one or two aspects of measurement like operational or developmental metrics, may not provide the required results. As a matter of fact, there are chances of it having a negative impact on behavioral improvements in the organization.

To start off, here are a few dimensions which can be used to measure the effectiveness of DevOps:

• Collaboration and sharing – This literally forms the base of a DevOps program and is hence the most important measure. They help in judging the acceptance or resistance to the program, on an ongoing basis, which is a valuable indicator of the effectiveness of DevOps. As mentioned before, some of the metrics in this dimension might be easier to collect such as staff retention rates, training, and turnover, while others like employee morale might be more difficult. Another aspect to look into here, is how metrics in other dimensions are affecting elements in this dimension. For example, how far are MTTR (Mean Time To Repair) changes affecting employee morale, retention, absenteeism and the like. Automated surveys and other means to get employee feedback are other areas that may be considered for this dimension.

• Efficiency – This dimension mainly focuses on developmental and operational aspects. The capacities and capabilities. Moving from the traditional ratios like server to admin, businesses are now using customer-centric ratios like FTE (Hours worked by a Full-Time Employee) to customers. This value is expected to increase in the coming years, as more enterprises are now moving to automation and the cloud. Other metrics such as examining costs on an application basis and cost of release are good measures in improving data center efficiencies.
• Quality – This dimension focuses on elements related to service delivery. For example, metrics like percentage of applications rolled-back because of code defects. Now this metric could initially be high for organizations that have just begun on their DevOps initiative. This is probably due to extra time required for the purpose of making the new processes effective, and other related things. These metrics might give other useful insights when combined with other indicators. For example, the rate of rollbacks when combined with the change volume indicator, could provide more important insights.
  These are some other metrics in this dimension:
  Cycle time – time required to complete a stage or several stages within a project
  MTTR – average time taken to restore a service or repair a defective part

• Business value – This dimension is focused on external things – like the impact of DevOps on meeting business goals. It includes elements like customer value or loyalty, time to market and the like. The lead time too provides businesses with an analogous metric that helps to know how well DevOps is meeting the need to deliver high-quality software services fast. This is specifically important as a long lead time may mean more defects in code and issues in testing.

The Net Promoter Score (NPS) is another important metric, which is a simple method to measure customer loyalty. Even though this measure has been traditionally used for marketing purposes for a long time, customer loyalty is also affected by the fast and timely delivery of software services through high-quality web and mobile apps.
All these metrics contribute towards analyzing the effectiveness and success of DevOps. Keeping track of these, can help an organization in deciding whether to continue with the program or do the needful to make it further effective.

Image credits: Prashant Arora’s blog