Shift Left, Measure Right: Calculating DevSecOps Success


Earlier and more frequent inclusion of security in the software development lifecycle and throughout the CI/CD pipeline, aka “shifting security left,” has become a widely accepted best practice for DevSecOps teams. With organizations spending millions on people, processes and tooling, how do we know if shifting left is delivering results?

If you measure success based on the number of security tests and tools in your CI/CD pipeline, the number of vulnerabilities you detect, or mean time to remediate, you’re missing the bigger picture. It’s not about the number of vulnerabilities you detect or how quickly you fix them. In the end, detecting and fixing is not equivalent to securing. Unless you fix the vulnerabilities and threats that are exploitable, your applications will remain insecure once deployed.

This guide highlights some ways to think about and measure the effectiveness of your DevSecOps process and security posture.

But First….DevOps and DevSecOps: What’s the Difference?

DevOps was the starting point for what is now DevSecOps. DevOps first focused on integrating teams, CI/CD tools, and processes to speed software development and delivery. DevSecOps emerged soon after when everyone realized that they needed to embed security into their CI/CD process.

DORA and More

Google’s DevOps research assessment (DORA) team tracks the following five metrics to determine DevOps success and maturity: deployment frequency, lead time for code changes, mean time to recovery, change failure rate, and reliability. Over time, DORA metrics offer insight into how mature DecOps organizations have become. But even DORA has no real metrics that measure the effectiveness of security within DevSecOps.

Step 1: Map Your Security Gates and Tools

Before you can measure the effectiveness of DevSecOps, you need to map out the security checkpoints that you’ve established in your CI/CD pipeline. With each stage, list the tools and processes that you use.

PlanCode BuildTestReleaseMeasure
Threat Modeling
Code Reviews
Unit Testing
Security Architecture Reviews

Step 2: Find the KPIs that Matter

The next step is to find the KPIs to measure the effectiveness of your tools and gates. Many industry leaders use DORA metrics, but there are still too many vulnerabilities to possibly fix, too many signals and alerts that lack business risk context, and too much toil.

Here are some KPIs that can give you a better understanding of your level of DevSecOps success. In this context, a critical risk is any threat that is exploitable and has access to sensitive data.

Critical Risk Volume# of critical risks detected in pre-productionTo understand the volume critical risks found in pre-prod. Requirement for Critical Risk Escape Rate KPI.
Critical Risk Remediation Queue # of critical risk ticketsTo understand the stress levels and workload placed on the developers that need to fix these issues.
Known Risk Acceptance # of known risks not remediated (accepted) and pushed to production with reasonTo understand how many risks you are accepting and why.Requirement for Known/Accepted Risk Severity Evolution KPI.
Critical Risk Volume # of critical risks detected in deployed appsTo compare to the volume of critical risks found in pre-prod.Requirement for Critical Risk Escape Rate KPI.
Critical Risk Escape RateRatio: # of critical risks found in pre-production to # found in productionTo assess effectiveness of pre-production security.
Critical Risks Per DeploymentAverage # of critical risks for each deploymentTo benchmark expected critical risks.
Mean Time Between Critical RisksAverage time between critical risk detectionTo understand how successful your team is at preventing or reducing future critical risks.
Known/Accepted Risk Severity Evolution# of issues pushed to production as an accepted risk that evolved into critical risksTo understand how real life production context changes risk severity.

The final metric, Attack Surface Coverage, is a maturity assessment rather than a KPI. Here are the three maturity levels of Attack Surface Coverage:

  • Level 1: 33% coverage (Infrastructure/cloud)
  • Level 2: 66% coverage (Infrastructure/cloud + network)
  • Level 3: 99% coverage (infrastructure/cloud + network + applications)

Step 3: Assess the Team Experience

DevSecOps is both a technical and cultural endeavor, so it’s important to take into account the sentiment of key stakeholders. You can gauge experience ratings in many different ways. Depending on the size and geographic distribution of your team, you might choose an informal Slack poll or a short survey. Whatever form your survey takes, here are some questions worth asking.

Developer Experience

  • Do you have the context and understanding needed to fix critical risks?
  • How much time do you spend on manual code reviews and security checklists?
  • What percentage of your time is spent on fixing security issues? Do you classify this as toil or meaningful work?
  • Do you have adequate time to innovate?

Security Experience

  • Do your tools provide clear and meaningful signals?
  • Do you understand how risk is prioritized in your organization?
  • How much time do you spend on manual code reviews and security checklists?
  • Do you have time to take on proactive security projects?

SRE/Operations Experience

  • How resilient are your applications? How many single points of failure are present?
  • How are you managing your application inventory and dependencies?
  • Do you have the necessary tools and processes to quickly pinpoint the root cause of an outage and restore service?

CISO Experience

  • How many application threats exist right now in production that are exploitable and connect to sensitive data?
  • Is the security posture of your applications in production getting better or worse?
  • Do you have the tools and data that you need to answer these questions?

DevSecOps shouldn’t just be measured on deployment frequency, agility, and speed. Rather, it should be about understanding what is making you more secure.To learn about how Bionic ASPM can provide vital data about your application security posture to help you measure DevSecOps excellence, book your demo today

Bionic Named 2021 Cool Vendor by Gartner