Introduction
Earlier and more frequent inclusion of security in the software development lifecycle and throughout the CI/CD pipeline, aka “shifting security left,” has become a widely accepted best practice for DevSecOps teams. With organizations spending millions on people, processes and tooling, how do we know if shifting left is delivering results?
If you measure success based on the number of security tests and tools in your CI/CD pipeline, the number of vulnerabilities you detect, or mean time to remediate, you’re missing the bigger picture. It’s not about the number of vulnerabilities you detect or how quickly you fix them. In the end, detecting and fixing is not equivalent to securing. Unless you fix the vulnerabilities and threats that are exploitable, your applications will remain insecure once deployed.
This guide highlights some ways to think about and measure the effectiveness of your DevSecOps process and security posture.
But First….DevOps and DevSecOps: What’s the Difference?
DevOps was the starting point for what is now DevSecOps. DevOps first focused on integrating teams, CI/CD tools, and processes to speed software development and delivery. DevSecOps emerged soon after when everyone realized that they needed to embed security into their CI/CD process.
DORA and More
Google’s DevOps research assessment (DORA) team tracks the following five metrics to determine DevOps success and maturity: deployment frequency, lead time for code changes, mean time to recovery, change failure rate, and reliability. Over time, DORA metrics offer insight into how mature DecOps organizations have become. But even DORA has no real metrics that measure the effectiveness of security within DevSecOps.
Step 1: Map Your Security Gates and Tools
Before you can measure the effectiveness of DevSecOps, you need to map out the security checkpoints that you’ve established in your CI/CD pipeline. With each stage, list the tools and processes that you use.
| Pre-production | Production | ||||
| Plan | Code | Build | Test | Release | Measure |
| Threat Modeling | |||||
| Code Reviews | |||||
| SCA | |||||
| Unit Testing | |||||
| SAST | |||||
| Security Architecture Reviews | |||||
| IAST | |||||
| DAST/Pentesting | |||||
| ASPM | |||||
Step 2: Find the KPIs that Matter
The next step is to find the KPIs to measure the effectiveness of your tools and gates. Many industry leaders use DORA metrics, but there are still too many vulnerabilities to possibly fix, too many signals and alerts that lack business risk context, and too much toil.
Here are some KPIs that can give you a better understanding of your level of DevSecOps success. In this context, a critical risk is any threat that is exploitable and has access to sensitive data.
| Pre-production | ||
| KPI | Description | Purpose |
| Critical Risk Volume | # of critical risks detected in pre-production | To understand the volume critical risks found in pre-prod. Requirement for Critical Risk Escape Rate KPI. |
| Critical Risk Remediation Queue | # of critical risk tickets | To understand the stress levels and workload placed on the developers that need to fix these issues. |
| Known Risk Acceptance | # of known risks not remediated (accepted) and pushed to production with reason | To understand how many risks you are accepting and why.Requirement for Known/Accepted Risk Severity Evolution KPI. |
| Production | ||
| KPI | Description | Purpose |
| Critical Risk Volume | # of critical risks detected in deployed apps | To compare to the volume of critical risks found in pre-prod.Requirement for Critical Risk Escape Rate KPI. |
| Critical Risk Escape Rate | Ratio: # of critical risks found in pre-production to # found in production | To assess effectiveness of pre-production security. |
| Critical Risks Per Deployment | Average # of critical risks for each deployment | To benchmark expected critical risks. |
| Mean Time Between Critical Risks | Average time between critical risk detection | To understand how successful your team is at preventing or reducing future critical risks. |
| Known/Accepted Risk Severity Evolution | # of issues pushed to production as an accepted risk that evolved into critical risks | To understand how real life production context changes risk severity. |
The final metric, Attack Surface Coverage, is a maturity assessment rather than a KPI. Here are the three maturity levels of Attack Surface Coverage:
- Level 1: 33% coverage (Infrastructure/cloud)
- Level 2: 66% coverage (Infrastructure/cloud + network)
- Level 3: 99% coverage (infrastructure/cloud + network + applications)
Step 3: Assess the Team Experience
DevSecOps is both a technical and cultural endeavor, so it’s important to take into account the sentiment of key stakeholders. You can gauge experience ratings in many different ways. Depending on the size and geographic distribution of your team, you might choose an informal Slack poll or a short survey. Whatever form your survey takes, here are some questions worth asking.
Developer Experience
- Do you have the context and understanding needed to fix critical risks?
- How much time do you spend on manual code reviews and security checklists?
- What percentage of your time is spent on fixing security issues? Do you classify this as toil or meaningful work?
- Do you have adequate time to innovate?
Security Experience
- Do your tools provide clear and meaningful signals?
- Do you understand how risk is prioritized in your organization?
- How much time do you spend on manual code reviews and security checklists?
- Do you have time to take on proactive security projects?
SRE/Operations Experience
- How resilient are your applications? How many single points of failure are present?
- How are you managing your application inventory and dependencies?
- Do you have the necessary tools and processes to quickly pinpoint the root cause of an outage and restore service?
CISO Experience
- How many application threats exist right now in production that are exploitable and connect to sensitive data?
- Is the security posture of your applications in production getting better or worse?
- Do you have the tools and data that you need to answer these questions?
DevSecOps shouldn’t just be measured on deployment frequency, agility, and speed. Rather, it should be about understanding what is making you more secure.
To learn about how Bionic ASPM can provide vital data about your application security posture to help you measure DevSecOps excellence, book your demo today