Menu Icon
< back to main
 • 5 min read

Pen Test Metrics 101: Benefits and Types

This post is the first of a series in which I'll cover how to use pen test data in security metrics to analyze and improve your application…

Pen Test Metrics 101: Benefits and Types
Caroline Wong
Caroline Wong

Caroline Wong is the Chief Strategy Officer at Cobalt. As CSO, Caroline leads the Security, Community, and People teams at Cobalt. She brings a proven background in communications, cybersecurity, and experience delivering global programs to the role.

Want to see the platform in action?
get a demoArrow Right
Want to see the platform in action?
get a demoArrow Right

This post is the first of a series in which I’ll cover how to use pen test data in security metrics to analyze and improve your application security program.

Metrics 1

AppSec pen tests are time-limited, fixed price, manual application security tests performed by top security researchers. Organizations typically have a fixed cost to work with when designing a pen test strategy, and want to use it to optimize quality (talent, results) and coverage (across an application portfolio and within an application).

Benefits of AppSec Metrics

It’s been said that “if you can’t measure it, you can’t manage it.” While it turns out that Peter Drucker never actually said that, it is indisputable that measuring results and performance is crucial to an organization’s effectiveness, and this definitely applies to application security.

Here are a few benefits of using metrics to evaluate an application security program:

  1. Measurement provides visibility.

  2. Measurement educates and provides a common language for understanding the security program.

  3. Measurement improves. It enables the best possible management of the security program, it enables investment planning and decision making, and it drives necessary change throughout the organization.

“Okay,” you might say to me, “Caroline, I’m convinced that metrics are important. But how do I apply the concept of measurement to appsec pen testing? What metrics should I use?”

Well, there’s a super smart guy named Victor R. Basili who came up with a model that makes it easy. The model is called G-Q-M, and it stands for Goal-Question-Metric.

Let’s take a look at an example from real life. I visited the doctor recently for my annual physical, and one of the topics we discussed was exercise. My doctor recommended that I aim for 30 minutes of physical activity per day.

Metrics 2

Metrics 3

During any given week, I want to achieve the highest ratio possible. 100% = 7 days with 30+ minute workouts / 7 total days in a week.

Using Metrics to Improve AppSec

Just as I did for my fitness metric above, when designing metrics for application security you want to specify an objective and assess security activity against that objective. This will allow you to determine how much progress you’re making in terms of achieving your goal. Most commonly, a security metric will consist of a ratio — a number of instances divided by a total number.

Measurement should be conducted frequently and consistently in order to generate trend data. Once you begin to gather data and calculate metrics, that information can be visualized in a chart or graph. Visualizations can make the data easier to consume and analyze. If the goal has been clearly stated and the visualization has appropriate labels, then you should be able to easily see from the visualization what is going well and what needs improvement.

Different Types of AppSec Metrics

In this blog series, I plan to discuss two groups of appsec metrics. The first group can be used to analyze the overall pen test setup for an organization and make strategic decisions. The second group can be tied to an individual pen test engagement to determine how well it is performing. Here are some examples:

Pen Test Program Metrics

  1. Portfolio Coverage: An organization should apply security controls in a risk-based manner across its entire software portfolio.

  2. Pen Test Frequency: An organization should conduct a penetration test on critical applications once a quarter.

  3. Time-to-Fix: Finding is great, but fixing is what actually improves application security. Critical findings should be fixed as soon as possible.

Pen Test Engagement Metrics

  1. Talent Ratings: The most important attributes of any security researcher are their skillset, experience, and performance. You want a security researcher to have skills that are matched to your application’s technology stack. You want them to have many years of professional experience conducting security tests. And you want them to be highly rated by team members and clients on their past performance.

  2. New Issues Found: Ideally, you want to find all the security bugs. It’s impossible to know if all of the security bugs have been discovered in a given pen test, but an indicator that can be counted and tracked is the number of new high criticality issues which were found in the most recent test.

  3. Vulnerability Types: There are many different types of security vulnerabilities. By visualizing and analyzing how many instances of each vulnerability type have been found in a pen test, an organization can begin to strategically eliminate certain types of vulnerabilities by focusing prevention strategies on a particular category.

  4. Findings Criticality: Some findings are more critical than others.

  5. Issues Fixed: Finding is great, but fixing is what actually improves application security. An organization should track how many issues found in each pen test actually get fixed.

What kinds of metrics are you using to manage your AppSec pen test program? Share your thoughts in the comments below.

In my next post, I talk about each of these metrics in greater detail.

Download the full report containing 2016 pen test data from Cobalt’s crowdsourced pen test platform here.

Related Stories

Cybersecurity Statistics for 2021
Cybersecurity Statistics for 2021
What's new in ransomware, social engineering, and many other security threats
Read moreArrow Right
The State of Pentesting 2021: Common Vulnerabilities, Findings, and Why Teams Struggle With Remediation
The State of Pentesting 2021: Common Vulnerabilities, Findings, and Why Teams Struggle With Remediation
Each year, we publish The State of Pentesting report to provide a detailed overview of vulnerabilities and identify the trends and hazards that impact the cybersecurity community.
Read moreArrow Right
How to Build Resilience in Cybersecurity: 4 Lessons Learned From Military Experience
How to Build Resilience in Cybersecurity: 4 Lessons Learned From Military Experience
What better group to turn to for advice than security leaders who have worked on the front lines of risk and uncertainty?
Read moreArrow Right
New Ebook: Beginner’s Guide to Compliance-Driven Pentesting
New Ebook: Beginner’s Guide to Compliance-Driven Pentesting
Find out more about the role of pentesting in your company’s compliance effort.
Read moreArrow Right

Never miss a story

Stay updated about Cobalt news as it happens