Pen Test Metrics 101: Benefits and Types

This post is the first of a series in which I’ll cover how to use pen test data in security metrics to analyze and improve your application security program.

Benefits of AppSec Metrics

It’s been said that “if you can’t measure it, you can’t manage it.” While it turns out that Peter Drucker never actually said that, it is indisputable that measuring results and performance is crucial to an organization’s effectiveness, and this definitely applies to application security.

Here are a few benefits of using metrics to evaluate an application security program:

1. Measurement provides visibility.

2. Measurement educates and provides a common language for understanding the security program.

3. Measurement improves. It enables the best possible management of the security program, it enables investment planning and decision making, and it drives necessary change throughout the organization.

“Okay,” you might say to me, “Caroline, I’m convinced that metrics are important. But how do I apply the concept of measurement to appsec pen testing? What metrics should I use?”

Well, there’s a super smart guy named Victor R. Basili who came up with a model that makes it easy. The model is called G-Q-M, and it stands for Goal-Question-Metric.

Let’s take a look at an example from real life. I visited the doctor recently for my annual physical, and one of the topics we discussed was exercise. My doctor recommended that I aim for 30 minutes of physical activity per day.

During any given week, I want to achieve the highest ratio possible. 100% = 7 days with 30+ minute workouts / 7 total days in a week.

Using Metrics to Improve AppSec

Just as I did for my fitness metric above, when designing metrics for application security you want to specify an objective and assess security activity against that objective. This will allow you to determine how much progress you’re making in terms of achieving your goal. Most commonly, a security metric will consist of a ratio — a number of instances divided by a total number.

Measurement should be conducted frequently and consistently in order to generate trend data. Once you begin to gather data and calculate metrics, that information can be visualized in a chart or graph. Visualizations can make the data easier to consume and analyze. If the goal has been clearly stated and the visualization has appropriate labels, then you should be able to easily see from the visualization what is going well and what needs improvement.

Different Types of AppSec Metrics

In this blog series, I plan to discuss two groups of appsec metrics. The first group can be used to analyze the overall pen test setup for an organization and make strategic decisions. The second group can be tied to an individual pen test engagement to determine how well it is performing. Here are some examples:

Pen Test Program Metrics

1. Portfolio Coverage
   An organization should apply security controls in a risk-based manner across its entire software portfolio.

2. Pen Test Frequency: An organization should conduct a penetration test on critical applications once a quarter.

3. Time-to-Fix: Finding is great, but fixing is what actually improves application security. Critical findings should be fixed as soon as possible.

Pen Test Engagement Metrics

1. Talent Ratings: The most important attributes of any security researcher are their skillset, experience, and performance. You want a security researcher to have skills that are matched to your application’s technology stack. You want them to have many years of professional experience conducting security tests. And you want them to be highly rated by team members and clients on their past performance.

2. New Issues Found: Ideally, you want to find all the security bugs. It’s impossible to know if all of the security bugs have been discovered in a given pen test, but an indicator that can be counted and tracked is the number of new high criticality issues which were found in the most recent test.

3. Vulnerability Types: There are many different types of security vulnerabilities. By visualizing and analyzing how many instances of each vulnerability type have been found in a pen test, an organization can begin to strategically eliminate certain types of vulnerabilities by focusing prevention strategies on a particular category.

4. Findings Criticality: Some findings are more critical than others.

5. Issues Fixed: Finding is great, but fixing is what actually improves application security. An organization should track how many issues found in each pen test actually get fixed.

What kinds of metrics are you using to manage your AppSec pen test program? Share your thoughts in the comments below.

In my next post, I talk about each of these metrics in greater detail.

Download the full report containing 2016 pentest data from Cobalt’s Pentest as a Service (PtaaS) platform.