Less Findings in your Pentest? Measuring the Effectiveness of a Penetration Test

Have you been doing pentesting for a while and getting fewer and fewer findings or fewer high-severity findings? Is it time to switch pentesting vendors?

The answer is that this is an outdated point of view. The real reason you're finding fewer vulnerabilities is that your organization is becoming more mature and you need to change testing tactics. This could entail rotating pentesters or changing your testing methods. Attackers are constantly refining their techniques, and you need to do the same to stay a step ahead. Maybe instead of just performing a black box pentest you need to do a code-assisted pentest. Or perhaps instead of a network pentest, you should run red teaming.

In this blog, we'll share how to measure the effectiveness of your pentesting methodology and evaluate what type of testing you need to do next. We'll cover:

• Key performance indicators (KPIs) for pentesting programs
• Benefits of code-assisted pentest (code-assisted pentest) assessments and secure code reviews
• Benefits of red teaming
• Transcending pentesting for holistic security

KPIs for Your Penetration Testing Program

To evaluate the maturity of your pentesting program and plan your next steps, it helps to establish a baseline. Tracking pentesting key performance indicators can help you assess where you are. 

Depending on your pentest maturity, KPIs can differ. For example, a pentest program usually starts with ad hoc testing with minimal structure. At level 1, companies are often testing for a specific reason such as to achieve compliance or show a potential customer their systems are secure. At level 2, companies start to mature to the point that they have tools and processes put in place to add consistency and direction to their testing program.

Once at level 3, teams start to add automation into their security practices. This is where a mature program starts to form with consistent testing cadences and automated scans for continual monitoring. Moving towards level 4, companies start to change their end-goal with testing and will prioritize cross-functional initiatives. 

Read more about the different pentest maturity levels in The PtaaS Book. Some important pentesting KPIs for pentest maturity include:

1. Attack surfaces or assets tested
2. Test cadence
3. Testing methods
4. Vulnerability discovery rate
5. Critical vulnerabilities
6. False positive rate
7. MTTR (Mean time to resolve)
8. Regulatory compliance
9. End user impact

Some of these metrics can be expressed as discrete binary values, while others can be expressed as percentage rates or continuous variables.

1. Attack Surfaces Tested

This metric defines the scope of your pentest by itemizing the systems, applications, and endpoints covered in your test. As an itemized list, it can be expressed in discrete form as a yes-or-no checklist.

2. Test Cadence

This measures how often you conduct a given type of test, expressed as a rate of tests over a given period of time. This can be a key metric to determine your pentest maturity.

3. Testing Methods

This itemizes which types of methodologies you deployed in your pentest, such as Open-Source Security Testing Methodology Manual (OSSTMM), Open Web Application Security Project (OWASP), or Penetration Testing Execution Standard (PTES), along with what types of attacks you simulated, such as social engineering, SQL injection, or cross-site scripting.

5. Vulnerability Discovery Rate

This ratio measures how often your security team discovers and reports vulnerabilities.

6. Critical Vulnerabilities

This discretely classifies vulnerabilities based on their severity.

7. False Positive Rate

This number of findings that are not reproducible.

8. MTTR (Mean time to resolve)

This measures how long it takes your team to patch vulnerabilities after they've been introduced or identified. Security and development teams together should establish agreed upon SLAs that dictate how long remediation should take based on the severity of the vulnerability . 

9. Regulatory Compliance

This checks off whether your pentest achieves specific regulatory requirements.

Using KPIs to Measure Pentesting Maturity

If you're noticing your vulnerability discovery rate and critical vulnerability frequency are falling, this may be a sign that your pentesting has reached maturity for the level of testing you're administering. In this case, you may need to adjust your testing methodology by considering whether it's time to take your pentesting to the next level with a code-assisted pentest.

Benefits of Code-assisted Pentests and Secure Code Reviews

A code-assisted pentest differs from secure code reviews in that they focus on the use of coding access to gain insight into vulnerabilities, rather than focusing on the code itself. 

For example, code-assisted pentest  combines black box testing techniques of how an application behaves in its environment with white box techniques for internal testing of code. It simulates an attack where the attacker has access to the application's code. This can include access to source code of third-party dependencies.

Access to source code gives code-assisted pentest tests advantages over standard pentests. With source code, pentesters can quickly locate the causes of vulnerabilities. This improves the ability to detect certain types of vulnerabilities which may be hard to detect with regular pentesting, while making it easier to test fixes, saving time, and reducing false positives.

Code-assisted pentest can help uncover vulnerabilities such as:

• SQL injections: Use of malicious queries to manipulate and steal data from application backends
• XML external entry injections (XXE injections): Use of malicious XML input to manipulate weakly configured XML parsers into resolving unintended documents
• Code injection: inserts malicious code into applications
• Command injection: Passing malicious commands through vulnerable apps to operating systems
• Server-side template injections: insert malicious code into templates run on servers
• Failures with encryption and Secure: Random random number generation
• Failures with backend access protections: such as certificate validation, encryption, user authentication, and user authorization

Essentially, code-assisted pentest provides superior coverage against injection attacks and misconfiguration vulnerabilities. Combined with its accuracy and time-saving benefits, this makes code-assisted pentest a step up from standard pentesting.

To take code-assisted pentest to the next level, the next step is to conduct a full secure code review. In a secure code review, the end goal aims to identify security flaws in the application related to its features and design, along with the exact root causes. This helps uncover security vulnerabilities and malware injections as well as code readability, uniformity, understandability, output correctness, and performance issues.

After a full secure code review, you can further harden your defenses by conducting a pentest to further validate findings. This combination will show deeper insights and provide improved coverage because some vulnerabilities can’t be validated by simply looking at the code. 

After securing your applications, you can start to look at the underlying network with a red teaming exercise. Pentest, code-assisted pentest, and secure code reviews will focus on the application security side of your tech stack, while a red teaming engagement will benefit the network that supports the app and benefit your overall corporate security strategy. 

Benefits of Red Teaming: Network Security

Red teaming simulates realistic attacks on their networks, bridging the gap between application and network security. A red team attack adopts an attacker's perspective, setting pragmatic objectives and utilizing common tools to achieve them. Red teaming is important to stop an attacker from getting into the broader corporate network.

Red team attacks begin by performing reconnaissance on system vulnerabilities, using tools such as active scanning, phishing, and searching the open web. After gaining initial access to systems, red team attackers seek to execute different vulnerabilities such as escalating privileges or taking over additional accounts, while evading detection. This sets the stage to seize command of networks and execute data theft or system disruption.

The goal of a red team engagement is to stay under the radar of the SOC team. Red teaming uses threat intelligence to attack the network and helps teams improve their ability to respond to an attack. Red teaming helps security teams identify vulnerabilities in networks and measure their response to intrusions. Additionally, working with red teams can help organizations assess security controls and incident responses. Information gathered during red teaming can help security team training by improving threat awareness and providing insights into how to fix vulnerabilities and counter-attacks.

Beyond Pentesting: How Red Teaming Enables a Holistic Security Strategy

By going beyond pentesting and secure code reviews, red teaming empowers organizations to implement a holistic security strategy. Where pentesting and secure code reviews also uncover vulnerabilities, red teaming uses threat intelligence to simulate real-world attacks and can both identify vulnerabilities and help with the response to an attack. This makes red teaming useful for uncovering both external and internal risks.

This provides information on vulnerabilities attackers can exploit to gain initial access, as well as the integrity of initial defenses. Internal red teaming uses a technique called an assumed breach, where they assume that an attacker has already gained access. This discloses information on vulnerabilities that open up in the event of a breach, as well as the effectiveness of security control strategies and incident response procedures.

In this way, red teaming closes the loop between external and internal threats and brings everything together. Red team attacks remove limitations on tests, enabling you to see your applications and networks as an attacker sees them and to test how well your defenses hold. 

Red teaming can also support your security team’s SLAs and ensure remediation occurs in the proper timeframe based on each vulnerability's severity. This is achieved by measuring your team’s response time and ensuring their incident response playbook is properly tuned and confirmed. Together this provides you with a holistic perspective on both your vulnerabilities and your security strategy.

Optimize Your Security Testing Strategy with Cobalt

No matter the maturity level of your security strategy, there's always room for improvement. Offensive Security Services provides a full range of options to assist you with whatever your needs. 

For teams looking for more, our secure code review services combine automated tools with manual expert review of your business logic to pinpoint vulnerabilities precisely and identify fixes. Furthermore, a red team engagement allows experts to work with your team to deploy the industry-leading MITRE ATT&CK framework to test your security controls and incident responses to recommend remediations. 

Or if you're still early in your security maturity journey, our Pentest as a Service (PtaaS) delivery model can simulate attacks on your system with a traditional manual pentest or run code-assisted pentests to help you identify vulnerabilities using industry standard methodologies.