A Manager’s Guide to Selecting the Best Testing Approach for Your Application Security Needs

Security testing has emerged as a common best practice among application security programs of all types and sizes. The need for testing — or finding potential security issues — has long been well understood. It isn’t the “why” that challenges most organizations — it is the answers to how, when, and even what to test that can often be elusive. The evaluation of testing approaches has only grown more complex as many organizations move toward more iterative, continuous software development methods, further straining even long-held notions of effective security testing best practices.

Selecting the best application security testing approach, and the tools associated with it, requires a careful alignment with an organization’s unique development environment and culture, risk profile and business demands. It is also important to recognize the strengths and limitations of each testing type to accurately interpret test results and manage application security risk. In fact, many organizations with mature security programs use a combination of testing tools and approaches since each can be useful — so long as the handling of their outcomes aligns with an accurate understanding of their capabilities.

This article is designed to help organizations select the best application security testing types for their needs. It will describe three of the most popular testing types, comparing them in four important areas: Coverage, Scalability, Ease of Testing Process and Triage Support.

Application Security Testing Approaches: Descriptions

This article will focus on three types of application security testing approaches: Scanners, Bug Bounties, and Pentesting as a Service.

Scanners

Commercial vulnerability scanners are programmed to discover vulnerabilities in applications and systems, whether a single computer, multiple computers, network connectivity devices, applications, or services. Scanners provide consistent and scalable results — they will reliably find everything they are programmed to find but will miss everything they are not. For this reason, they tend to be most effective when customized to a particular environment or application.

Scanners can augment an in-house application security team’s creativity and self-direction. There are certain tasks for which security scanners are extremely helpful, for example, static analysis can really assist with manual source code review.

Bug Bounties

Public bug bounties emerged out of the once novel idea of working with the hacker community — the same people who are likely already poking at your software for fun or notoriety. Leading companies recognized that creating a more formal way of working with security researchers could improve the process of responsible vulnerability disclosure and that their findings could augment their security programs.

With a public bug bounty, anyone in the world can submit a potential security vulnerability to an organization, and if they are the first to find a valid bug, they will be paid a “bounty” and potentially even recognized publicly for their work. An organization running a bug bounty will pay the cost of each bounty and manage the overhead of reviewing and filtering all the reports to identify false positives, duplicate reports, and other issues.

Pentesting

A penetration test, also known as a pentest, is an analysis and simulated attacks on an application (web, mobile, APIs) or network to check its security posture. Manual pentesting layers human expertise on top of professional penetration testing software and tools, such as automated binary static and automated dynamic analysis. A manual pentest provides complete coverage for standard vulnerability classes, as well as other design, business logic and compound flaw risks that can only be detected through manual (human) testing.

Pentesting as a Service combines manual, human testing with a modern delivery platform. Heavily vetted domain experts are chosen from a carefully sourced pool of freelance researchers to work collaboratively on a penetration test of a selected application. This keeps costs low while still providing credentialed researchers and customer-defined, focused code coverage.

Pentesting as a Service offers a simplified triage process as results are expert reviewed and risk-prioritized before delivery, and reports are integrated with a vulnerability management platform to streamline developer-tester collaboration throughout the mitigation process. More scalable than traditional security consulting, Pentesting as a Service is ideally suited for organizations looking to increase frequency of penetration tests and more deeply integrate manual security testing into their release cycle.

Application Security Testing Approaches: Key Considerations for Evaluation

There are four key areas that an organization should consider when determining the best application security testing strategy for its needs. These are: Coverage, Scalability, Ease of Testing Process and Triage Support. These may be prioritized differently depending on an organization’s specific testing goals and internal resources. An organization that needs to pass PCI compliance or a vendor security assessment may prioritize these factors differently than an organization with an agile testing process or one that is focused on training developer teams in application security.

Coverage

Perhaps the most important of the consideration for selection, coverage is understanding what the test does and does not tell you. Coverage is a function of two key test attributes: 1) what is the testing program capable of finding (or conversely, what is it likely to miss); and, 2) how much of the application is covered. If test coverage is not well understood, any assumptions of how the test results relate to application security risk will be flawed. To ensure adequate coverage, many mature organizations will use a combination of testing approaches and tools.

Scanners: Scanners work best when highly customized to a particular environment or application. They will find everything you program them to search for and nothing you don’t. Scanners allow organizations to look for a defined set of vulnerabilities across a defined area of code. As such, coverage is both known and predictable. However, the findings will be limited. Scanners cannot “think” beyond their programming. As such, they will not find the design and more complex business logic issues that human creativity in manual testing can uncover.

Public Bug Bounty: Coverage is a challenge for Public Bug Bounties. Theoretically, they provide a large number of human testers looking for a diverse number of issues across the full amount of code opened to the bounty. However, coverage will almost always remain unknown and unpredictable. This is because the provenance of the security researchers and their tools will not be known, nor will the scope of issues and code they test against. As such, it is possible some sections of the application may remain untested or only scanned for a limited class of security flaws.

Pentesting as a Service: Organizations hire penetration testers to review specific parts of an application, sometimes taking a broad look across an application or digging down into specific features and functionality. These types of tests can be especially useful for providing depth of coverage for applications that require specialized knowledge to test effectively, such as mobile apps, because researchers with those unique credentials can be specifically requested to review their complex functionality. Most penetration tests use a procedural approach to ensuring coverage, using something like OWASP Top 10 to ensure they are looking for as many relevant classes of security issues as possible. As such, coverage should be known. However, human error will insert some unpredictability into the testing process as people can and do make mistakes. One of the more cited benefits of using Pentesting as a Service over traditional consulting services is the ability to constantly find “fresh eyes” to conduct tests. Many organizations report that researcher creativity often declines when tasked with repeatedly reviewing the same application parts.

Scalability

For application security testing, scalability is not simply about the volume of code to be tested, but also the frequency of testing. As companies move toward more iterative, continuous development approaches, the demand for test frequency has also increased. Scalability will be most impacted by the level of automation in testing and delivery, staffing requirements (i.e. “man hours”), and fixed costs of each method.

Scanners: As the most automated of all the approaches, scanners can scale very well for monolithic environments, assuming that all of the requisite customization, internal training, and other “care and feeding” is in place. As long as they were used by knowledgeable team members in a fairly uniform environment, they can both test a large volume of code and test frequently.

Public Bug Bounties: Public bug bounties can attract a large number of researchers if a program is appealing, providing eyes across a potentially large volume of code. In addition, bug bounties are relatively easy to initiate and can be set up to run continually or quickly initiated whenever a change demands it. One challenge is making sure long-standing bug bounties remain attractive enough to draw researchers even after the initial low hanging fruit is found.

Pentesting as a Service: Using paid consultants to perform penetration testing has historically been the least scalable of all the testing options given the high price of their expertise and creativity. However, pentesting as a service offers significantly improved scalability of manual testing due to its modernized approach to researcher identification and testing delivery.

With pentesting as a service, heavily vetted domain experts are chosen from a carefully sourced pool of freelance researchers to work collaboratively on a penetration test of a selected application. This keeps costs low while still providing highly credentialed researchers. Test initiation, management and triage are all performed within an application security platform that automates many previously-manual tasks across the complete find-to-fix workflow. Reduced costs combined with more streamlined test delivery allows organizations to penetration test more code, more frequently.

Ease of Testing Process

Ease of testing process refers to the level of work in initiating and managing an application security test for an organization’s internal staff, sometimes referred to as management overhead. Though highly related to scalability, ease of testing should be considered in its own right given its impact on internal resource planning and total cost of ownership.

Scanners: Scanners are most effective when customized to an organization’s applications and environment. They also tend to require periodic “tuning” to maintain effectiveness. This often results in a high initial investment in total cost of ownership that decreases over time. Once customized, test initiation and the scanning process are fairly streamlined assuming the in-house team has the skills necessary to manage the test. In fact, one of the more prevalent use cases of scanners is a “self-service” model where developers are provided scanning tools, such as static analysis, and encouraged or required to use them on their own at various phases in the development process.

Public Bug Bounties: Public Bug Bounties are comparatively easy to initiate and manage since organizations have so little control over the testing approach and process itself. The biggest ease-of-use challenges of a bug bounty program are: 1) working to ensure the terms of your program are attractive enough to draw and retain researcher attention, and 2) planning for the unpredictable changes in the volume of effort required by internal staff to maintain effective and productive communications with external researchers. Some organizations invest in a bug bounty platform to help outsource and streamline some aspects of program management.

Pentesting as a Service: Penetration testing has historically required a high level of staff resources across the full scope of test design and delivery. This has been an obstacle to many organizations seeking the expertise and coverage penetration testing provides but lacking the resources to manage the testing process itself. Penetration Testing as a Service can reduce this management overhead for internal security teams. Its modern delivery model matches organizations with the right testers for their application and environment and then leverages an intuitive management platform to support organizations at each stage of the find-to-fix workflow. Pentesting as a Service improves the ease-of-use of penetration testing through increased testing transparency and automation and streamlined communications across the full test workflow.

Triage Support

Triage support refers to how test findings are delivered, and if/how the remediation of identified issued is supported. Though related to scalability and ease-of-use, triage support requires special consideration in the evaluation process because, for most use cases, test findings are arguably only valuable if they lead to remediation and application security risk reduction.

Application security test findings are often discussed in terms of the ratio of signal to noise. Think of tests with a with a low signal to noise ratio like listening to a radio station with lots of static, making it difficult to hear the song you tuned in to hear. Tests with a high signal to noise ratio would be like listening to a radio station with very little static, so the song comes through loud and clear.

In addition to considering the signal-to-noise ratio, organizations may also want to think about how much support is provided to the mitigation after initial findings are delivered.

Scanners: The low signal to noise ratio is one of the most often cited areas of dissatisfaction with vulnerability scanners. Scanners tend to produce a large volume of findings that have to be manually filtered for what is often a high number of false positives. Careful customization and continuous tuning can help reduce false positives, but typically not enough to reduce the need for a lengthy and resource-intensive filtering and prioritization process. As such, high risk vulnerabilities may take longer to be identified as a high priority and thus their remediation delayed.

Public Bug Bounties: Public Bug Bounties also tend to have a low signal to noise ratio. An attractive program will produce a large number of findings, but they will need to be carefully reviewed and filtered for duplicates and false positives. For example, in Facebook and Google’s independent bug bounty programs only 4% and 5% of the reports submitted actually end up with a reward. This means that more than 90% of submissions are duplicates or invalid reports. As with scanners, this also leads to a longer time-to-fix for even the most critical of findings.

With Public Bug Bounties, communication after a finding is reported tends to be inconsistent. While some researchers may answer questions and offer support after submitting their findings, others may not be as willing.

Pentesting as a Service: Triage support is perhaps the most cited advantage by organizations opting for Pentesting as a Service, which offers the highest signal to noise ratio of the options discussed here. With this approach, the penetration testing team works collaboratively to find security issues and then filters all of its findings before delivering them to the application owner. It will also flag any critical, high priority findings. These extra steps result in far fewer duplicate findings and false positives, as well as help ensure that high priority findings are quickly seen (and fixed!) by the application owner.

While traditional consultancies deliver findings in a lengthy PDF, Pentesting as a Service presents them in a vulnerability management platform that enables real-time collaboration and communication between the researchers and internal security and engineering teams. This reduces the overhead associated with previously time-consuming triage tasks, such as the numerous conference calls and emails used during remediation. Further, Pentesting as a Service extends its support to re-test and fix verification so organizations can leverage researcher expertise far beyond the initial delivery of findings.

Conclusion

With so many application security testing approaches and tools to consider, deciphering the options seem daunting. To make sense of it, an organization should first review all of its internal drivers for testing and think through questions such as:

• What is driving the need for testing?

• What do we hope to gain from conducting the tests?

• How will the results be used?

• Is coverage depth or breadth more important?

• How much time do we have to test?

• What internal resources can we dedicate to the process?

Once internal priorities and constraints are understood, organizations can then begin to evaluate testing options using the four most impactful assessment criteria discussed here: coverage, scalability, ease of testing process and triage support. This thoughtful approach to application security test evaluation will not only help an organization to select a testing approach that most aligns with their needs but also to recognize each testing option’s strengths and limitations to achieve an accurate understanding of their true application security risk.