Crowdsourced Penetration Testing and PCI DSS Compliance

Various regulatory requirements that are lurking across different industries — involving different aspects of Information Security — have never been more mature and demanding than they are today. Take PCI DSS (Payment Card Industry — Data Security Standard) for example, the standard itself has been around for quite some time now, but still lots of organisations are finding it quite difficult to comply with it without experiencing a lot of disruption within their business operations.

One of the only ways to ensure compliance, while avoiding major impact on business and its continuity, is through automation. Automation of controls for PCI-DSS like user access reviews, vulnerability scanning, change management, etc. is fairly easy to achieve if you have the right tools in place and your organisation is not struggling with its internal processes from the complexity point of view.

Although automation can cure lots of pain and get us on the right track from a maturity perspective — as it will practically shift compliance-related work from “project work” to BAU (Business As Usual) — there are certain controls that are simply impossible to consider for automation. Those being: Information Security Policies Review exercises, firewall rule reviews in most cases, and of course — independent penetration testing assessments. Now, is that really true when it comes to this last point and is there a way to embed this process into your activities?

Penetration Testing Requirements Under PCI DSS

Penetration testing exercises are usually painful, expensive, and they drag along a lot of reactive remediation which usually impacts development teams and their productivity. PCI DSS standard provides several different requirements on Penetration Testing.

Let’s have a look at what these requirements actually are and what additional information this standard provides in order to help organisations understand requirements better as well as get a more holistic view over compliance requirements. Let’s start with a simple Penetration Testing description exercise:

• PURPOSE: to identify ways to exploit vulnerabilities to circumvent or defeat the security features or system components.

• WHEN: at least annually or upon significant changes (infrastructure upgrade or modification, new system component installation)

• HOW: manual process plus vulnerability scanning or use of other automated tools

• REPORT: description of each vulnerability verified potential issues discovered. More specific risks that vulnerability may pose, including specific methods how and to what extent it may be exploited.

• DURATION: engagement may last days or weeks depending on the scope of the test and size of the environment. May grow in time and complexity if efforts uncover additional scope

Ok — Seems pretty standard stuff so let’s move on to the actual environment and segments that should be assessed by this exercise:

SCOPE: As defined in PCI DSS requirement 11.3, a penetration test must include entire CDE perimeter and any critical systems that may impact the security of the CDE as well as the environment in scope for PCI-DSS. This includes both the external (public facing attack surface) and the internal perimeter of CDE (LAN-LAN attack surfaces).

Term “Critical Systems” described by PCI DSS “security systems, public facing devices and systems, databases and other systems that store, process or transmit cardholder data.” (requirement 6.1) firewalls, IDS, authentication servers..) any assets utilised by privileged users to support and manage CDE.

PCI-DSS defines the cardholder data environment (CDE) as “the people, processes, and technology that store, process, or transmit cardholder data or sensitive authentication data.” Now, if we have a look into the external assessment which is the most important one as it may present the highest risk to the organisation it states.

EXTERNAL: Exposed external perimeter of the CDE and critical systems connected or accessible to public network infrastructures. Testing must include both application-layer and network-layer assessments. External penetration tests also include remote access vectors such as dial-up and VPN connections.

Ok, so far all of this seems fairly reasonable and does not contain anything that might be too surprising, right? Let’s move on to the more interesting bit, related to the people who are actually performing the assessment and any requirements around that.

WHO: “Qualified third party may perform the penetration test as long as they are organizationally independent.“

Diving deeper into potential security certification requirements reveals pretty general indication:

“Might be indication of the skill level and competence of a potential penetration tester or company. These are not required certifications, they indicate a common body of knowledge held by a candidate: OSCP, CEH, GIAC, GXPN, CREST, CESG/CHECK …

.. PCI SSC Does not validate nor endorse these certifications.”

Advice from PCI-DSS on experience and details about the organisation/people performing the assessment:

• “How many years experience does the penetration tester have?”
• “Has the penetration tester performed assessments against organisations of similar size and scope?”
• “… experience with the technologies in the target environment (OS, web technologies, customised apps, network services, protocols etc.)”
• “Qualifications that will contribute to their ability to assess the environment.”

Crowdsourcing Perspective on PCI DSS Requirements

Basically, we’ve gotten to an interesting point where we can see that nothing is strictly defined within the standard itself, most of these things are advising organizations on how to recognize a good vendor for independent security assessments and are providing guidelines on scoping. So, let’s turn now and look at previously identified PCI DSS requirements / guidelines from the crowdsourcing perspective, through bug bounty programs or on-demand assessments:

WHEN: at least annually…; Fully applicable and of course possibility to do 24/7/365 through curated bug bounty programs with more flexibility and much better remediation tracking.

DURATION: days or weeks …; Pretty much anytime and and for as long as you like. Fully flexible just like the point above.

HOW: manual + automated tools …; Of course, manually with unlimited skill-set pools including lead industry experts with unlimited well known and publicly unknown tools coverage.

PEN TESTER EXPERIENCE: experience with the technologies in the target environment … Pretty much any preferable experience you can imagine involving any technology set out there.

PEN TESTER CERTIFICATION: Might be indication of the skill level and competence; Again, fully applicable on an individual basis and in addition combined with professional experience?

METHODOLOGY: Both 6.5 (web app) and 11.3 (network) section of the standard are quite broad and crowdsourced assessments are definitely satisfying all of the PCI DSS requirements as they are following industry standard methodologies for both web and network assessments.

REPORT: description of each vulnerability verified potential issues discovered …; There is no difference in final report template between traditional penetration test reports and Cobalt crowdsourced ones except that in addition to that you get real time full reporting capability.

I won’t even go deeper into additional benefits of a crowdsourced approach when it gets to remediation tracking through Jira integration in example or flexible resources for re-test etc.

Conclusion

As we can see, Cobalt on-demand security assessments and bug bounty programs that are using crowdsourced approach are fully compliant with all of the PCI-DSS requirements and are providing much more in comparison to traditional penetration testing services. Why is there so much fuss around it then when it gets to implementation of the service and achieving compliance?

There are two sides of this story i would say. First one is definitely lack of understanding of the PCI-DSS standard itself and what is actually required by it. Secondly, many organisations tend to “integrate” with traditional security companies and their services where they get both, penetration testing and PCI-DSS audit services bundled. Also, standard is actually not strictly defining or enforcing any of these processes and is mostly advising on all the activities while people tend to think there has to be something hiding behind these requirements as it usually is in those horrific contractual agreements or terms and conditions they bump into on a daily basis.

Well, let’s be honest here, there isn’t anything that’s hiding behind it and there is absolutely no reason to do something just because someone else is doing it or because you have been doing it in the past. “When in Rome, do as the Romans do …” does not really apply to the ever evolving technology field, especially when it gets to Information Security where only experimenting with different approaches and methodologies can potentially put us one step ahead of threats we’re facing today and will potentially face in future.

—

Learn more about how companies seeking to achieve PCI Compliance benefit from working with Pentest as a Service (PtaaS) Platform.