Cobalt Crowdsourced Application PentestCobalt Crowdsourced Application PentestCobalt Crowdsourced Application Pentest



Algolia’s mission to help companies build consumer-grade search through a hosted API is built on a unique trust between the company and its customers. Its desire to protect personal data and tangibly demonstrate its security commitment to customers led the search company to modernize and expand its penetration testing program.

Adam surak
Adam Surak
Director of Infrastructure

Founded in 2012, Algolia enables developers and product teams to build consumer-grade search. With its hosted search API, customers benefit from the power of a lightning-fast engine with out-of-the-box support for a typo-tolerant, as-you-type search experience. Algolia recognizes that its customers place a great deal of trust in the company when they provide data for the search API. Its commitment to protecting customer data led the company to create a robust, effective and transparent security program. Pen testing is a significant part of its application security work, used both to reduce vulnerabilities and provide tangible evidence of its security posture to customers.

Looking for a New Approach

Adam Surak, Director of Infrastructure at Algolia, and his team recognized that pen testing had to be a part of its application security strategy. Continuous testing is not only used to identify and mitigate security issues, but also to demonstrate to customers and potential customers that Algolia is committed to security, testing applications at least once every six months. However, traditional penetration testing was creating a host of challenges for the company, including push back from the engineering team that found the final PDF reports cumbersome and limited in value.

"It was a challenge to find a pen testing company that fit how we think pen testing should work. Our great frustration was black box testing. With this model you can’t discuss, influence or be involved at all. You just get a PDF at the end and then ‘this is your pen test, goodbye.’ So we started to look for solutions with a different approach. That is how we found Cobalt."

Cobalt provides an innovative model, Pen Testing as a Service, completely supported by a platform that manages and captures the full find-to-fix workflow. For Surak, it was the transparency at each stage of the testing process that unlocked the value of pen testing for their security program.

Ease of Pen Test Preparation

Cobalt’s ability to find, provide, and manage different security researchers for requested pen test engagements solved two problems for Algolia. First, Cobalt’s team of security researchers use its platform to capture all relevant data from the entirety of its work with Algolia. This streamlines the knowledge transfer required for new testing engagements and significantly reduces the time spent by Algolia’s security team in preparing a new set of researchers every time it initiates a test. Second, Cobalt’s ability to quickly add new researchers to the testing team ensures fresh eyes whenever needed.

"Cobalt’s model ensures we do not have to start from scratch every time a new test is needed, without having to worry that the pen testers have grown biased as they become more knowledgeable about the system. Avoiding bias is important because it sometimes leads to missed findings through things like skimming parts of the application that they have seen before where nothing has been previously found. Being able to add new researchers to the team without having to invest our limited internal resources in a long knowledge transfer process is very valuable."

While Cobalt does reduce the time Algolia’s security team has to spend preparing for new tests, it also provides a means for the team to communicate with the security researchers ahead of the test initiation to draw their attention to new areas of concern, architectural changes or specific goals. This transparency has helped Algolia ensure that pen testing continues to stay aligned with its testing goals and avoids wasted effort.

"One very interesting thing that we have seen with Cobalt is their rigorous preparation. For example, they give us the IP addresses that are going to be testing our system, ask us for AWS permission numbers to actually perform the pen test on the AWS platform, and discuss how to speed up the pen test and lower the load on our site, etc. We have never seen this level of thoroughness before and it has greatly increased our engineering team’s comfort with the testing process."

Seamless Communication

Cobalt’s transparency during the testing itself provides Algolia’s engineers with peace of mind. They know they can ask questions, point out mitigations that may not be obvious, ask whether the right limit was set, and through Slack and other channels, actually get to see a little bit about what the researchers were thinking as they review an application.

"With traditional pen testing firm there is no platform. You send an email with the description of the service, you get a PDF back. The ‘in-between’ stays the magic for the consultancy. Cobalt is different - there is transparency throughout the entire process."

This transparency also created a feedback loop that is was not existent with traditional pen testing. After the first test with Cobalt, the Algolia team had feature ideas and was able to see them implemented in subsequent tests.

"Hearing from other consultancies that the test they provided was the best they could do despite its flaws was frustrating. There was no feedback loop whether or not the test met our needs. It was a disaster."

Not Just a Point in Time Pen Test

The communication with Cobalt’s researchers continues after the test. Through its platform, Cobalt provides very clear explanations of its findings and proposed mitigations for Algolia’s engineers. But that is not where the engagement ends. There is an opportunity for engineers to use the platform to collaborate during the fix process and easily request retests as needed.

"Now, engineers almost looking forward to testing. They know that the interactions with the researchers make them better engineers. The researchers clearly explain security issues and proposed mitigations. Then our engineers can review the findings, ask questions and bring their own expertise to the mitigation process. Together, we come up with solutions that benefit our customers. It is no longer a burden for engineers to be included in the pen testing process. In fact, Cobalt works exactly how you want a penetration test to work."

The Cobalt platform’s reporting capabilities are also helpful to Algolia. They are able to simply create different reports with varying levels of detail for different audiences. As security documentation for customer and sales support is one of the drivers for continuous external testing, the ability to be able to automatically create customer-facing reports to share under NDA has proven to be another valuable time-saver for the security team.

"We want to we provide customers with penetration testing results and tangibly demonstrate that we care about security and are always working to find and fix issues before they become their problem. Our customers know that at the end of the day, security is my team’s responsibility and we take it very seriously."

Cobalt bug hunters@2xChallenges

  • Finding a pen testing approach that would strengthen security without slowing down the software development process through burdensome test initiation and confusing findings reports
  • Reducing the testing support strain on the internal security team and product engineers without sacrificing test quality

Icon  checklistSolutions

  • Utilize Cobalt's Pen Testing as a Service Platform

Cobalt certified researchers@2xBenefits

  • Access to Cobalt’s verified pool of security researchers that apply diverse backgrounds and skill sets to application pen testing
  • A pen testing process that is completely transparent, enabling productive collaboration between security researchers and engineers throughout the entire find-to-fix workflow
  • Customizable reporting options that empower Algolia to share relevant test findings with customers and tangibly demonstrate its ongoing security commitment