Cobalt Crowdsourced Application PentestCobalt Crowdsourced Application PentestCobalt Crowdsourced Application Pentest


Algolia’s mission to help companies build consumer-grade search through a hosted API is built on a unique trust between the company and its customers. Its desire to protect personal data and tangibly demonstrate its security commitment to customers led the search company to modernize and expand its penetration testing program.

Adam Surak
Director of Infrastructure

Founded in 2012, Algolia enables developers and product teams to build consumer-grade search. With its hosted search API, customers benefit from the power of a lightning-fast engine with out-of-the-box support for a typo-tolerant, as-you-type search experience. Algolia recognizes that its customers place a great deal of trust in the company when they provide data for the search API. Its commitment to protecting customer data led the company to create a robust, effective and transparent security program. Pentesting is a significant part of its application security work, used both to reduce vulnerabilities and provide tangible evidence of its security posture to customers.

Looking for a New Approach

Adam Surak, Director of Infrastructure at Algolia, and his team recognized that pentesting had to be a part of its application security strategy. Continuous testing is not only used to identify and mitigate security issues, but also to demonstrate to customers and potential customers that Algolia is committed to security, testing applications at least once every six months. However, traditional penetration testing was creating a host of challenges for the company, including push back from the engineering team that found the final PDF reports cumbersome and limited in value.

"It was a challenge to find a pentesting company that fit how we think pentesting should work. Our great frustration was black box testing. With this model you can’t discuss, influence or be involved at all. You just get a PDF at the end and then ‘this is your pentest, goodbye.’ So we started to look for solutions with a different approach. That is how we found Cobalt."

Cobalt provides an innovative model, Pentest as a Service, completely supported by a platform that manages and captures the full find-to-fix workflow. For Surak, it was the transparency at each stage of the testing process that unlocked the value of pentesting for their security program.

Ease of Pentest Preparation

Cobalt’s ability to find, provide, and manage different pentesters for requested pentest engagements solved two problems for Algolia. First, Cobalt’s team of pentesters use its platform to capture all relevant data from the entirety of its work with Algolia. This streamlines the knowledge transfer required for new testing engagements and significantly reduces the time spent by Algolia’s security team in preparing a new set of pentesters every time it initiates a test. Second, Cobalt’s ability to quickly add new pentesters to the testing team ensures fresh eyes whenever needed.

"Cobalt’s model ensures we do not have to start from scratch every time a new test is needed, without having to worry that the pentesters have grown biased as they become more knowledgeable about the system. Avoiding bias is important because it sometimes leads to missed findings through things like skimming parts of the application that they have seen before where nothing has been previously found. Being able to add new pentesters to the team without having to invest our limited internal resources in a long knowledge transfer process is very valuable."

While Cobalt does reduce the time Algolia’s security team has to spend preparing for new tests, it also provides a means for the team to communicate with the pentesters ahead of the test initiation to draw their attention to new areas of concern, architectural changes or specific goals. This transparency has helped Algolia ensure that pentesting continues to stay aligned with its testing goals and avoids wasted effort.

"One very interesting thing that we have seen with Cobalt is their rigorous preparation. For example, they give us the IP addresses that are going to be testing our system, ask us for AWS permission numbers to actually perform the pentest on the AWS platform, and discuss how to speed up the pentest and lower the load on our site, etc. We have never seen this level of thoroughness before and it has greatly increased our engineering team’s comfort with the testing process."

Seamless Communication

Cobalt’s transparency during the testing itself provides Algolia’s engineers with peace of mind. They know they can ask questions, point out mitigations that may not be obvious, ask whether the right limit was set, and through Slack and other channels, actually get to see a little bit about what the pentesters were thinking as they review an application.

"With traditional pentesting firm there is no platform. You send an email with the description of the service, you get a PDF back. The ‘in-between’ stays the magic for the consultancy. Cobalt is different - there is transparency throughout the entire process."

This transparency also created a feedback loop that is was not existent with traditional pentesting. After the first test with Cobalt, the Algolia team had feature ideas and was able to see them implemented in subsequent tests.

"Hearing from other consultancies that the test they provided was the best they could do despite its flaws was frustrating. There was no feedback loop whether or not the test met our needs. It was a disaster."

Not Just a Point in Time Pentest The communication with Cobalt’s pentesters continues after the test. Through its platform, Cobalt provides very clear explanations of its findings and proposed mitigations for Algolia’s engineers. But that is not where the engagement ends. There is an opportunity for engineers to use the platform to collaborate during the fix process and easily request retests as needed.

"Now, engineers almost looking forward to testing. They know that the interactions with the pentesters make them better engineers. The pentesters clearly explain security issues and proposed mitigations. Then our engineers can review the findings, ask questions and bring their own expertise to the mitigation process. Together, we come up with solutions that benefit our customers. It is no longer a burden for engineers to be included in the pentesting process. In fact, Cobalt works exactly how you want a penetration test to work."

The Cobalt platform’s reporting capabilities are also helpful to Algolia. They are able to simply create different reports with varying levels of detail for different audiences. As security documentation for customer and sales support is one of the drivers for continuous external testing, the ability to be able to automatically create customer-facing reports to share under NDA has proven to be another valuable time-saver for the security team.

"We want to we provide customers with penetration testing results and tangibly demonstrate that we care about security and are always working to find and fix issues before they become their problem. Our customers know that at the end of the day, security is my team’s responsibility and we take it very seriously."


  • Finding a pentesting approach that would strengthen security without slowing down the software development process through burdensome test initiation and confusing findings reports
  • Reducing the testing support strain on the internal security team and product engineers without sacrificing test quality


  • Utilize Cobalt's Pentest as a Service Platform


  • Access to Cobalt’s verified pool of pentesters that apply diverse backgrounds and skill sets to application pentesting
  • A pentesting process that is completely transparent, enabling productive collaboration between pentesters and engineers throughout the entire find-to-fix workflow
  • Customizable reporting options that empower Algolia to share relevant test findings with customers and tangibly demonstrate its ongoing security commitment