The State of Product Security

From waterfall, to agile, to shift left, product security teams are continually looking for ways to keep up with the phenomenal rate of product development. To find out where product security currently stands, we held a panel of four security experts at our recent virtual conference, Sec Talks.

The speakers were:

• Josh Bressers (moderator), Head of Product Security at Elastic
• Doug DePerry, Director of Defense at Datadog
• Nazira Carlage, Director of Product Security at Salesforce
• David Lenoe, Director of Secure Software Engineering at Adobe

This article is a summary of topics covered during the panel. To watch a full recording, scroll to the end.

What is Product Security?

“It used to be very simple to say this is a product, it was something you took, it was in a box. Now, where do we draw the line with the shift to more ‘as a Service’ models? How do we define this wild world of product security?”

Josh Bressers kicked off the panel by posing a question. What is product security?

The first thing to understand about product security is that it varies hugely depending on the product.

Enterprise vs. SMB. Hardware vs. software. SaaS vs. on-prem. Depending on the customer served and the delivery model used, product security can look completely different at every stage — from design through manufacturing, delivery, and maintenance.

For products that are owned and operated by the customer, patching is a huge factor. To be effective, the product security team must:

1. Create and push security patches quickly; and,
2. Effectively sell the need for patching to the customer.

Ultimately, if customers don’t apply patches, the product won’t be secure.

And that’s just one example. As Doug DePerry explained during the session:

“Product security is kind of in the eye of the beholder. For me, product security encompasses application security as well as ensuring the product you’re securing is safe to the end user, security configurations make sense, it uses secure defaults, security configs are well-documented, and so on.”

David Lenoe continued:

“I started doing security when we were focusing on on-premise or desktop products. At that point, product security was easier to define because it was anything you sent to the customer. As we moved into the cloud, it all started to blend together. Now, product security can encompass application security, operations security, and enterprise security.”

How Does Security Change Based on the Product?

Every product, regardless of customer base and delivery model, needs to be secure and bug free. And naturally, some of the processes needed to achieve this are very similar.

However, as stated earlier, product security can vary widely.

In SaaS products, for example, patching is much easier to implement. Typically, all customers are running the same version of the product, and a single patching run can be enforced each time a security update is needed. This takes away the need for a great deal of careful communication and prompting, which has historically been a requirement for product security.

Meanwhile, with cloud native apps, there is usually a lot of reliance on third-party infrastructure. Many apps rely on services provided by the major cloud platforms, as well as third-party libraries, code sourced from code repositories, and commercial solutions. In effect, the creators of these products become third-party vendors, and must rigorously vet the security of both internal and — as far as possible — external code.

In some ways, hardware products are among the simplest to secure. The product security team has more of an opportunity to serve as ‘gatekeepers’, and more security control over the final product. However, patching once again becomes an issue, and product security teams are often forced to spend a lot of time explaining risk to customers and convincing them to apply patches promptly. It often pays to adopt creative solutions to this problem, such as enabling auto-updates and sandboxing by default.

What About ‘Shifting Left’?

In recent years, there has been a lot of talk about shifting left — an approach where security testing is completed earlier in the product development lifecycle. Many software vendors have adopted this approach to product security, because it helps to ensure that bugs are identified and fixed quickly.

During the session, Doug DePerry explained why shifting left is appropriate for product security, but less needed for more traditional security functions:

“Areas like infrastructure, by definition, are centralized. It’s easier to put gates around them or rely on role-based access control and things of that nature. When you have hundreds or thousands of developers all contributing small pieces of code to a larger application, that is much more difficult to manage or secure in a centralized manner. The rate of change is very high, and there are a lot more variables.”

Under these circumstances, it’s desirable to identify bugs quickly and frequently, rather than uncovering dozens of bugs later on during the product testing phase.

However, as valuable as shifting left can be, Nazira Carlage was quick to point out the importance of a light touch:

“When I think about shifting left, it’s about ‘how do we bring visibility early?’. It’s not about blocking somebody or slapping them on the hand early. It’s bringing visibility so they can make changes that will impact the security posture of the product early enough to make a difference.”

“We need to enable engineering to do the right thing. If they see that, for example, the library they’re trying to use or the way they’re trying to use infrastructure is not secure, they’ll change it.”

Product Security Team Structure

When it came to team structure, each of our panelists had their own experiences.

David Lenoe highlighted the value of consolidating security teams of different disciplines under the remit of the CSO. From his experience working at very large organizations, this approach helps to break down silos and ensure all security teams — including product security — can work together effectively.

Meanwhile, Nazira Carlage identified one of the key decisions to make when it comes to team structure: whether product security should be part of engineering. While this can be a valuable arrangement — particularly for building trust and relationships — it has drawbacks. Most notably, if the product isn’t performing well, security is likely to be one of the earliest budget items to be cut. On the other hand, while keeping product security under the control of the CSO avoids this problem, it also presents challenges when it comes to building and maintaining trust between the two functions.

The New Culture of Security

All our panelists emphasized the role of culture — both within the security function and across the organization.

Historically, security teams of all sorts were separate and spent most of their time saying “No” to other areas of the organization. Modern product security teams don’t have this luxury — they must build strong, lasting relationships with engineering, and work to understand the business and technical demands on product development. If this is done consistently, the team will build credibility within the organization.

As Nazira Carlage explained:

“If you don’t understand how your product is used, deployed, maintained, and end-of-lifed, you can’t be successful in security. You can’t even make sensible suggestions. We have to get better at understanding the whole lifecycle of how a product is developed and used before we can be effective security advisors.”

Watch a full video of our ‘Where Product Security Fails and Wins’ panel:

Where Product Security Fails & Wins from Cobalt.io on Vimeo.