DevSecOps: Dos & Don'ts, Automation and Strategies for Success

For organizations that rely on releasing new product updates daily or weekly, building security into the development pipeline is huge.

However, following DevSecOps is rarely a clear-cut path, wrought with stakeholder resistance, misalignment between teams and budgetary constraints. To learn how to navigate these challenges, we held a Q&A with two experts: Caroline Wong and Larry Maccherone.

DevSecOps Dos and Don’ts

DO tie security feedback into the ‘pull request’ workflow.

Engineers care most about their recent work — the longer you wait, the less invested they are in fixing issues.

Most modern engineering teams have switched to a ‘pull request’ model for their workflow. A pull request is an automated set of checks developers use to review new code. If it contains serious issues, developers must resolve them before merging it into the project.

According to Larry, tying security feedback into this process is the most valuable step you can take to enable your DevSecOps transformation. By incorporating security checks before code makes it into production, engineers are naturally more motivated to quickly fix identified issues. The faster they fix their code, the sooner it can be acknowledged.

DON’T assume developers can use tools and practices designed for security specialists.

Most engineers aren’t security experts. If you try to implement tools or practices that require them to think like security specialists, you’re unlikely to see much uptake.

Instead, identify solutions that fit more easily with engineering workflows and mental models. Automated scanners that are fast enough to fit into a pull request — and provide outputs that engineers understand — are an excellent place to start.

DO learn the ins and outs of your organization.

Caroline points out a simple, yet important fact:

The purpose of security is to protect the value your organization creates.

To do that, learn as much as possible about its business model. Even further, aim to understand how engineering links to that model, how it develops software on a day-to-day basis, anything that can guide you on how to make security work with this specific team.

DevSecOps Questions

How To Transition from Manual to Automated Testing?

Larry uses one question to identify how mature an engineering team’s automated testing capabilities are:

“For what percentage of your products do you trust that the automated testing will invalidate a dangerous or bad artifact before it gets to production?”

If the response is close to 100%, the organization has made substantial progress towards DevSecOps. If the percentage is much lower, the team has a way to go.

Larry suggests beginning with a pipeline that includes automated unit and functional testing for teams that are just starting. If these simple tests aren’t in place, more complex controls won’t add value because engineers won’t have tight feedback loops to motivate them. Instead, the security team will run the process, and results will be “thrown over the wall”.

Of course, you can’t automate everything.

Here’s an example Caroline pointed out: User Acceptance Testing (UAT) and story level testing aren’t suited to automation. They require opinion and judgment, which computers can’t provide.

Since you don’t have a limitless budget for all kinds of checks, you can follow the BSIMM recommendation:

1. Risk rank all products in your portfolio; and,
2. Decide what level of defect discovery to require for each level of risk.

For example, you could use DAST/SAST scanning and manual pentesting for your most critical applications, while you keep medium risk applications to SAST scanning. Pentesting should also act as the check to see what went past your initial testing, and where you need to make improvements in the process.

How to Win Over People Who Don’t Value Security?

There will always be people who understand the value of DevOps, but think security is not as important.

To win them over, work with your sales team to show that security is more than an internal concern — it’s a customer demand. This type of feedback often emerges from vendor security questionnaires and third-party risk management processes.

Caroline also suggests aiming to reach an agreement on these three points:

1. Prevent the same application security defects from occurring over and over again.
2. Reduce the probability that bad actors can stop critical applications from functioning.
3. Fix bugs for which well-known attacks exist.

At the same time, try to hook into your development teams’ natural desire for engineering excellence. They want to be proud of their work, and that’s difficult if they know it includes security vulnerabilities.

At Comcast, Larry leads DevSecOps transformation one engineering team at a time. He starts with a workshop for each team, and he’ll only go ahead if the product owners and managers are present. If they aren’t, he sends everybody away and reschedules.

How to Track and Communicate Success with DevSecOps?

You could use thousands of metrics to track DevSecOps success. However, Larry and Caroline suggest keeping things simple.

For Larry, the most critical measures are:

• Has the engineering team been through the initial workshop?
• Do they have an active plan to implement DevSecOps processes?
• When the team adopts new processes, do they successfully improve maturity?

Caroline suggests another approach. DevSecOps transformation happens for a reason, so ask yourself: Why did you start it in the first place?

• Are you trying to speed up your time to a first result?
• Are you trying to respond to customer feedback more quickly?
• Are you trying to generate more data and use it to improve your applications?

Any metric you choose must match the purpose behind your DevSecOps transformation.

How to Build Successful DevSecOps Teams?

DevSecOps success is dependent on your engineering team’s ability to take ownership of security processes. For this, you need engineers who have specific characteristics. For instance, they must:

1. Be on board with the DevSecOps transformation initiative.
2. Have a mindset of curiosity and humility.
3. Have the ability to learn quickly.
4. Be able to see the big picture, but also ‘zoom in’ and fix specific issues.

Larry suggests finding 1-2 well-respected engineers to co-lead your DevSecOps transformation alongside a security champion. Aim to have as many engineers as possible on your transformation team — understanding you’ll need to teach them security practices — and assign them to two roles:

1. DevSecOps transformation coaches, who train engineers on what they need to do differently and help them implement it.
   
   
2. Pipeline technical engineers, who set up and implement self-service tools to help engineers adopt security practices.

The Future of DevSecOps

Caroline and Larry identified two significant trends that will play a critical role in the future of DevSecOps:

The DevSecOps Conductor

In any transformation, people become more specialized. The more that happens, the greater the need for a ‘conductor’ to keep everything (and everybody) operating effectively. The conductor’s ability to keep things moving in the right direction is arguably the most critical factor in long-term DevSecOps success.

Overcoming resistance

Often, the toughest opponents of DevSecOps are mid-level managers responsible for a domain that will be drastically altered by the transformation. These people often worry their teams will become redundant or that changes will cause problems for them personally.

Your ability to overcome this resistance will be paramount to your organization’s success in DevSecOps.

Check out the full webinar here: