Imagine, you set up your perfect DevSecOps pipeline.
A developer pushes source code to a favorite source version control system, which activates a Continuous Integration (CI) pipeline, starts security testing, and it finds a critical vulnerability, which in turn stops the entire pipeline and the developer goes fixing the issue. Perfect, the security toolchain has just prevented a disaster.
You have just saved the world (!) from releasing a vulnerable version of the product.
The perfect outcome is to block the CI pipeline until addressing all the identified issues.
You integrate a tool of your choice into the CI/CD pipeline, it gives you a ton of false positives, you start triaging, developers are irritated, they can't wait because the release of the product is scheduled for yesterday, you "temporarily" disable the integration until you can clean up the security testing report, and it never ends.
Let's take SAST as an example. SAST means a Static Application Security Testing tool.
Security scan is usually required at the moment of Pull Request verification or on the master branch commit. You don't want to pass the vulnerable code to the release.
However, what if it finds 1000 false positives (wrong findings)? What if takes 1 day to finish the analysis? How does the tool decide that the detected vulnerability is critical, how does it decide the finding is an actual vulnerability at all, what are the criteria for stopping the pipeline? How long does it take to analyze the code?
At the end of the day, it's often hard to deliver scan results in a synchronous way.
I would say, there is a challenge of synchronous vs asynchronous way of security testing.
Why it's often hard to deliver security scan results into CI/CD pipeline in a synchronous (blocking) way?
- Legacy code with numerous issues (with a huge SAST output).
- “Unclean” source code may result in a lot of security findings blocking a pipeline entirely until fixing/dismissing all of the findings.
- Scanning time.
- Different SAST tools have different efficiency in terms of time and findings.
- A huge code base may lead to slow scans (from 1 to a few hours).
- What we also found is that on-the-cloud solutions usually have a little bit slower and/or unpredictable scan time (depending on the SLA of the vendor).
- SAST tool limitations.
- Some tools have either rich or poor capabilities to report the scan result, e.g. a tool may have a true positive result having no means to report the scan result back immediately.
- SAST tools may have specific requirements for the project structure and build steps.
- Triaging: tools always generate false positives, there should be a way to conveniently fix and dismiss the security finding by a developer with
cross-validation by a security specialist. The challenge is having the pipeline which supports the business (not completely blocking the business).
- Budget compromises.
- There are sophisticated tools on the market, however, budget limitation may play a major role in what the team can afford.
- Filtering commits.
- Scans usually run only for commits and PRs to master and release branches.
A Staged Integration Plan
In practice, the solution here is to carry out a multi-stage SAST integration approach.
- Stage 1: Baselining.
Tools are purchased, a source code is initially scanned, false positives are reviewed/dismissed. Tickets are created, vulnerabilities are fixed and submitted.
- Stage 2. Initial asynchronous integration.
SAST is integrated into a CI/CD pipeline, however, scan results don’t block the pipeline (yet). The team executes an exercise of triaging and fixing the issues.
- Stage 3. Synchronous integration.
SAST blocks the pipeline in a case of critical findings (according to the security policy). In the ideal scenario, a Jira ticket is automatically created.
The last step is quite challenging because of false positives and because of all the challenges mentioned above. With some tools, e.g.
gosec for Golang, synchronous integration is easily achieved and is the only way of dealing with the commit (the tool is very quick and doesn’t produce a lot of output usually).
Having more sophisticated on-the-cloud tools (Coverity, Fortify-on-Demand, Veracode, etc.), we go with a hybrid model where we have asynchronous SAST scans for less critical projects, and synchronous scans in more critical cases wherever it’s possible (if it is technically and organizationally possible).
We have to come up with a few levels of issue severity. The simplest categorization is the following:
- Severity 1 (Highest) implies the quickest mitigation and release (1-2 days) for the most critical issues.
- Severity 2 (Middle) means no direct impact and implies slower mitigation (say, 30 days).
- Severity 3 (Lowest) with 3 months for mitigation.
Different classification schemes may use up to 5 severity types implying different severity properties.
There are 2 major cases to pass the build to the release:
- All issues are fixed.
- The security team manually allows the release of an urgent hotfix in case SAST blocks an urgent hotfix with minor findings. It happens from time to time. Usually, tools provide an ability to allow the build with a comment from a security specialist (appsec, security lead) why the build is allowed to be released.
Do integrate security tools incrementally. Triage issues semi-manually as a baselining to get adapted to the tools of your choice. Work with teams, gain knowledge and evolve with your system.