Best Practices for AppSec Tool Integration in CI/CD Pipelines

Development organizations continue to shift left to implement security earlier in the continuous integration/continuous delivery (CI/CD) pipeline. But software security group (SSG) leaders need to know where application security (AppSec) tools should go in the CI/CD workflow and their purposes in different phases. Fortunately, best practices can help them make these decisions by answering some essential questions:

• How do you pick the right AppSec tools for your CI/CD pipeline?
• Where should you integrate AppSec tools in the pipeline?
• How should you configure these tools?

How do you pick the right AppSec tools?

In the CI/CD pipeline, picking the right AppSec tool often depends on the phase. In the early phases of the pipeline, a static AppSec testing (SAST) tool can scan proprietary code for programming errors and design flaws that could lead to exploitable weaknesses. The phase for SAST can also depend on the features the tool offers and the structure of the organization’s process. For example, if the CI/CD pipeline enables IDE plugins in the pre-commit phase, a lightweight SAST tool at this point will enable developers to scan their code in real time as they code.

Later on, a software composition analysis (SCA) tool can scan source code and binaries to identify known vulnerabilities in open source and third-party components. For example, if the organization’s CI/CD pipeline has the integration for an SCA tool, it can be configured to automate scanning as part of the build phase. The SCA tool can show alerts for open source AppSec, license, and use policies or break the build.

Then a dynamic AppSec testing (DAST) tool can perform automated testing in the middle of the CI/CD pipeline, similar to a penetration test. For example, at the test phase, the CI/CD pipeline can integrate a DAST tool that uses existing quality assurance (QA) test suites to check the application’s URLs or crawl the application to collect the URLs and then run parallel tests at the application instance.

Constraints on tool selection

Selecting an AppSec tool for a CI/CD pipeline will be subject to constraints. Some constraints on tool selection include the following:

• Default rulesets and ability to customize rulesets. The default rulesets for tools may not catch all the vulnerabilities important to the development organization. Better known as false negatives, these happen when real security bugs are not covered by the tool’s existing rules and analysis. So there is a need to be able to write custom rules to catch these vulnerabilities. In addition to false negatives, default rules can cause false positives. So the organization may want to toggle off those default rules. Low-risk vulnerabilities and informational findings should also be disabled if the organization does not generally address them. Default rules that benchmark against popular lists (e.g., OWASP Top 10) can produce false positives such as trust boundary violation, path traversal, and various types of injection weaknesses (e.g., command, LDAP, XPath).
• Language, framework, and API support. Because languages (e.g., JavaScript, C++, Python), frameworks (e.g., Node.js, Spring, Hibernate), and APIs (e.g., RESTful, GraphQL) change all the time, the ability of an AppSec tool to support new ones is very important in CI/CD pipelines.
• Available integration with the build system. It’s critical that an AppSec tool can integrate with the CI/CD build environment. Otherwise, there’s potential for pushback from the DevOps team (e.g., fear of false positives, decreased development velocity, increased friction from a more manual process, too long to deploy, need to switch between tools). It’s also important that the tool can integrate with developer notification tools (e.g., Slack, email), defect tracking systems (e.g., Jira), command-line interfaces (CLIs), and metrics dashboards.
• IDE plugins for the tools. With the emphasis on shifting left growing stronger every day, AppSec tools must offer plugins to show developers security issues directly in their preferred IDE, so they don’t have to switch to a different tool to fix them.

Some other considerations in tool selection: Is the tool cloud native? Can it be containerized and licensed for containers? We’ll address these topics in a future article.

(Figure 1)

Where should you integrate and run AppSec tools in the CI/CD pipeline?

In a generic CI/CD pipeline (Figure 1), AppSec tools are typically integrated and run in different phases of the workflow depending on the type of tool and the activity in that phase. SAST tools should be integrated in the earlier phases of the CI/CD pipeline: pre-commit, commit, and build. In the pre-commit phase, a SAST tool should be available as a plugin to perform real-time scans as developers type to enable them to code securely. And technology such as pre-commit hooks can help stop bad code from being checked into the central code repository.

In the commit phase, an automated check will ensure that code can be compiled and built at all times. Then use a SAST tool to perform an incremental scan on any new code that was introduced since the last code check-in. If it detects any critical or high-risk issues, it will break the build and alert the team.

At the build phase, run a deeper-level SAST scan with expanded rulesets. For example, the tool could check for the OWASP Top 10 and any rules for custom frameworks or other risks that may not have been covered by comprehensive rules.

Also, use an SCA tool at the build phase of the pipeline to catch known open source and third-party vulnerabilities and license risks. The build environment can provide dependency information to the SCA tool to augment open source scanning. Then the data from both sources can be combined into a complete and accurate open source bill of materials (BOM).

For the test phase, use a DAST tool to scan web applications and APIs, if present. Configure the DAST tool to run rulesets for common critical and high-severity issues. And in the test phase, run more extensive SAST operations as an out-of-band activity or asynchronously in the CI/CD pipeline. If an unexpected number of vulnerabilities is found, this should trigger an out-of-band manual code review using penetration testing. Interactive AppSec testing (IAST) tools can also be used in the test phase.

How should you configure AppSec tools?

For different phases of the CI/CD pipeline, AppSec tools should be configured to find different security issues. For example, during the pre-commit phase, a SAST tool plugin should be configured to catch security issues that have zero false positives, such as SQL injection and cross-site scripting (XSS). Concentrating on security problems with no false positives at this stage will give developers confidence that the tool is working properly. The SAST tool plugin in the CI/CD pipeline (Figure 2) should provide just-in-time guidance to developers to help them remediate security issues while they are in the code. If too many vulnerabilities are found, the plugin can trigger an out-of-band activity, such as threat modeling or architectural risk analysis. 

(Figure 2)

At the commit phase, a SAST tool should scan for the organization’s list of top 10 security issues. For example, organizations should look for issues such as command injection and hard-coded credentials during this phase. Some organizations may need to conduct SAST scans in the commit phase for HIPAA or PCI DSS compliance. Any critical or high-severity risks found should break the build and alert the relevant teams.

As you move further to the right in the CI/CD pipeline, SAST scans take longer and cost more. For example, in the build phase, longer SAST scans will include the full OWASP Top 10 issues for web applications and any customized rules for web services, REST API services, or custom frameworks. The test can be run in parallel with other activities.

(Figure 3)

Also in the build phase, an SCA tool (Figure 3) should be configured to scan for open source and third-party vulnerabilities, especially those at the critical and high-risk level that have high impact and are highly exploitable. For example, one of the most common open source vulnerabilities currently found across many codebases is CVE-2019-11358, which concerns an improper input validation vulnerability discovered in jQuery, according to the 2020 OSSRA report. An attacker could exploit this vulnerability to execute cross-site scripting (XSS) attacks, trigger a denial-of-service (DoS) condition, or gain unauthorized access to the application. No exploit has been published as of this writing, and a fix is available. An appropriate SCA tool could help you identify this vulnerability throughout your application portfolio and tell you how to remediate it.

(Figure 4)

DAST scans (Figure 4) should be completed during the test phase of the CI/CD pipeline. They can be configured as active scans or passive scans in the background as the application runs. Scan to identify security vulnerabilities, security misconfigurations, and other common issues detailed in the OWASP Top 10 and the new CWE Top 25 (e.g., improper input validation, integer overflow/wraparound). If you’re testing APIs, you can configure a DAST tool to scan only the APIs that have changed. A DAST tool can detect vulnerabilities in APIs, including those on mobile back-end servers and IoT devices, as well as RESTful APIs. DAST scans for PCI DSS and GDPR audit and compliance should also occur in the test phase.

Benefits of AppSec tool integration in the CI/CD pipeline

The benefits of AppSec tool integration in the CI/CD pipeline are greater the further you shift left in the process. At the very beginning, in the pre-commit phase, SAST tools that can give feedback directly to developers in the IDE help reduce the cost and time to fix security issues. SCA tools at the build phase alert developers to open source components that have known vulnerabilities and potential fixes. And while they occur in the middle of the CI/CD process, DAST tools can give feedback on runtime errors that SAST tools typically do not. Overall, AppSec tools integrated directly into a CI/CD pipeline can give SSG leaders some level of assurance that developers and the security team have found all reproducible security issues in the codebase.

What AppSec tools are working best in your CI/CD process? Please share your experiences with the other members so we can all learn collectively.