Changes to the community structure and the menu bar
Significant changes have happened to the structure of our community and our drop down menu bar. READ ABOUT IT HERE
Protect Your Assets
cancel

SCA: Various approaches for including dependency source code during scans

SCA: Various approaches for including dependency source code during scans

Respected Contributor... ruudsenden Respected Contributor...

Introduction

An application may be using some proprietary frameworks or internally developed 'common' modules. Usually such common modules are built separately, and are included as binary dependencies when building the main application. This means that the source code for such common modules is not available while building or scanning the main application.


If there are no (custom) rules available for such modules, SCA will be unable to take the behavior of such modules into account while scanning the main application. For example, SCA cannot identify taint sources or sinks within such modules, or track taint flow through such modules. By including the source code for such modules during the scan, SCA can perform a more thorough analysis, usually resulting in better scan results.

This article describes several possible approaches for including the source code for such common modules during an SCA scan. This article is initially focused on Maven-based build systems to benefit from the Maven dependency resolution, but some approaches are also suitable for use with other build systems. In general, the following approaches are described:

  • Approach 1: Use the Maven dependencies plugin to download dependency source bundles.
  • Approach 2: Deploy an SCA mobile build session alongside each (common) module in your Maven repository or in some other shared location, and include such build sessions while scanning the application.
  • Approach 3: Use custom functionality to download the source code for some dependencies from your source code repository.

Approach 1: Download dependency source bundles from Maven repository

Assuming the source bundles for your dependencies have been deployed to your maven repository, you can use the Maven dependencies plugin to download and unpack those source bundles before running a scan for your application. You would usually build a batch file or shell script that performs the following:

  1. Clean-up resources from previous build/scan (remove dependency sources/libraries, sourceanalyzer -clean, remove FPR file, ...)
  2. Translate sources for current project:
    mvn clean install com.fortify.ps.maven.plugin:sca-maven-plugin:translate -Dfortify.sca.buildId=<BUILD_ID> -Dmaven.test.skip=true
  3. Get source code for specific dependencies:
    mvn org.apache.maven.plugins:maven-dependency-plugin:unpack-dependencies -Dclassifier=sources -DincludeScope=compile -DoutputDirectory=<absolute depsources output directory> -DincludeGroupIds=<included group id's> -DexcludeGroupIds=<excluded group id's> -Dmdep.failOnMissingClassifierArtifact=false
  4. Get all dependency binaries:
    mvn org.apache.maven.plugins:maven-dependency-plugin:copy-dependencies -DincludeScope=compile -DoutputDirectory=<absolute deplibs output directory> -Dmdep.failOnMissingClassifierArtifact=false
  5. Manually add dependency sources to the intermediate model:
    sourceanalyzer -b <BUILD_ID> -jdk 1.6 -extdirs <absolute deplibs output directory> <absolute depsources output directory>/**/*
  6. Run a manual scan:
    sourceanalyzer -b <BUILD_ID> <SCAN_OPTS> -scan -f <filename>.fpr
  7. Optionally upload results to SSC using fortifyclient or the SCA Maven plugin.

Advantages

  • Quite simple to implement.
  • Works quite well in many situations.
  • Works independent of which Continuous Integration environment is being used.
  • You can easily include sources for public Maven dependencies to be included in your scan (for example for libraries that are not supported by the standard Fortify rule packs).


Disadvantages & notes

  • Only usable for Maven-based projects or other build systems with similar dependency resolution mechanism.
  • Requires source bundles for common modules to be deployed to your onsite maven repository.
  • Depending on your project setup, it may be difficult to define and manage the correct inclusions and exclusions while unpacking source bundles in step 3. You will need to define how to differentiate between modules for the current project, common modules to be included in the scan, and any modules to be excluded from the scan. The example in step 3 uses groupId's to differentiate between these. Some things that you need to keep in mind while managing inclusions and exclusions:
    • The dependency sources downloaded by Maven may include any available source bundles for the (sub)-modules in your current project, causing some source files to be translated twice (once in step 2, once in step 5). This will result in duplicate class definition warnings from SCA, and may cause SCA to analyze some source files that are our of date.
    • You may want to exclude some other (common) modules from the scan even though their sources are available in the maven repository.
  • The source bundles may not always contain all required files to be included in the scan. You should check for example that the source bundles include generated sources and non-Java resources (JSP's, XML files, ...).
  • Dependency sources need to be re-translated for every application.
  • Dependency sources are translated directly, i.e. no build process is invoked. If you have an advanced build process for some of your common modules, this solution may not be sufficient.

Approach 2: Deploy SCA mobile build sessions in Maven repository

With this approach, the SCA translation phase will be run during the regular build job for all common modules that may be used as dependencies for other projects. The result of the translation phase is then bundled into a mobile build session (mbs) file and deployed to the onsite Maven repository along with other project artifacts.

Using a batch file, shell script or maven profile you would usually do something like the following to translate any common modules:

  1. Clean-up resources from previous build (sourceanalyzer -clean, ...)
  2. Perform a regular build with integrated SCA translation, for example:
    mvn clean install com.fortify.ps.maven.plugin:sca-maven-plugin:translate -Dfortify.sca.buildId=<BUILD_ID> -Dmaven.test.skip=true
  3. Export the mobile build session for the translated sources. This can be done using the sourceanalyzer command line, for example:
    sourceanalyzer -b <BUILD_ID> -export-build-session <BUILD_ID>.mbs
    A feature request has been filed to allow the SCA Maven plugin to be used to export the mobile build session (bug #44272). Once implemented this would allow the translation and export to be easily integrated into a Maven profile.
  4. Deploy the MBS file together with other project artifacts to your onsite Maven repository, using 'mbs' as the artifact classifier. You can do this using the command line or by integrating the deployment step in a Maven profile, both using the standard Maven deploy plugin.

Now that mobile build sessions are available for all common modules, the build jobs for any applications that list such modules as dependencies can easily use the build session artifacts to perform their scans:

  1. Clean-up resources from previous build (remove directory with previously downloaded MBS files, sourceanalyzer -clean, ...)
  2. Perform a regular build with integrated SCA translation, for example:
    mvn clean install com.fortify.ps.maven.plugin:sca-maven-plugin:translate -Dfortify.sca.buildId=<BUILD_ID> -Dmaven.test.skip=true
  3. Optionally, export a mobile build session for this project and store it alongside the MBS files downloaded in step 4. This may make the processing in step 5 slightly easier.
  4. Download all available MBS files for all dependencies listed by the current project, for example:
    mvn org.apache.maven.plugins:maven-dependency-plugin:copy-dependencies -DincludeScope=compile -Dclassifier=mbs -Dtype=mbs "-DoutputDirectory=<MBS_DIR>" -Dmdep.failOnMissingClassifierArtifact=false
  5. Perform a scan that includes both the <BUILD_ID> specified in step 2, and all MBS files downloaded in step 3. A feature request has been filed to allow passing MBS files directly to the scan command (bug #44271). Until this has been implemented, you will need to perform the following before running the scan:
    • Loop over all downloaded MBS files in <MBS_DIR>
    • Import each MBS file using: sourceanalyzer -import-build-session <MBS-FILE>
    • Determine the list of build id's imported from the MBS files, and use these to build the scan command line.
    • Tips:
      • Use the -Dcom.fortify.sca.ProjectRoot=<UNIQUE_WORK_DIR> command line argument when importing build sessions and during the scan. This will cause SCA to store the intermediate models from the MBS files in the specified directory, isolating them from any other translate and scan jobs.
      • Either process the output of 'sourceanalyzer -show-build-ids -Dcom.fortify.sca.ProjectRoot=<UNIQUE_WORK_DIR>' or simply iterate over the directories in <UNIQUE_WORK_DIR>\sca<version>\build to determine all imported build id's.
      • Write the '-b <build-id>' option for each discovered build id to a scan-args.txt file, and include this file during the scan using the '@scan-args.txt' command line argument. This is easier to implement (especially in Windows), helps with testing your implementation, and avoids command lines that are too long to be processed by Windows if you need to include many build id's.
  6. Optionally upload results to SSC using fortifyclient or the SCA Maven plugin.

A feature request has been filed to allow steps 4 and 5 to be performed together automatically using the SCA Maven plugin (bug #44273). Since this will be difficult to implement, it is unlikely that this will be implemented any time soon.

If you see a benefit in any of the before-mentioned feature requests, please send an email to fortifytechsupport@hp.com to express your interest in these features; please include the bug number(s) in your email. This will allow HP to update the list of customers interested in such features, and notify you about any updates.

Advantages

  • Intermediate model for all (common) modules is easily available via standard Maven functionality.
  • Translation for (common) modules only needs to be performed once while building that module; projects that reference those modules can re-use the intermediate model.
  • Dependency sources can be translated using regular build integration, and can be optimized for each individual project.
  • If necessary, the customer can also deploy MBS files for some third-party open-source frameworks to their Maven repository, and optionally include these in their scans (i.e. for frameworks that are not supported by current Fortify rule packs).

Disadvantages & notes

  • Only usable for Maven-based projects or other build systems with similar dependency resolution mechanism.
  • Until the feature requests mentioned in this section have been implemented, this approach may be somewhat more difficult to implement than some of the other approaches, although most steps can be automated using a batch file or shell script.
  • You may need to define inclusions and exclusions for the Maven dependencies plugin in step 3 to identify which MBS files should be included/excluded for the scan.

Approach 2, alternative 1: Deploy SCA mobile build sessions as build job artifact

This approach is very similar to approach 2, with the exception that the mobile build session is not stored in the onsite Maven repository but instead as build job artifacts. For example, when building common modules in Jenkins, you would use the 'Archive Artifact' option to archive the MBS file as a build job artifact. Using for example the Jenkins 'Copy Artifact' plugin, you can then copy any required MBS files to the application project workspace and include those during the scan.

Advantages

  • Intermediate model for all (common) modules is easily available via standard CI functionality.
  • Translation for (common) modules only needs to be performed once while building that module; projects that reference those modules can re-use the intermediate model.
  • Dependency sources can be translated using regular build integration, and can be optimized for each individual project.
  • If necessary, the customer can also make MBS files for some third-party open-source frameworks available, and optionally include these in their scans (i.e. for frameworks that are not supported by current Fortify rule packs).
  • Does not require an onsite Maven repository for storing mobile build sessions.
  • Can be easily applied to non-Maven projects.

Disadvantages & notes

  • Dependent on functionality provided by Continuous Integration (CI) environment.
  • Until the feature request to directly include MBS files during a scan has been implemented (bug #44271), you will need to manually import the mobile build sessions and determine the corresponding build id's to include in the scan.
  • No automatic dependency management. You will need to explicitly configure which MBS files need to be copied to the current workspace to be included during the scan.
  • No automatic version management. If different applications use different versions of common modules, MBS files for each individual module version will need to be available. You could use different build jobs for each module version (including the version number in the build job name for example), or include the version number in the file name for the archived MBS file names.

Approach 2, alternative 2: Deploy SCA mobile build sessions in shared location

Again very similar to approach 2, with the exception that all MBS files are stored in some shared location. The MBS file names would use a standardized format that includes module name and version number.

Advantages

  • Intermediate model for all (common) modules is easily available in a shared location.
  • Translation for (common) modules only needs to be performed once while building that module; projects that reference those modules can re-use the intermediate model.
  • Dependency sources can be translated using regular build integration, and can be optimized for each individual project.
  • If necessary, the customer can also make MBS files for some third-party open-source frameworks available, and optionally include these in their scans (i.e. for frameworks that are not supported by current Fortify rule packs).
  • Can be used if no CI system is available (i.e. for scans on developer workstations), or if CI system does not (sufficiently) support the functionality for storing/retrieving artifacts.
  • Does not require an onsite Maven repository for storing mobile build sessions.
  • Can be easily applied to non-Maven projects.

Disadvantages & notes

  • Requires setting up and maintaining a shared location that is accessible for all users and systems that may want to run a scan with dependency source code included.
  • Until the feature request to directly include MBS files during a scan has been implemented (bug #44271), you will need to manually import the mobile build sessions and determine the corresponding build id's to include in the scan.
  • No automatic dependency or version management. You will need to explicitly configure which MBS files need to included during the scan. However you could possibly use for example the Maven dependencies plugin to generate a list of project dependencies, and map these to MBS file names to be included during the scan.

Approach 3: Download dependency source code from source code repository

Within the build job for your application project, you download additional source code for any dependencies that you want to include during the scan from your source code repository. You can then translate those modules in addition to the current project modules, and scan everything together.

Alongside the source code for each (common) project, you can optionally store a script or similar functionality in your source code repository to perform a translation for that specific project. This functionality can then be separately tested and optimized to perform a full translation for that  project. The build job for the application project simply calls the scripts provided by each of their dependencies to perform the translation.

This approach would look somewhat like this:

  1. Check out main application source code from SCM.
  2. Translate the main application project using any SCA build integration approach.
  3. Determine dependencies for which source code needs to be included during the scan.
  4. Check out the source code for these dependencies from SCM.
  5. Translate each of these dependencies using the translation script included with the source code for that dependency.
  6. Perform an SCA scan that includes both the intermediate model for the main application, and all intermediate models for translated dependencies.
  7. Optionally upload results to SSC using fortifyclient or the SCA Maven plugin.

Advantages

  • Dependency sources can be translated using regular SCA build integration options, and can be optimized for each individual module (when storing a project-specific translation script alongside the source code for each module).
  • Usable for any build system (maven, ant, ...), even if different projects use different build systems.
  • Works independent of which Continuous Integration environment is being used.

Disadvantages & notes

  • Source code for dependencies needs to be checked out and re-translated for every build of the main application, requiring more disk space, causing more load on your SCM system, and build jobs take longer to execute.
  • No automatic dependency management. For each project you will need to define which SCM URL's to use to check out source code for dependencies. One could possibly use for example the Maven dependencies plugin to generate a list of dependencies, then map each dependency to an SCM URL to download the corresponding sources.
  • No automatic version management. If you change the version of the binary dependency deployed with your application, you need to make sure to change the SCM URL from which the dependency source code is checked out accordingly.
  • No assurance that the dependency source code you check out corresponds to the binary version that is deployed with the application, possibly resulting in inconsistent scan results.
Comments
Absent Member.

Awesome write-up.  Just to add the basic steps that I have used for mobile build sessions / cloudscan have been as follows:

export FORTIFY_SSC_URL="http://localhost:3030/ssc"

export FORTIFY_CSC_URL="http://localhost:8080/cloud-ctrl"

export BUILD_TAG=abc-dskjflsk-7823124

export FORTIFY_SSC_CSCTOKEN="c0ca568a-b9dd-4643-9d57-35f0a71e441b"

export WORKSPACE="/Applications/HP_Fortify/HP_Fortify_SCA_and_Apps_4.30/Samples/basic/eightball"


sourceanalyzer -b ${BUILD_TAG} -clean

sourceanalyzer -b ${BUILD_TAG} -debug -verbose -jdk 1.6 ${WORKSPACE}/*.java

sourceanalyzer -b ${BUILD_TAG} -export-build-session ${BUILD_TAG}.mbs

cloudscan -url ${FORTIFY_CSC_URL} start -upload -versionid 51 -uptoken ${FORTIFY_SSC_CSCTOKEN} -mbs ${BUILD_TAG}.mbs -scan

 Frequent Contributor..

Will there be a difference in scan results between Approach 1 & 2?

Approach one uses the effective POM whereas in approach 2, the dependent MBS files are generated from its immediate dependencies.

Does it matter if the versions of the transitive dependencies are different?

For example, the parent pom could resolve ESAPI to 2.1.0.1 where as the child modules or dependent artifacts could be specifying 2.0.x.

Also the MBS files need to be translated with the same SCA version as they are scanned with, isn't it? Will it be a maintenance problem with storing the MBS files in a source code repository or in a shared location? How do we resolve this when we migrate to higher SCA version?