Piloting Software Assurance Tools in the Department of Defense

In this article, we present and describe the JFAC Enterprise Software Licensing Pilot program activities during the 2016 fiscal year. During this period, JFAC provided limited quantities of Software Assurance tools to users in the DoD with an aim of evaluating how the use of these tools could improve the state of software assurance within the Department. The four Software Assurance tools procured and made available to the end-users consisted of a dynamic web testing tool, two static source code analyzers and an origin analysis tool that checked for vulnerabilities in third party libraries and components used to build software. After the licenses had been available to the licensees for nearly a full calendar year, researchers from the Software Engineering Institute (SEI) at Carnegie Mellon University conducted an outreach and survey effort to solicit feedback on user experiences. The results of the tools’ effectiveness are presented as well as findings on the impact use of the tools had on the software development process.

JFAC Enterprise Software Licensing Program Objectives

To increase the security posture of Department of Defense (DoD) software systems, the employment of Software Assurance techniques needs to shift from being part of a step in the development process to being an integrated element of the entire development process. Specifically, Software Assurance must be engineered into the entire lifecycle of applications and not just be something that is checked in a phase before deployment. The inclusion of Software Assurance testing earlier in the development process in parallel with development efforts, often referred to as “Shift Left”, is keenly effective when frequent and automated tests are performed during development phases to provide timely feedback to developers.

To promote a more mature approach to Software Assurance within the DoD, the Joint Federated Assurance Center (JFAC) established an Enterprise Software Licensing Pilot program in 2016. The goals of this program are to provide enterprise-wide licenses to the DoD development community for Software Assurance tools, to promote wider use of such tools, to provide training and expertise to engineers and developers on how and when to best use these tools, and to simplify the acquisition of Software Assurance tools by systems and software engineers [JFAC 2016].

During the first year of the JFAC Enterprise Software Licensing Pilot program, JFAC provided limited quantities of Software Assurance tools to users in the DoD with an aim of evaluating how the use of these tools could improve the state of software assurance within the Department. Specifically, during this pilot phase, JFAC procured limited quantities of four commercially available Software Assurance tools and provided them to selected DoD constituents at no direct cost to the product users. The product users of these tools were largely members of the Army, Navy, and Air Force in a near even distribution and a small number of selected other DoD personnel.

The four Software Assurance tools procured and made available to the end-users consisted of a dynamic web testing tool, two static source code analyzers, and an origin analysis tool that checked for vulnerabilities in third party libraries and components used to build software. The tools selected for use in the JFAC Enterprise Software Licensing Pilot program were selected in part based on their reputation in the industry, proliferation in the marketplace, and results of prior research and studies. On the whole, the tools proved to be capable enterprise class products as they supported and were implemented on a diverse set of operating systems, programming languages, and platform targets.

The software licenses for these tools were contracted and procured in bulk by JFAC. The JFAC Coordination Center (JFAC-CC), a subcomponent of JFAC, was then responsible for disseminating the individual licenses to each of the product users and also serving as an intermediary between the product vendors and the product users. Licensees were granted a license by request and approval of a representative from their organization. Licensees were a combination of individuals who specifically requested use of the product and those nominated by their organization to be pilot participants. In total, 248 licenses were distributed across the four product offerings. Approximately 66 different programs, projects, or organizations within the DoD received licenses through the JFAC pilot program.

After the licenses had been available to the licensees for nearly a full calendar year, researchers from the Software Engineering Institute (SEI) at Carnegie Mellon University conducted an outreach and survey effort to solicit feedback on user experiences.

Study Design

A formal online survey was developed by researchers at the SEI and distributed to each of the licensees. Additionally, an outreach program was conducted whereby all licensees were personally contacted via email and/or telephone and interviewed for any additional feedback, as well as reminded to complete the online survey. As part of the agreement to use the software license, pilot participants had to agree to complete a survey at the end of the license year.

The survey questions were the same for all four products. The survey questions focused on the technical environment the tool was deployed in, the effectiveness and perceived value of the tool, and the usability of the tool. Usability was measured using a modified version of the System Usability Scale (SUS), a widely used instrument that provides a quick and reliable indicator of usability [Brooke 1996].

49 out of 115 distributed surveys were completed. These 115 survey candidates represented 248 licenses that were distributed during the pilot program. In some cases, the same program, project, or organization was issued licenses for more than one of the Enterprise Software Licensing Pilot program licenses. In these cases, the participant had to fill out one survey for each product that they were licensed. For various reasons, the sample size may be less than 49 in some cases of individual questions examined below. These reasons include applicability of response, lack of response (some questions were optional), or question was not presented to respondent due to conditional survey logic. On the other hand, some survey questions allowed more than one response so the total responses for that question can be higher than the number of respondents.

Impact of Software Assurance Tools on Software Quality

Survey respondents were asked to indicate the total number of lines of code they scanned with each Software Assurance tool. The cumulative number of all lines of code scanned as indicated on the responses was 22,442,902. These lines of code represented 1,391 unique projects or applications. Note that not all respondents answered these two questions. These totals are based on just the respondents who answered these questions.

The actual number of lines of code scanned during the JFAC Enterprise Software Licensing Pilot program can be estimated based on these responses. A simple approximation of the number of lines of code scanned during the pilot, based on extrapolating known figures, would be 49,181,213. Likewise, the number of projects or applications scanned during the Pilot was approximately 9,121.

Table 1: Projects and Lines of Code Scanned in FY2016 Pilot

	Licenses Issued	Survey Respondents	Reported Projects Scanned (per Respondents)	Reported Lines of Code Scanned (per Respondents)	Approximate Actual Projects Scanned (extrapolated)	Approximate Actual Lines of Code Scanned (extrapolated)
Dynamic Code Analysis Tool	24	11	16	8,029,000	35	17,517,818
Static Code Analysis Tool “A”	31	15	243	9,805,861	502	20,265,446
Static Code Analysis Tool “B”	19	10	521	4,357,041	990	8,278,378
Origin Analysis Tool	174	14	611	251,000	7,594	3,119,571
PILOT TOTALS	248	49	1,391	22,442,902	9,121	49,181,213

Note that some caution should be taken when examining the results for the origin analysis tool in particular for two reasons. First, this product had a much lower percentage of survey respondents per licenses issued, so there is a greater chance for variance between actual figured reports and extrapolated figures. Further, for an original analysis tool, the number of projects scanned is a more relevant scanned than lines of code scanned.

Issue Detection

Survey respondents were asked to identify how many total issues, warnings, and/or vulnerabilities each tool identified. In total, the tools identified 419,189 issues as reported in survey submissions. Extrapolating this figure out across all license holders, the tools likely helped identify 866,697 issues during the Pilot. It is important to remember that not all issues identified by the tools are actual items to be addressed. Potential issues can often be determined to not be applicable for various reasons and the triaging and handling of such issues is another important feature of Software Assurance tools.

Table 2: Issues Detected in FY2016 Pilot

	Issues Detected	Approximate Actual Issues Detected (extrapolated)
Dynamic Code Analysis Tool	102	223
Static Code Analysis Tool “A”	49,838	102,999
Static Code Analysis Tool “B”	363,344	690,354
Origin Analysis Tool	5,905	73,391
PILOT TOTALS	419,189	866,697

Issue Correction

Survey respondents were asked questions about whether issues, warnings, and/or vulnerabilities discovered using each tool caused them to take corrective actions or make plans for corrective actions. The first question in this area was whether respondents thought that the issues detected were valid issues that need addressed. 100% of respondents using the static code analysis tool “B” and the dynamic code analysis tool thought the discovered issues were valid and needed addressed. Nearly all respondents for the static code analysis tool “A” felt similarly, while just over ½ the respondents for the origin analysis tool thought the discovered issues warranted attention.

Figure 1: Did the tool find meaningful issues that need addressed?

Respondents were then asked further if they thought the discovered issues were in need of immediate attention, meaning the issues posed a risk of some urgent importance. Nearly ¾ of the dynamic code analysis tool and the static code analysis tool “A” respondents felt some of the detected issues required immediate attention while more than ½ of the static code analysis tools “B” respondents felt the same. 40% of the origin analysis tool respondents thought the detected issues required immediate attention.

Figure 2: Did the tool find issues that you felt required IMMEDIATE attention?

Having found the detected issues, respondents were asked if they had actually implemented any corrective actions to address these issues. Across all tool products, about 40% of respondents indicated that they had already initiated some corrective action.

Figure 3: To date, have you fixed/addressed any issues, warnings, and/or vulnerabilities as a result of the tool feedback?

While perhaps not yet initiated, respondents were asked if they had any future plans to implement any corrective actions to address the detected issues. 80% of respondents for the origin analysis tool indicated they had already made plans to correct issues detected and the other 20% indicated they were considering making such plans (“Maybe” response). For the static code analysis tool “B”, only 40% of respondents indicated they had already made plans to correct issues detected, while the other 60% indicated they were considering making such plans. For the static code analysis tool “A”, just over ½ the respondents indicated they had already made plans to address discovered issues and approximately 30% more indicated they were considering making such plans, while about 15% indicated they had no plans to correct detected issues. Almost exactly ½ of the dynamic code analysis tool respondents indicated they had already made plans to address discovered issues and another ¼ indicated they were considering making such plans, while about 25% indicated they had no plans to correct detected issues.

Figure 4: Are there future plans to fix or address any issues, warnings, and/or vulnerabilities as a result of the tool feedback?

Impact of Software Assurance Tools on Development Processes

Beyond making changes to correct specific issues that were detected by each tool, survey respondents were asked whether use of the tools had prompted them to make any changes to their development processes. Across each of the tools, about 20% to 40% of respondents indicated that the tool use had prompted them to make changes in their development process that had already been implemented. About the same number of respondents (20% to 40%) for each tool indicated they had future plans to change their development processes as a result of tool use (Figure 6). However, another 20% to 50% of respondents indicated they were still considering (“Maybe” response) making changes to their development processes as a result of tool use.

Figure 5: Have you made changes to your design, development, or build processes as result of having used this tool?

Figure 6: Are there future plans to make changes to your design, development, or build processes as result of having used this tool?

Respondents were asked, subjectively, if they felt that each tool is effective at finding meaningful issues, warnings, and/or vulnerabilities in their projects and applications. For the origin analysis tool, the dynamic code analysis tool, and the static code analysis tool “A”, roughly 75% of respondents felt that the tool was effective in finding meaningful issues. For the static code analysis tool “B”, only 40% of respondents felt the tool was effective, despite it having been reported to find a significant amount of potential vulnerabilities during the Pilot.

Figure 7: Do you think this tool is effective in finding meaningful issues, warnings, and/or vulnerabilities in your projects and applications?

Respondents were asked whether they would like to continue using each tool in their development and testing processes. 100% of the dynamic code analysis tool respondents indicated that they would like to continue using the tool. For the static code analysis tool “A”, 85% of respondents indicated that they definitely would like to continue using the tool, while the other 15% said they “maybe” would like to continue using the tool. For both the static code analysis tool “B” and the origin analysis tool, 60% of respondents indicated that they would like to continue using the tool, while the other 40% of respondents said they “maybe” would like to continue using the tool.

Figure 8: Would you like to continue using this tool in your projects and applications?

Usability of Software Assurance Tools

Respondents were asked to complete a ten-question questionnaire regarding the usability of each tool. The usability was measured using a modified version of the System Usability Scale [Brooke 1996], which is a commonly used scale for garnering “quick measures” of usability, and is noted to be reliable with small sample sizes.

In general, the average score for a system measured on the SUS is 68. However, it is recommended to normalize scores within a given study to control for environment. In the SUS usability measurements for the Enterprise Software Licensing Pilot program, the dynamic code analysis tool fared the best with an average score of 70.63. The origin analysis tool (61.5) and the static code analysis tool “A” (61.43) had similar average scores, while the static code analysis tool “B” had the lowest average score at 46. Given the prior comment on normalization, the rank order of each product’s usability with respect to the other products is more telling than the raw score.

Figure 9: Average SUS usability score for each tool

JFAC Enterprise Software Licensing Program Effectiveness

The JFAC Enterprise Software Licensing Pilot program provided limited quantities of Software Assurance tools to users in the Department of Defense with an aim of evaluating how the use of these tools could improve the state of Software Assurance within the Department. The immediate impact of this pilot was that nearly 50 million lines of code were scanned and approximately 867,000 issues or potential issues were detected. 86% of respondents to the survey felt the issues were meaningful and needed addressed, and more than ½ of respondents thought these detected issues required immediate correction.

So, in a vacuum, the Pilot program achieved success by detecting numerous issues and driving corrective action on a selected set of systems. However, the larger impact delivered by the Pilot is in the way in which the introduction of Software Assurance tools promoted a more mature approach to Software Assurance throughout the development process. More than 25% of survey respondents indicated that they have already made changes to their development and testing process as a result of using the tools in the Pilot program. Another 35% of respondents indicated they are considering making such changes in their development and testing processes. Collectively, more than 50% of the users of the piloted Software Assurance tools not only are making changes to their codebases, but are also making changes to their processes as a result of this program.

Changes and improvements to these processes are likely to yield a much more exponential impact than just the changes to specific lines of code, resulting in greater savings of time, money, and other resources. However, changes in these development processes will require continuous use of Software Assurance tools in the processes. This is reflected in the sentiments of survey respondents as more than 75% of respondents indicated that they felt the tools were useful in finding meaningful issues and more than 75% of respondents wished to continue use of these tools.

While not specifically studied, it is noted anecdotally that no performance issues were reported with the products either. So, in general, the tools proved capable and worthwhile.

Given the effectiveness of this Pilot program, an expansion of the program would have a significant impact on the security of software systems within the DoD. Publicly available data on non-DoD software systems shows that 84% of software breaches exploit vulnerabilities at the application layer [Clark 2015], while funding for information technology (IT) defense versus software assurance is 23-to-1 [Feimann 2014]. It is likely similar numbers exist for DoD software systems and that greater attention to Software Assurance would yield a more mature software security posture for these systems. The potential savings and increase in the programs security posture is significant given the fact that between 1% and 5% of defects discovered in deployment are potentially exploitable vulnerabilities [Woody 2014] and the cost of fixing defects is 10 times more costly to fix after coding and 100 times more costly to fix post deployment [Subramaniam 1999].

Areas for Future Work

There are several areas related to this Pilot program that would be worthwhile areas for further investigation. First, as a substantial number of survey respondents indicated they planned to make changes to their development process as a result of using these tools, it would be beneficial to explore which changes to the process they made or were planning to make. Taking that topic a step further would be to investigate which changes to a development process are most effective for Software Assurance.

A further topic for investigation would be studying the best places in the development process to utilize Software Assurance tools. That is, where should tools be used to get the “best bang for the buck”? Additionally, it should be explored which tool or combination of tools is most effective for certain types of projects and systems.

Lastly, an interesting topic to broach is the impact the usability of a tool has on its perceived value in finding issues. As noted in this study, the static code analysis tool “B” scored very well in terms of finding meaningful/valid issues, yet users were lukewarm on continued use of this product. One reason for this may be the low usability scores the tool received. Is there a correlation between usability and perceived effectiveness and value in Software Assurance tools? Or was this merely a spurious finding? There is a rich canon of study usability of tools and adoption rates, but it would be interesting to explore this relationship specifically for Software Assurance tools.