Outcome-Based Acquisition

Opportunity

The Government Auditing Office (GAO) Study of leading companies [6] found that “software developers and acquirers all use three fundamental management strategies to ensure the delivery of high-quality products that are on time and within budget:

1. Working in an evolutionary environment,
2. Following disciplined development processes,
3. Collecting and analyzing meaningful metrics to measure progress.”

Figure 2: Enablers for Outcome-Based Acquisition

These management strategies also form the basis of the DoD’s Data and Analysis Center for Software (DACS) Gold Standard for Software Acquisition [7].

The availability of 1) the 2008 updated Inspection Standard, 2) Computerized Inspection tools in 2009, 3) the Inspection Compliance Matrix in 2010 along with mitigation of the ten most common  Inspection pitfalls in 2007 now provide the capabilities to implement Outcome-Based Acquisition (figure 2).

We will examine these capabilities and see how they converge to provide an auditable and actionable outcome-based acquisition process for delivering high quality products on time and within budget.

1) IEEE Inspection Standard 1028™-2008 (Section 6) [3]

The management strategy of having a disciplined development process – is satisfied by the latest IEEE Inspection Standard which was updated and released in August 2008. The standard specifies formal software Inspections in enhanced and verifiable processes, guidelines, and measurements.

Before continuing, we need to understand the role of Process Assessments (e.g., CMM®, CMMI®).  A report of the Defense Science Board [8] observed that “Process Assessments by themselves do not examine the outputs of any development effort and are therefore silent with respect to the quality attributes of any particular product. A positive Process Assessment finding lowers the risk that an organization will produce a low quality product but the [actual] quality of the product itself must be assessed usingother methods.”

It is well established that the most effective Product Assessment method available is formal Inspections. Inspections are used to examine work products, such as requirements, design, and code, during development. “The data in support of the quality, cost, and schedule impact of [formal] Inspections is overwhelming. They are an indispensable part of engineering high-quality software” [9].  And formal Inspections are also a DACS Software Acquisition Gold Practice [10].  Inspections complying with the 2008 updated Inspection Standard are the most rigorous and effective means of ‘peer review’, a generic phrase representing a wide variety of review techniques and resulting effectiveness.

IEEE Inspection Standard 1028™-2008 states that the purpose of Inspections is to detect and remove software product anomalies and Inspections are systematic peer examinations that do one or more of the following:

1. Verify that the software product satisfies its specifications
2. Verify that the software product exhibits specified quality attributes
3. Verify that the software product conforms to applicable regulations, standards, guidelines, plans, specifications, and procedures
4. Collect software engineering data (for example, anomaly and effort data)
5. Provide the collected software engineering data that may be used to improve the Inspection process itself and the activities used to produce software products
6. Use the [software engineering] data as input to project management decisions as appropriate

The 2008 Inspection Standard provides comprehensive and detailed criteria against which to measure Inspection execution compliance – a key element of a disciplined development process.  However, while formal Inspections have been shown to be highly effective in a disciplined development environment, they must have the proper infrastructure support and upper management’s ongoing adherence to Inspection pitfall prevention techniques to be successfully implemented and sustained.  Upper management’s responsibilities for mitigation of Inspection pitfalls can be found in our earlier CrossTalk article [11].

Software Inspections are critical for outcome-based acquisition.   Inspections are the best monitoring / “other” method referred to in the Defense Science Board report [8] for ensuring, throughout development, that:

• success is being achieved
• the software will be capable of performing the required functions
• the software is delivered according to schedule, including short-term milestones

A development environment with defined end-state requirements including defined increments accommodates using Inspections to verify requirements, design, system interfaces and other defining material (e.g., Statements of Work). These up-front activities are where the majority of defects (~80%) are historically introduced into a software product [12, 13].