An Innovative Approach to Quantifying Uncertainty in Early Lifecycle Cost Estimation

Results and Future Research

QUELCE as an approach to early cost estimation is unprecedented in many ways. The SEI spent much of the past year developing and refining the analytical methods used. So far, trials of the earlier steps of the method have been conducted in workshops, and post hoc reviews of previous estimation artifacts were used for later steps. The SEI’s experience and the results achieved thus far suggest that the approach has considerable merit. Feedback about the value of the approach from the participants in workshops and from leaders in estimation research has been very positive.

Empirical validation of QUELCE is ongoing, and the results of this evaluative research will be used to refine the approach and demonstrate its value. Future efforts will benefit from the participation of programs that are willing to provide access to the artifacts developed prior to Milestone A or to use the QUELCE method in upcoming Milestone A estimates. Through these joint efforts, the SEI will evaluate the extent to which the probabilistic methods proposed improve the accuracy and precision of cost estimates for DoD programs.

Conclusion

Extensive cost overruns have been endemic in defense programs for many years. A significant part of the problem is that cost estimates for unprecedented systems must rely heavily on expert judgments made under uncertain conditions. QUELCE aims to reduce the adverse effects of that uncertainty. Important program change drivers and the dependencies among them that may not otherwise be considered are made explicit to improve the realism and likely accuracy of the estimates. The basis of an estimate is documented explicitly, which facilitates updating the estimate during program execution and helps others to make informed judgments about their accuracy. Variations in the range of possible states of the program change drivers that may occur under different likely scenarios are explicitly considered. The use of probabilistic methods combining Bayesian belief systems and Monte Carlo simulation will ultimately place the cost estimates within a more defensible range of uncertainty.

References

1. Ben-Gal, Irad. “Bayesian Networks” in Encyclopedia of Statistics in Quality and Reliability. John Wiley & Sons, 2008.
2. Boehm, B., Abts, C., Brown, A. W., Chulani, S., Clark, B. K., Horowitz, K., Madachy, R., Reifer, D. & Steece, B. Software Cost Estimation with COCOMO II. Prentice Hall, 2000 (ISBN 0130266922).
3. Browning, Tyson R. “Applying the DSM To System Decomposition And Integration Problems – A Review and New Directions.” IEEE Transactions On Engineering Management 48, 3 (August 2001): 292-306.
4. Ferguson, R.,  Goldenson, D., McCurley, J., Stoddard, R., Zubrow, D. & Anderson, D. Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE) (CMU/SEI-2011-TR-026). Software Engineering Institute, Carnegie Mellon University, December 2011.
5. Mooney, Christopher Z. Monte Carlo Simulation (Quantitative Applications in the Social Sciences). Sage Publications, 1997 (ISBN: 978-0803959439).
6. Naval PoPs Guidebook Version 2.2., Guidance for the Implementation of Naval PoPS. A Program Health Assessment Methodology for Navy and Marine Corps Acquisition Programs, 2010.
7. Sangal, Neeraj; Jordan, Ev; Sinha, Vineet; and Jackson, Daniel. “Using Dependency Models to Manage Complex Software Architecture,” OOPSLA ‘05 Proceedings of the 20th annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications; ACM New York (2005): 167-176.
8. Software Engineering Measurement and Analysis (SEMA) Cost Estimation Research Group: Ferguson, Robert; Goldenson, Dennis; McCurley, James; Stoddard, Robert; Zubrow, David; Anderson, Debra.Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE). (CMU/SEI-2011-TR-026). Software Engineering Institute, Carnegie Mellon University, 2011. http://www.sei.cmu.edu/library/abstracts/reports/11tr026.cfm

Footnotes

1. This work was originally described on the SEI blog in a two-part series, “A New Approach for Developing Cost Estimates in Software Reliant Systems.”