Estimation Accuracy vs. Estimation Usefulness
How accurate is the average software cost estimate? Industry statistics vary as widely as the estimates they seek to measure. One oft-cited study – the Standish Group’s Chaos Report – concludes that only one third of software projects deliver the promised functionality on time and within budget1. A later IEEE study2 noted several gaps in the Standish Group’s criteria for estimation accuracy:
…the [Standish] definitions don’t cover all possibilities. For instance, a project that’s within budget and time but that has less functionality doesn’t fit any category. … The Standish Group’s measures … neglect under runs for cost and time and over runs for the amount of functionality.
When studies rely on different definitions of estimation success or failure, we should expect their assessments of estimation accuracy to exhibit considerable variability. The existence of different standards raises an intriguing question: what makes an estimate “accurate”?
Most quality control measures for estimates compare estimated cost/effort, schedule, or scope to their actual (final) values. The problem with this formulation is that “accurate” estimates are an integral part of feasibility decisions made very early in the project lifecycle; long before anything but the most generic information about the system’s intended features or use can be known with reasonable certainty. The technologies used to implement the requirements may be unknown and the schedule, team size, required skill mix, and project plan have yet to be determined. As design and coding progress, the list of unknowns grows shorter and decreasing uncertainty about the estimation inputs lowers the risk surrounding the estimated cost, schedule, and scope. Unfortunately, most organizations must make binding commitments before detailed and reliable information about the project is available.
Given the degree of uncertainty surrounding early estimates – and the correspondingly broad range of possible time/effort/scope combinations – estimation accuracy may be less important than estimation usefulness. In an article for the ACM, Philip Armour explores the difference between these two concepts3:
The commitment is the point along the estimate probability distribution curve where we promise the customer and assign resources. This is what we need to hit, at least most of the time. It is not a technical estimation activity at all but is a risk/return based business activity. It is founded on the information obtained from the estimate, but is not the estimate. Using Figure 3 as an example, if we needed an accurate commitment in the earliest (Initial Concept) phase based on how the diagram shows the project actually worked out, we would have had to commit at around a 75% probability. From the figure, committing to the “expected” result at Initial Concept would have led to a significant overrun beyond that commitment, and the project would have “failed.” We can consider the 50% (expected) result to represent the cost of the project and the 25% increment to the higher commitment level to represent the cost of the risk of the project.
Measures of estimation accuracy that treat an estimate as “wrong” or a project as “failed” whenever the final scope, schedule, or cost differ from their estimated values penalize estimators for something outside their control: the uncertainty that comes from incomplete information. We should measure deviations between estimated and actual project outcomes because this information helps us quantify estimation uncertainty and account for it in future estimates. But if measurement becomes a stick used to punish estimators, they will have little incentive to collect and use metrics to improve future estimates.