Quantify Risk To Manage Cost And Schedule

Too many projects suffer from unachievable budget and schedule goals, caused by unrealistic estimates and the failure to quantify and communicate the uncertainty of these estimates to managers and sponsoring executives. Some projects are beginning to use Monte Carlo methods to improve cost and schedule estimating; however, it is not always done in a systematic manner, mutually understood by both customer and supplier. This article provides a systematic approach based on quantification of expert judgment in a pragmatic and consistent fashion.

Projects often fail to predict or accommodate the risk and uncertainty of budgets and schedules in a rigorous or structured manner. This is of particular consequence in the "better, faster, cheaper" era of program management. The risk is seldom quantified in a manner that the estimators, the management hierarchy, and the customer mutually understand, accept, and acknowledge. Furthermore, projects are often planned to "best-case" even though few of the participants, particularly the managers and sponsoring executives, recognize and understand the implications of a best-case, "green-light" plan. Although it may indeed be prudent to work to such a best-case plan, managers and executives fail to concede that the probability of achieving the "best-case" goal is by its very nature zero [1]--and that faster and cheaper is also riskier. This failure inevitably results in misunderstandings and unrealistic goals.

Some projects, as a result of such unrealistic goals and with pressure from an uninformed customer, are driven to establish harsh policies that serve to undermine the very objectives they hope to achieve. Methodical and rigorous quantification of the uncertainty of cost and schedule estimates is key to interpreting, and realistically managing and mitigating, cost and schedule risk. This, in the end, will lead to improvement of overall project management.

EXPERT JUDGMENT

Expert judgment is typically the crux of cost and schedule estimates, but in the spectrum of the risk management process, quantification of expert judgment is the weakest area. Transitioning from English language statements of experts to the mathematical expressions required by analytical tools is done inconsistently if done at all. Professional program planners are beginning to apply statistical methods to schedule and cost analysis in an attempt to deal with this problem.

This article offers a methodology for statistically quantifying the risk to cost and schedule resulting from the uncertainty [2] of the estimates that underlie any cost-schedule analysis. Some simple rules for cost-schedule risk mitigation are postulated that provide a structured focus for the methodology offered. These rules provide the expert with a logic basis that is fundamental to consistent and sensible quantification of the risk elements. These rules are:

* Plan "best case" and preclude implementing a self-fulfilling prophesy.

* Budget "most likely" and recognize real-world risk and uncertainty.

* Protect for "worst case" and acknowledge the conceivable.

Crucial to implementation of these rules is the credibility of the best-case estimates. They must be honest and truly achievable in the best-case scenario; unrealistic targets cannot drive them. Only then can a credible quantification of risk be applied. On this basis, a specific approach to interpreting expert judgment and quantifying cost and schedule risk is offered.

STATISTICAL ANALYSIS

The term "risk" implies a stochastic (probabilistic) process, and quantification requires a model. Such a model can be developed by ascribing to each element of the cost estimate, or the schedule projection, a probability distribution function (PDF) representing the likelihood of completing the particular cost element or scheduled task at a specific cost or for a specific duration. Monte Carlo simulation techniques can then be applied to the model to forecast the entire range of possible end results.

A simple triangular distribution is a reasonable PDF for describing the risk or uncertainty for a cost element or task duration estimate. Its structure is based on the minimum possible cost and duration (plan best case), the most likely cost and duration (budget most likely) and the maximum possible cost and duration (protect for worst case) as illustrated in Figure 1.

The height of the triangle is 2/(max-min) such that its area is unity. The parameters are simple, intuitively easy to comprehend, and amenable to a mathematical formulation compatible with cost and schedule models and fast Monte Carlo analysis. Other more complex distributions could be used such as the Beta or Weibull, but little if anything is gained, [3] and the intuitive simplicity of the triangular distribution is lost. The triangular PDF will form the basis of the quantification of risk offered here.

QUANTIFYING JUDGMENT

The "expert" is first asked to provide a green-light, best-case estimate of each cost or schedule element in the model, and then is asked to provide an assessment of risk (his or her unsureness) associated with the estimate for each element. Table 1 relates the risk adjectives of low, moderate, high, and very high to a set of risk factor multipliers.

The table has been developed to facilitate and guide the expert in risk estimation. These factors, when multiplied by the best-case estimate, are the parameters of the triangular PDF for a particular element of cost or schedule based on its associated risk.

This table is intended to provide a reasonable range of risk quantification that fits well with the perceptions and experience of engineers and estimators. It is based on the author's experience as a project manager, planner, proposal evaluator, and as an expert providing cost and schedule estimates for many years and on many projects. The table has also been used by engineers at the Naval Research Laboratory to develop cost and schedule estimates for spacecraft projects. The engineers adapted readily to the table and were content with the adjectives and the range of the associated multipliers. The author is unaware of any other reference that would serve to substantiate (or refute) Table 1. It is suggested that an approved standard for these multipliers be developed that is generally accepted and consistently applied industry-wide. Table 1 can be used in the interim and can serve as a point of departure for a more widely accepted standard.

These particular risk factor multipliers are arithmetically derived from a few basic assumptions, which provide a structure and logic to the risk-factor multipliers that may facilitate the debate and development of a standard. They are as follows:

* For an element of a cost or schedule estimate coded low risk, "max" (worst-case performance) is defined as 10% greater than "ruin" (best-case performance) and "most likely" is defined as 4% greater than "mm." This reasonable premise for a low-risk element provides a base for the derivation.

* The "most likely" set is then derived as a geometric progression of the percent increase for the eight risk codes with a common ratio of 1.5 (e.g., 6% = 1.5 x 4%, 9% = 1.5 x 6%, and so forth as the risk increases from low to very high).

* For an element coded as "very high+," "max" is capped at 200% greater than "min." The rationale for this cap is that no prudent manager would allow a growth beyond three times plan without intervention and mitigating action.

* The common ratio (1.534) for the "max" geometric progression is then derived from the first and last element of the set (e.g. 10% and 200%).

The multiplier factors associated with the hierarchy of risk approximate a geometric progression. Therefore, as the risk increases, the probability distribution becomes more asymmetrical on the "max" side, as we intuitively would expect. An additional degree of flexibility is provided by the inclusion of the "+" categories.

These factors, illustrated graphically in Figure 2, provide a convenience to the estimator; their use will provide a degree of consistency from estimator to estimator and from estimate to estimate.

RISK FACTOR ATTRIBUTES

Below, typical attributes are suggested that characterize the risk factors. These attributes illustrate situations that could be a basis and substantiation for a particular risk evaluation. Risk factor attributes can be tailored for a particular project and perhaps they could be expanded as part of establishing an approved standard; however, care should be taken not to develop rigorous cookbook bureaucratic tests. We must accept that in the end all estimates are judgments, hopefully made by experienced individuals who are motivated and unfettered in their task. Most expert evaluators will rely on their own experience as guided by Table 1.

Low-Risk ATTRIBUTES

As applied to design tasks. Existing proven designs are used extensively; requirements are well defined and readily achieved; development effort is minimal; and an innovative approach materially simplifies design implementation.

As applied to production. Extensive use is made of proven hardware or software produced by previous suppliers; exotic processes and tooling are not required for production; materials and parts are readily available; and an innovative approach materially simplifies production.

As applied to test and verification. Extensive use is made of proven hardware and software produced by previous suppliers; alignments and calibrations are not critical; test tools and equipment are readily available; and an innovative approach materially simplifies testing; performance demands are reasonable and have been realistically suballocated; design provides significant margin above the requirement; achievement of the design margin is not precipitously crucial to mission success; and an innovative approach materially simplifies accomplishment of the mission requirement.

VERY HIGH RISK ATTRIBUTES

As applied to design. Extensive use is made of new and unproven designs; requirements are poorly defined and unlikely to be achieved; development effort is extensive; and an "innovative" approach materially complicates the design.

As applied to production. Extensive use is made of unproved hardware or software never previously produced; many exotic processes and undeveloped exotic tooling are essential for successful production; materials and parts are not in production, require development, are in short supply, or otherwise are not currently available through normal vendor procurement; an "innovative" approach materially complicates production.

As applied to test and verification. Extensive use is made of unproved hardware or software never previously produced; most alignments and calibrations are difficult and critical to performance; exotic test tools and equipment are essential, not readily available, and require development; designs result in unstable platforms, timing interfaces, and electrical outputs; an "innovative" approach materially complicates testing; the allocated performance budget is unrealistic; the design provides no significant margin above the requirement; achievement of the design margin is precipitously crucial to mission success; and an innovative approach materially complicates accomplishment of the mission requirement.

MODERATE AND HIGH-RISK ATTRIBUTES

A grade of "moderate" or "high" is based on the evaluator's judgment, considering the risk extremes as defined for "low-risk" and "very high risk." Attributes will range from modification of existing design or catalog design to new designs and high technology.

INTERPRETING THE RESULTS

The composite results of a Monte Carlo analysis of a series of cost elements in a hypothetical project will be a probability forecast such as that illustrated in Figure 3. In concert with the risk mitigation rules cited at the beginning of this article, the project manager would initially allocate a budget to his project elements totaling about $110 million, while retaining about $8 million as his total operating management reserve.

Furthermore, he would alert his customer to the possibility that if all goes bad, the project could cost as much as $128 million. The customer may elect to take action to protect his funding allocation for this worst case. The analysis will reveal the high-risk cost elements so that they can receive the special management attention warranted.

The corresponding Monte Carlo probability forecast of a series of inter-dependent schedule elements for our hypothetical project might be similar to that in Figure 4. The schedule risk is quantified and illustrated for all to see and better understand the uncertainties of the project. Based on this analysis the project manager would take action necessary to accommodate both the possibilities of an early delivery or a late delivery. The Monte Carlo schedule risk analysis may well reveal that the critical path does not determine the most likely delivery date as commonly assumed. A noncritical path that has much higher risk may drive delivery. Quantification and identification of these high-risk paths will direct and help focus management attention to the truly critical program elements. [4]

CONCLUSION

Cost and schedule risk mitigation can best be done if the risk is quantified. Building detailed cost and schedule models are always formidable tasks, and adding the complexity of risk estimates can be a significant extra effort. But with the simplified, pragmatic approach suggested here, such quantification of risk is a practical and productive effort. The computational tools and the software to support the methodology suggested are currently available. The benefit of quantified risk is best illustrated by the wealth of management information clearly communicated by the previous two summary forecast charts. Forecasts, made for projects at the Naval Research Laboratory, such as the interim control module (ICM) for the NASA space station, have provided a management recognition and comprehension of the impact of uncertainty and risk associated with schedule and cost projections. All managers will be well served by using such data.

Frederick W. Raymond retired from government service and the Navy Space Program in 1986. At the time of his retirement he was serving as spacecraft acquisition manager for the Navy Space Special Projects Office. Previously, he was extensively involved in the design development and production of Navy satellite systems as deputy head of the Spacecraft Technology Center of the Naval Research Laboratory. He has more than 30 years' experience in space technology and procurement management. He was a founder and vice president of Welkin Associates, Chantilly, VA. In August 1997 he returned to the Naval Research Laboratory and is now a senior advisor at the Naval Center for Space Technology. He holds a B.A. degree and an M.S. degree in physics from Syracuse University.

ENDNOTES

(1.) Cost and schedule estimates are made up of many independent elements. If each element is planned as best case-- say with a probability of achievement of 10%--then the probability of achieving best case for a two-element estimate is 1%; for three elements, .01%; and for many elements, infinitesimal. In effect, it is zero.

(2.) No attempt will be made in this article to distinguish between risk and uncertainty. Risk involves uncertainty but it is indeed more. For purposes of this article it is unimportant. The effect is combined into one statistical factor we call "risk," which will be described by a single probability distribution function.

(3.) The difference that may result from a Monte Carlo forecast of a triangular PDF and a forecast using a corresponding Beta or Weibull PDF is most certainly masked by the accuracy of their estimated parameters.

(4.) This article addresses the forecast of possible outcomes of cost and schedule and not the system or program impact. With these forecasts managers can anticipate and plan for possible impacts.

Table 1.
Risk Factor Multipliers
Code            Min  Most Likely  Max
Low         L    1      1.04      1.10
Low+        L+   1      1.06      1.15
Moderate    M    1      1.09      1.24
Moderate+   M+   1      1.14      1.36
High        H    1      1.20      1.55
High+       H+   1      1.30      1.85
Very high   V    1      1.46      2.30
Very high+  V+   1      1.68      3.00

[Graph omitted]

[Graph omitted]

[Graph omitted]

[Graph omitted]

COPYRIGHT 1999 Defense Acquisition University Press
COPYRIGHT 2004 Gale Group