The Impact Of The Packard Commission's Recommendations On Reducing Cost Overruns On Defense Acquisition Contracts

Using data from selected acquisition reports, Drezner et al. (1993) show that reform initiatives from 1960 to 1990 did not reduce cost growth on 197 defense programs. The average cost growth on these programs was 20 percent and did not change significantly for 30 years. Using data from the Defense Acquisition Executive Summary data base, we show a similar result. Initiatives based on the recommendations of the Packard Commission did not reduce the average cost overrun percent experienced on 269 completed defense acquisition contracts evaluated over an 8-year period (1988 through 1995). In fact, the cost performance experienced on development contracts and on contracts managed by the Air Force worsened significantly. Although many factors contribute to poor cost performance, estimation error is a casual factor identified in each study.

President Ronald Reagan established the Packard Commission in 1986 to reduce inefficiencies in the defense procurement system. [1] Although the commission examined defense management practices in general, it focused on the acquisition process. The commission concluded that the primary problems with the acquisition process were the same ones identified in previous decades (cost growth, schedule delays, performance shortfalls). It recommended streamlining the acquisition process, increasing tests and prototyping, changing the organizational culture, improving planning, and adopting the competitive firm model where appropriate. Like the problems, the recommendations were strikingly similar to reform efforts of the past and to those of the present (Dews, Giles, Barbour, Harris, and Hesse, 1979; Gates, 1989).

With this research we set out to test the effectiveness of the Packard Commission's recommendations on reducing defense cost overruns. As similar initiatives are identified and implemented today, it is important for policy makers to understand the effectiveness of past policies. Based on a review of 269 completed defense contracts, we found that the Packard Commission's recommendations did not reduce cost overruns. This result is consistent with similar research involving an analysis of cost growth on 197 defense acquisition programs (Drezner, Jarvaise, Hough, & Norton, 1993). Despite the implementation of more than two dozen regulatory and administration initiatives, there has been no substantial improvement in the cost performance of defense programs for more than 30 years.

LITERATURE REVIEW

Unplanned cost increases in defense procurements can escalate to staggering amounts and can adversely affect resource allocation decisions, especially when defense budgets are decreasing. The Department of Defense (DoD) has not ignored this problem. As shown in Table 1, Drezner et al. (1993) identify several important regulatory and administrative initiatives to improve defense cost performance. The expectation was that as these initiatives were implemented, cost performance would improve through time.

Unfortunately, these initiatives did not improve defense cost performance. Using data from the DoD Selected Acquisition Report (SAR), Drezuer et al. (1993) computed the average cost growth on 197 acquisition programs with start dates from 1960 through 1990. Results show that cost growth fluctuated around 20 percent, with no substantial improvement through time.

A factor contributing to defense cost growth is estimation error, possibly due to excessive competition. [2] Contractors have an incentive to understate initial costs to win new contracts. Likewise, because programs compete for limited resources, there is an incentive to accept low estimates as reasonable. Drezner et al. (1993) could not test this assertion, but did report that the average initial cost estimates were systematically understated. This finding is consistent with results reported by others (General Accounting Office [GAO], 1988; Tyson, Harmon, & Utech, 1989; McNichols, McFarland, KcKinney, & Balut, 1984; Christensen, 1994, 1996).

There are two weaknesses with Drezner et al. (1993). First, the SAR data base focuses on defense programs. The highly aggregated nature of program data may mask differences that may otherwise be apparent at the contract level. Second, the analysis focuses on cost growth, defined as the difference between the initial budget and the final cost of the program. Cost growth does not distinguish between uncontrollable factors, such as changes in scope or technology, and controllable factors, such as inadequate planning or poor control techniques. For example, Czelusniak and Rodgers (1997) report that Congressional decisions to shift funds to near-term priorities external to a program (e.g., unplanned contingency operations in Bosnia) account for up to one-half of the cost growth in major weapon systems. Program managers cannot control this kind of cost growth.

Our study attempts to overcome these weaknesses by analyzing cost overruns on completed defense acquisition contracts. A cost overrun, defined as the difference between a contract's final budget and final cost, is a more appropriate metric for measuring the impact of the DoD initiatives because the initiatives focus on actions that program managers can influence, such as thorough planning and disciplined control (McNaughter, 1990). Funding instability and changes in requirements are typically beyond the control of program managers and lead to cost growth, but not necessarily to cost overruns (Christensen and Gordon, 1998). The final budget of a project is a better estimate of what a well-managed contract should cost, because it includes all the authorized changes. that may not have been known at the start of a contract.

Another difference between our study and Drezner et al. (1993) is our focus on the Packard Commission's recommendations. Prior initiatives (e.g., McNamara, Carlucci) had been ineffective in reducing cost growth (Dews et al.; 1979, Gansler, 1989; Drezner et al., 1993), and current initiatives (e.g., the Federal Acquisition Streamlining Act (FASA), the Federal Acquisition Improvement Act) were building on the framework of the Packard Commission initiatives (Gates, 1989).

METHODOLOGY

This study tests whether the recommendations implemented as a result of the Packard Commission affected the cost performance experienced on defense acquisition contracts. The expectation is that cost performance would improve after implementation. To test this hypothesis, we compared the average cost overrun percent on populations of defense contracts before and after the commission's recommendations were implemented using the nonparametric Mann-Whitney test at the 10 percent level of significance. [3]

The hypothesis was tested using cost performance data on 269 contracts, completed between January 1, 1988, and December 31, 1995. The 8-year period provided for approximately four years before and after the treatment date of December 31, 1991. The decision to use four years of performance data before and after the treatment date was subjective, but seemed adequate to account for a gradual implementation of the commission's recommendations.

Based on assessments made by the GAO, we chose December 31, 1991, as the treatment date. In 1990, the GAO reported that efforts to implement the recommendations were in various stages, with some initiatives under way, and others planned for implementation in the "near term" (GAO, 1990). In 1991, the GAO reported that most of the recommended changes had been made and the remaining changes would be completed in the "short term" (GAO, 1991).

The average cost overrun percent (CO%) is defined as

CO% = (Final cost - Final budget) /Final budget) [yen] 100 (1)

Average CO% = ? (CO%) / n (2)

where the final cost and budget of the contract were the actual cost of work performed (ACWP) and the budget at completion (BAG) reported by the contractor on the last cost performance report, and n is the number of contracts.4 As a relative measure of cost performance, CO% adjusts for the effects of inflation and contract size.

The monthly cost performance report is prepared by the contractor and submitted to the government throughout the life of the contract. Data from the report are stored in the Defense Acquisition Executive Summary (DAES) data base, maintained by the Office of the Secretary of Defense. At the time of this study, the data base contained cost information from 378 defense acquisition programs encompassing 1,843 individual contracts from 1977 through 1995. The reliability of the data is controlled by earned value management control systems criteria, [5] a DoD requirement on most large defense contracts. Virtually all of the data in the DAES data base comes from criteria-compliant contractors. But because of technical problems with the data base and our focus on completed contracts, we could only include 269 contracts in our study. [6]

We also evaluated the sensitivity of our results to contract phase (development, production) and managing Service (Army, Navy, Air Force). Because of greater uncertainties, the cost performance on development contracts is often worse than the cost performance on production contracts (Christensen, 1994). Given the different missions and requirements of the Army, Navy, and Air Force, it is possible that the Packard Commission's recommendations were implemented differently across the Services. Table 2 summarizes the relevant cost data on the contracts used in our study. [7]

RESULTS

As illustrated in Figure 1, the recommendations of the Packard Commission affected cost performance, but in the wrong direction. Cost performance worsened rather than improved after implementation. As Table 3 shows, the average final overrun percentage for contracts before implementation was 5.6 percent. After implementation, the average final overrun percentage was 9.5 percent. With a p-value of 0.055, this change is statistically significant at an a of 0.1 0. [8]

This result was sensitive to contract phase. The cost performance on the sample of development contacts worsened from 4.1 to 15.3 percent (p = 0.014). The cost performance on production contracts also worsened from 6.2 to 7.2 percent, but the change was not significant (p = 0.294).

The result was also sensitive to the military service that managed the contract. The average cost overrun on Air Force contracts worsened significantly from 2.8 percent to 12.7 percent (p = 0.003). The average cost overruns on Navy and Army contacts did not change significantly. This was true regardless of contract phase.

DISCUSSION

Overall, these results show that the Packard Commission's recommendations did not improve the cost performance of defense acquisition contracts. As such, the results are consistent with results reported by Drezner et al. (1993). Our study also shows, however, that for development and Air Force contracts, the final cost overrun percentages more than tripled after the recommendations were implemented. In general, development contracts are more risky than production contracts, and appear to be more sensitive to policies affecting cost performance. We do not know why the cost performance of Air Force contracts worsened significantly, while the cost performance of the other Services did not change significantly. Although the Packard Commissions s recommendations were to be implemented DoD-wide, it is possible that each Service implemented them differently. Clearly, further research on this issue is needed.

Drezner et al. (1993) also reported strong evidence of understated initial budgets. Based on the same data base used in our study, Christensen (1994, 1996) reported that the predicted final cost on defense acquisition contracts is also systematically understated. Thus, not only are the initial budgets understated, but the revised estimates of final cost are also understated throughout the lives of most defense acquisition contracts. Although there are many other causal factors for poor cost performance, it is apparent that prior policies that encourage cost realism have been ineffective.

This study raises some concerns regarding the appropriateness of current reform initiatives. Presently, the DoD is operating in an era of acquisition reform. The passing of the FASA in 1994 and the Federal Acquisition Improvement Act in 1995 marked the first major revisions of acquisition policy since the Packard Commission's recommendations. A review of these policies indicates striking similarities between their major provisions and those of earlier reform efforts, including the Packard Commission (Gates, 1989). Themes such as streamlining, decentralization of authority, empowerment, and cultural change simply re-emerge in a newly packaged policy. Because the current provisions are so similar to prior reform efforts that were ineffective, the DoD should not realistically expect improvements in cost performance.

CONCLUSION

The recommendations of the Packard Commission have been ineffective in reducing cost overruns on major defense acquisition contracts. Cost performance on development and Air Force contracts actually worse after implementation of the commission's recommendations. Estimation error was identified as one contributing factor, but additional causal factors are possible. We recommend that the impact of more recent policy changes be investigated. The FASA of 1994 and the Federal Acquisition Improvement Act of 1995 are the first major rewrites of government procurement regulations in a decade. Of course, it will take several years before the cost impact of these initiatives is known. In the meantime, a comparison of new policies with prior reform efforts should yield insight into their likely effectiveness.

David S. Christensen is an associate professor of accounting at Southern Utah University. He taught at the Air Force Institute of Technology from 1987 to 1997. In 1994 he retired from the Air Force as a major. With research and consulting interests in defense cost management, he has published more than 50 papers in academic and professional journals, including previously in the Acquisition Review Quarterly.

Capt David A. Searle, U.S. Air Force, is presently serving as a contracting officer at the Space and Missile Center at Los Angeles Air Force Base, CA. This paper is a summary of his master's thesis, which he completed in September 1997, at the Air Force Institute of Technology.

Caisse Vickery, Ph.D., was an assistant professor of contract management at the Air Force Institute of Technology at the time of this research. Presently, he is the director of the Acquisition Center for Excellence at the National Reconnaissance Office. His research interests include acquisition reform, foreign military sales, and the use of information technology in contracting and purchasing.

ENDNOTES

(1.) The commission's formal name was the "President's Blue Ribbon Commission on Defense Management." David Packard was its chairman.

(2.) Of course, estimation error is not the only factor contributing to unplanned cost. Other factors often cited in the literature include schedule slips, changes in requirements, improvements, management inefficiencies, and organizational culture (Scott, 1983).

(3.) The parametric t-test failed the key assumptions of normality and equal variances. For a description of the Mann-Whitney test, see Conover (1980).

(4.) To simply the technical jargon, we prefer the term "cost overrun" to "adverse cost variance." An adverse cost variance occurs when ACWP exceeds the flexible budget, termed the "budgeted cost of work performed" (BCWP).

(5.) EMVS criteria were formerly termed "cost/schedule control systems criteria" (C/SCSC). In 1996, the criteria were slightly revised and renamed. Despite the multiple names, the criteria have not changed significantly since their inception in 1967.

(6.) A contract was included in our analysis if it was completed within the 8-year period, and the necessary data were available (contract phase, ACWP, BCWP, and BAG). The DAES data base that we used did not include classified programs. See Searle (1997) for a complete listing of the data used in this study.

(7.) This study has limitations that threaten its internal validity. In particular, political, macro-economic, and technological events may have occurred during the 8 year-period that may have influenced contract performance. We made no attempt to control for such events. In addition, our sample was not evenly distributed across the Services, with only 8 percent of the contracts managed by the Army. It is possible that the uneven distribution may have biased our results.

(8.) All hypothesis tests in this study were directional or one-sided.

REFERENCES

Christensen, D. S. (1996, Spring). Project advocacy and the estimate at completion problem. Journal of Cost Analysis, 35-60.

Christensen, D. S. (1994, Winter). Cost overrun optimism--Fact or fiction?" Acquisition Review Quarterly 1, 25-38.

Christensen, D. S. & Gordon, J. (1998, September). Does a rubber baseline guarantee cost overruns on defense acquisition contracts? Project Manager Journal.

Conover, W. J. (1980). Practical nonparametric statistics (2nd ed). New York: John Wiley & Sons.

Czelusnniak, D. P. & Rodgers, P. D. (1997, September-October). The Kaminski initiative. Program Manager 24, 59-65.

Dews, E., Giles, K. S., Barbour, A., Harris, E.& Hesse, M. (1979, October). Acquisition policy effectiveness: Department of Defense experience in the 1970s (R-25 16-DRE). Santa Monica, CA: RAND Corporation.

Drezner, J. A., Jarvaise, J., Hess, R., Hough, P., & Norton, D. (1993). An analysis of weapon system cost growth (MR-29 1 -AF). Santa Monica, CA: RAND Corporation.

Gansler, J. S. 1989. Affording defense. Boston: MIT Press.

Gates, W. (1989, January) Department of Defense procurement policy reform: An evolutionary perspective (NPS-54-89-01). Master's thesis. Monterey, CA: Naval Postgraduate School.

GAO. (1988, November). Weapons cost: Analysis of major weapon systems cost and quantity changes (GAO/NSIAD-89-32FS). Washington, DC: Government Printing Office.

GAO. (1990, December). Acquisition reform: Defense management report savings initiatives (NSIAD-91-11). Washington, DC: Government Printing Office.

GAO. (1991, August). Acquisition reform: Implementing defense management review initiatives (NSIAD-91-269). Washington, DC: Government Printing Office.

McNaughter, T. L. (1990). Defense management reform: For better or worse? Washington, DC: The Brookings Institute.

McNichols, G. R., McFarland, J. P., McKinney, B. J. & Balut, S. J. (1984, April 30). The problem of cost growth (analysis of weapon system costs as of December 1983). McLean, VA: Management Consulting and Research, Inc.

Scott, E. L. (1983, Winter). The cost growth phenomenon. National Contract Management Journal, 37-45.

Searle, D. A. 1991. The impact of the Packard Commission's recommendations on reducing cost overruns in major defense acquisition programs. Master's thesis. Wright-Patterson Air Force Base, OH: The Air Force Institute of Technology. Tyson, K. W., Harmon, B. R. & Utech, D. M. (1994, July). Understanding cost and schedule growth in acquisition programs. Alexandria, VA: Institute for Defense Analysis.

Weida, W. J. & Gertcher, F. L. (1987). The political economy of national defense. Boulder, CO: Westview Press.

Table 1
Acquisition Regulations and Initiatives 1
Year  Regulation or Initiative
      published
1969  Packard Initiatives
1971  Blue Ribbon Defense Panel
      (Fitzhugh Commission)
1972  DoDD 5000.1 (Major System
      Acquisitions); Commission on
      Government Procurement
1973  DoDD 5000.4 (CAIG); DoDD
      5000.3 (T&E)
1975  DoDI 5000.2 (Major System
      Acquisitions); DoDD 5000.28 (DTC)
1976  0MB Circular A-109
1978  Defense Science Board Acquisition
      Cycle Task Force
1979  Defense Resource Management Study
1981  Carlucci Initiatives; Defense
      Acquisition Improvement Program
1982  Nunn-McCurdy (thresholds)
1983  Grace Commission
1985  DoD 5000.43 (streamlining)
1986  Packard Commission
1987  DoDD 5134.1 (USD(A);
      DoDD 5000.49 (DAB)
1989  Defense Management Review
1991  Revised DoDI 5000.2 (Major System
      Acquisitions)
1994  Federal Acquisition Streamlining
      Act (FASA)
1995  Federal Acquisition Improvement
      Act (FASA II)
(1)Modified from Drezner et al., 1993.
Table 2
Summary of Contract Data
                                   Phase
                                    All   Development  Production
Before Implementation (1998-1991)
Number of contracts (n)             148       47              101
Average final cost                  356       312             377
(millions of dollars)
Standard deviation                 1105       598            1276
Average final budget                341       294             363
(millions of dollars)
Standard deviation                 1061       542            1232
Average final overrun               15        17               14
(millions of dollars)
Standard deviation                  55        59               53
After Implementation (1992-1995)
Number of contracts (n)             121       34               87
Average final cost                  571       768             494
(millions of dollars)
Standard deviation                 1121      2011             426
Average final budget                511       647             458
(millions of dollars)
Standard deviation                  935      1661             390
Average final overrun               60        121              36
(millions of dollars)
Standard deviation                  274       499              84
                                   Service
                                        AF  Navy  Army
Before Implementation (1998-1991)
Number of contracts (n)                 64    70    14
Average final cost                     428   328   174
(millions of dollars)
Standard deviation                    1581   544   233
Average final budget                   412   310   174
(millions of dollars)
Standard deviation                    1524   505   250
Average final overrun                   16    17     0
(millions of dollars)
Standard deviation                      60     L
After Implementation (1992-1995)
Number of contracts (n)                 49    64     8
Average final cost                     724   502   190
(millions of dollars)
Standard deviation                    1674   465   117
Average final budget                   613   476   173
(millions of dollars)
Standard deviation                    1379   438   114
Average final overrun                  111    26    17
(millions of dollars)
Standard deviation                     421    69    27
Table 3.
The effect of Packard Commission Recommendations on Defense Cost
Performance
                                         Contract Phase
                                 All      Development    Production
                              Contracts    Contracts     Contracts
Numberofcontracts(n)           269             8          188
Final overrun before             5.6           4.1          4.1
implementation (%)
Final overrun after              9.5          15.3          7.2
implementation (%)
Difference (%)                   3.9          11.2          1.0
Statistical significance (p)     0.055         0.014        0.294
                              Managing Service
                                    Air         Navvy     Army
                                   Force
Numberofcontracts(n)               113           134      22
Final overrun before                 2.8           7.6     8.1
implementation (%)
Final overrun after                 12.7           6.1    17.0
implementation (%)
Difference (%)                       9.9          -1.5     8.9
Statistical significance (p)         0.003         0.206   0.110

[Graph omitted]

COPYRIGHT 1999 Defense Acquisition University Press
COPYRIGHT 2004 Gale Group