The Costs And Benefits Of The Earned Value Management Process - cost control in defense research and programmes

David S. Christensen

Up to now assessments of the earned value management process have focused on either the positives or the negatives--but a complete and objective evaluation has not been made. This review surveys the literature and paints a balanced picture of the issue.

Do the benefits of the earned value management process exceed its costs? Several published studies report the costs but ignore the benefits. Others focus on the benefits, but ignore the costs.

A widely circulated report by Coopers & Lybrand and TASC (1994), for example, concludes that the Department of Defense regulatory cost is significant, and the requirement for earned value management systems (EVMS) criteria is among the largest cost drivers. But in this study the benefits of the earned value information derived from criteria-compliant contractors were not considered. Clearly, a report that addresses only the costs or the benefits of earned value can be misleading. Both the costs and the benefits of earned value must be assessed.

Here I provide a comprehensive literature review, and summarize and synthesize studies reporting the costs or the benefits of earned value. The result is a more complete and objective evaluation of the earned value management process.

EARNED VALUE AND EVMS

There is a difference between earned value and EVMS criteria. Earned value is a special metric that can be used to manage any project. The criteria are standards for management control systems that use earned value. Since 1967 the criteria have been required on large, flexibly priced defense contracts. The purpose of the criteria was to assure the reliability of the earned value metric. Although earned value does not require the criteria, it does require a management control system that meets at least some of the standards described by the criteria. In this paper, the term "earned value management process" includes both earned value and the EVMS criteria.

EARNED VALUE

The earned value concept originated in industry and was developed primarily by the Department of Defense (DoD) as a management tool for use on defense acquisition contracts. Earned value is a metric devised to achieve meaningful comparisons between planned and completed work. It is similar to what accountants call a "flexible budget," where the original budget for work is adjusted for the actual level of output. Cost variances result when the actual cost of the work and its flexible budget (earned value) differ. Significant variances are analyzed to identify and correct problems before they worsen.

A major difference between a flexible budget and earned value is the time dimension associated with earned value. Initially, the work on a project is divided into pieces, assigned a budget, and assigned a schedule. Because each increment of work is time-phased, a schedule variance occurs if work is not completed (earned) when it was scheduled to be completed. Because the work has a budget, the schedule variance is often reported as a dollar amount. The flexible budget used in cost accounting does not provide any information about schedule variances. Like the cost variances, significant schedule variances are analyzed and corrected when possible.

When variance analysis is conducted properly (e.g., on time, and at the proper level), it can be an effective control against further cost and schedule problems that may jeopardize the successful completion of a project. Unfortunately, variance analysis can be untimely or excessive and even contribute to project failure by drawing project managers, engineers, and others away from more urgent problems.

EVMS CRITERIA

A key to the effective use of earned value is an adequate management control system that fosters the proper planning and integration of work on a project. EVMS criteria define the attributes that management control systems must possess for earned value to be used effectively. Originally, the criteria were established by the Air Force as cost/schedule planning and control specifications (C/SPEC) for application on major defense acquisition contracts. Later the criteria were adopted by all the military services as cost/schedule control systems criteria (C/SCSC). Recently, the criteria have been slightly revised and renamed earned value management control systems criteria (DoD, 1996).

Despite the multiple names, the criteria have not changed significantly since their inception. Presently, there are 32 EVMS criteria, organized into five categories, that pertain to major project management activities:

* organization;

* planning and budgeting;

* accounting;

* analysis; and

* revisions.

Each criterion addresses a major principle necessary for effective management of large, flexibly priced defense projects. For example, one criterion requires that each element of work on the project has a budget. Another criterion requires that each element of work has a schedule. Without a budget and a schedule, it would be difficult to properly manage a project of any size, much less a major defense project that can cost over a billion dollars and last for many years. Thus, criteria are often described as common-sense management practices that any well-managed defense contractor would use.

Over the years, however, implementing the criteria became an administrative burden that was eventually viewed as a non-value-added activity by contractors and program managers (Government Accounting Office, 1997). Like many government documents, the DoD's Joint Implementation Guide (JIG), which described how to implement the criteria, grew in size and complexity (DoD, 1987). Its checklist of 158 questions (Appendix E of the JIG) was often perceived as a contractual requirement and administered with audit-like rigor by the review teams. Additionally, earned value data was mistakenly judged "guilty by association" and occasionally ignored by project managers who may have benefited from it. According to Abba (1995), for example, large cost overruns on some major defense projects were foreseeable from contractor earned value reports but not recognized by program managers.

There are several factors that contributed to the implementation problem (e.g., a lack of industry ownership, inadequate training, and an awkward technical jargon). A major factor was a failure in the early years to make the earned value process the responsibility of program managers and contractors. Based on a two-year review of the DoD's earned value management process, the GAO (1997) concluded that while the process was intended to serve the needs of several user groups, financial personnel managed the process. It was natural for this group to focus on their oversight responsibilities and stress criteria compliance. But it was also natural for other user groups, including the program managers, to perceive earned value as a purely financial reporting requirement. According to the DoD, "the needs of the program manager were often not met when EVMS were viewed primarily as a financial reporting system" (GAO, 1997, p. 29).

Through the years, the criteria implementation problem has prompted studies that addressed either the benefits or the costs of the earned value management process. In general, studies that focused on benefits concluded that earned value and the criteria concept were sound, while those focused on cost reported the cost of compliance to be relatively small, ranging from less than one to five percent of contract cost. I found no study that directly compared benefits with costs, possibly because the benefits are largely nonquantifiable. Clearly, any study that focuses on only one side of a cost-benefit issue may be misleading.

Regardless of the focus, nearly all of the studies are unpublished, thus contributing to the difficulty of comparing costs with benefits. My purpose is to remedy this problem by providing a summary of all I have found. I have compiled and reviewed most of the cost and benefit studies related to earned value. The results of this survey follow, as does a conceptual framework for comparing the costs and benefits of the EVM process.

THE COSTS OF THE EVM PROCESS

Most of the cost studies reviewed appropriately focus on the incremental cost of EVMS compliance and reporting. The "normal" costs of operating a management control system are not considered relevant because they would be incurred in the absence of any requirement for an earned value management system (DoD, 1987, p. viii).

COOPERS & LYBRAND/TASC

The most recent study of this kind, "The DoD Regulatory Cost Premium: A Quantitative Assessment," was conducted jointly by Coopers & Lybrand and TASC (C&L/TASC) under the auspices of the Under Secretary of Defense for Acquisition and Technology (1994). Its purpose was to estimate the industry cost of DoD regulation and oversight, including the regulatory requirement for EVMS. It did not include the DoD's direct oversight costs (e.g., government auditors). Based on an analysis of 10 contractor facilities, activity based costing was used to report an average regulatory cost premium of 18 percent of value-added costs.[1] The cost of EVMS ranked third among the top 10 cost drivers, and was estimated to be about 0.9 percent of the value-added costs. For example, in a graph depicting value-added costs, C&L/TASC indicate that material purchases are about 40 percent of the cost of a contract (p. 4a) [2] Thus, on a $100 million contract, $60 million (60 percent) would be value-added, $10.8 million (18 percent of $60 m illion) would be the regulatory premium and $0.54 million (0.9 percent of $60 million) would be the regulatory cost of EVMS to industry.

Most of the EVMS costs were in the areas of engineering and program management (65 percent), and finance (25 percent). Program managers are responsible for the entire management control system. Engineers are typically responsible for variance analysis and reporting. Finance personnel are typically required to ensure compliance with the EVMS criteria. A DoD working group that investigated these costs reported that most of them (two-thirds to three-fourths) were unnecessary and not required by criteria (DoD, 1997, pp. 8-9). An example is preparing written variance reports at detailed levels in the work breakdown structure, termed "control accounts." Although the criteria do not require a written variance analysis for each control account, government review teams came to expect them as tangible evidence of criteria compliance. Contractors with written variance reports were more likely to be found compliant than contractors without written variance reports. Eventually, a written variance report for every control account became an unwritten rule. Other examples of non-value-added activities that came be to expected for criteria compliance are provide by Abba (1997, p.3).

Clearly, the 0.9 percent EVMS cost premium is only a rough estimate. The cost premium would be greater if the direct government costs were included. The premium would be smaller if the activities not required by the criteria were eliminated. In addition, C&L/TASC warn against generalizing the results of their study (p. 3). The sample was nonrandom and only 2 of the 10 sites had contracts subject to EVMS criteria. Further, the results cannot be independently verified because the data are proprietary. In general, C&L/TASC concluded that the contractors viewed EVMS positively, but that it was costly to implement (p. 22):

All contractors subject to C/SCSC (EVMS criteria) agree that, as currently required by DoD, cost-schedule reporting is too detailed, repetitive, and voluminous to be used effectively as a management tool by either the government or industry, and that the requirement may in fact undermine program performance by diverting the time and attention of the company program manager.

OTHER COST STUDIES

Lampkin (1992) reviewed five studies that estimate the marginal cost of implementing and maintaining a criteria-complaint EVMS (Table 1). [3] The cost range is expressed as a percentage of contract cost. The first three estimates are based on opinion surveys of industry pr DoD experts. The MITRE estimate pertains to Air Force software development contract only. Decision Planning Corporation (1992) uses a cost-estimating model that assumes a generic, 3-year research and development contract of $75M. Humphreys and Associates distinguish between nonrecurring and recurring costs. Nonrecurring costs pertain to implementing a criteria-compliant system for the first time, and range from 2.5 to 4.0 percent. Recurring costs pertain to maintaining criteria-compliance, and range from 0.5 to 1.5 percent. The final row in the table is an average of these estimates, also reported by Lampkin (p. 37). [4]

UNNECESSARY COST DRIVERS

Survey and interview research indicate that at least some of the cost of EVMS is unnecessary and due to an over-implementation of the criteria (Table 2). Examples include lengthy "system descriptions" of EVMS, written variance analysis at the control account level, and over-specified work breakdown structures. The National Security Industrial Association (NSIA) (1980, p. 17) estimated the number of pages required to achieve and maintain criteria-compliance for the industry to be 32.8 million pages annually. Based on a survey and interviews, Arthur D. Little (1983, 1984) concluded that EVMS was a good approach to controlling contract performance, but that there was room for improvement (p. I-3).

Prompted by this conclusion, the DoD and NSIA formed a total quality management team to review EVMS in 1989. One of its purposes was to determine where there may be excessive cost and to what extent the cost could be reduced. The team concluded that there are excesses in EVMS implementation and reporting that result in unnecessary cost (p. 3.9.7).

EVMS REFORMS AND COST REDUCTION

The sensational cancellation of the Navy's A-12 program (Beach, 1990) and a subsequent DoD audit report (1993) were additional catalysts for reform. To promote a program management orientation, the DoD policy-making body for EVMS was shifted to an executive steering group with representatives from the services, acquisition executives, and the Defense Contract Management Command (DCMC) in 1995. [5] To refocus program management attention to the information in earned value reports, compliance responsibility was transferred from the military services to DCMC in 1996. Finally, the DoD criteria used since 1967 were replaced with industry standards in 1996, and the possibility for industry self-certification was offered (Christie, 1994).

Removing unnecessary requirements related to EVMS will likely reduce but not eliminate the marginal cost. The NSIA estimated industry-wide savings from reforming EVMS could reach over one billion dollars annually (Christle, 1996). A DoD working group estimates the EVMS regulatory premium can be reduced by one third or 0.3 percent of value-added costs (DoD, 1997). Thus, two-thirds of the EVMS regulatory premium or 0.6 percent will remain.

THE BENEFITS OF THE EVM PROCESS

None of the marginal cost studies describes the marginal benefits of EVMS, perhaps because the benefits are difficult to quantify. Accordingly, most benefit studies I review here are qualitative assessments. However, to the extent that the criteria help a company use or continue to use the management principles required by the criteria, I believe the marginal benefits of the criteria are greater than zero.

THE LEGACY OF EVMS

The full application of the criteria is appropriate for large, cost-reimbursable contracts where the government bears the cost risk. For such contracts, the management discipline described by the criteria is essential. The box on following page is an abbreviated list of EVMS benefits, described by Fleming and Koppelman as the legacy of using the criteria on government contracts for three decades (1996, p.22). Note that they do not separate the benefits of earned value data from the benefits of the criteria, perhaps because the reliability of data depends on the disciplined application of the management practices described by the criteria.

Benefit 1. Although the criteria do not require an external report, managing with one system while reporting from another is neither efficient nor effective. The criteria concept encourages the company to use its own internal management control systems, provided those systems meet the management standards described by the criteria.

Benefit 2. The criteria require that all the authorized work and related resources are defined and integrated using a product-oriented work breakdown structure.

For a company that has managed by functional areas only (e.g., engineering, manufacturing, accounting), the product orientation can help organize and coordinate the contributions of each area, and ensure that work, schedule, and cost are properly integrated.

Benefit 3. The consistent reporting of criteria-compliant projects for over 30 years has resulted in a database useful for comparative analysis analysts have used this database to create important insights for managers. For example, a comparative analysis of the cost performance of similar aircraft was compelling evidence that the Navy's A-12 project was out of control (Beach, 1990).

Benefit 4. The cumulative cost performance index (CPI), defined as the earned value to-date divided by the cost to-date, has been shown to stabilize to within 10 percent by the 20 percent completion point for most defense acquisition contracts. In most cases the cumulative CPI only worsens (Christensen and Heise, 1993). Among other things, this indicates that cost management must occur early to be effective.

Benefit 5. The schedule performance index (SPI), defined as earned value divided by planned value, is useful for identifying schedule problems, especially when used with critical path information (Fleming and Koppelman, 1996, p. 5). Because schedule problems are often resolved by additional spending, an adverse SPI is also predictive of later cost problems. The criteria recommend that all the work is scheduled and traceable from the master program level to the detailed levels. Consistent with the criteria concept, no specific scheduling system is required.

Benefit 6. The cumulative CPI is also useful for determining a reasonable lower limit for the estimated final cost of a contract, termed the estimate at completion (EAC) (Christensen, 1996). A lower bound is useful for planning and control purposes. The criteria recommend that the estimate be evaluated regularly.

Benefit 7. It has been shown that the SPI and CPI can be combined to estimate a reliable upper bound to the EAC (Christensen, 1996). When combined with the CPI-based lower bound, a "most likely" range of EACs is determined. When the contractor's EAC is outside this range, there may be a problem with the contractor's estimation system.

Benefit 8. Another earned value index, the to-complete performance index (TCPI), is useful for evaluating the reasonableness of the contractor's EAC or other financial goals (Christensen, 1994). The TCPI is the ratio of the remaining work to the remaining financial resources. It indicates the level of performance that the contractor must achieve to reach a financial goal. Thus, this earned value metric can help the manager assess the reasonableness of critical financial goals, such as completing the remaining work within the targeted cost.

Benefit 9. While cumulative performance indices are useful for predicting trends at summary levels in the work breakdown structure (WBS), weekly or monthly CPIs are useful for cost performance trends at the detailed levels of the WBS (Fleming and Koppelman, p. 28). The criteria recommend an analysis of these and all other metrics at the frequency and level needed by management for effective control (DoD, 1996).

Benefit 10. By directing management attention to only the most critical problems, information overload can be reduced. Although not always implemented properly, the criteria encourage variance thresholds and tailoring to reduce the potential for overload.

Other benefit studies. Survey research shows that most managers agree that EVMS has benefits (NSIA, 1980; Little, 1983 and 1984; DoD/NSIA, 1991). Based on a survey of 534 managers, for example, more than 70 percent agree that "a major benefit of the criteria is more thorough planning than would otherwise be accomplished," and that EVMS is "effective in helping managers control contract performance" (Little, 1983, 0. III-3). Government audits of the EVMS process have not challenged this perception (DoD, 1993; GAO, 1997). For example, the GAO (1997) reports that the earned value concept is "recognized as a sound way to measure progress on major acquisition programs" (p. 3).

SYNTHESIS

Figure 1 identifies qualitative characteristics that a report should possess to be useful for decision making. [6] I believe it is a useful conceptual framework for comparing the costs and benefits of EVMS. The output of the EVMS process is the earned value report. To be useful for decision making, the report should have relevance and reliability.

To be relevant, the report should have predictive value, feedback value, and be timely. Many of the marginal benefits of EVMS pertain to these characteristics. The benefit studies indicate that managers and oversight personnel find the EAC and the variance information useful, if not timely. Surveys indicate some dissatisfaction with the report being too late (DoD/NSIA, 1991); however, recent improvements in computer technology and software have made real-time access to contractor databases achievable (GAO, 1997).

To be reliable, the report should be verifiable, valid, and objective. Oversight personnel are primarily concerned with reliability. The marginal costs of EVMS arise primarily from the need for reliability. The government needs access to contractor data verify its accuracy. The government also needs assurance that the contractor is reporting and managing from the same database. Finally, because the criteria are applied primarily to cost-reimbursable contracts, the cost data must be objective.

Achieving these qualitative characteristics is not easy. The double-headed arrow between relevance and reliability suggests a tradeoff. A report may not be timely, for example, if every number must be verified. The GAO (1997) observed that a delicate balance exists between managers (needing relevance) and the oversight community (needing reliability), and concluded that because the oversight community managed the EVMS process, the need for reliability was stressed over the need for relevance.

However, EVMS reforms have restored the balance. The marginal cost of EVMS has decreased, while the marginal benefits have increased. As the unnecessary activities related to implementing the criteria (e.g., written variance reporting) are eliminated, the DoD estimated a one-third reduction in the EVMS cost premium. With the shift in ownership and management responsibilities to program managers, the value of the earned value reports should be easier for program managers to recognize (Abba, 1997).

Ultimately, the decision of whether the marginal benefits of EVMS exceed the marginal cost is subjective. Perhaps the most compelling evidence that benefits exceed the costs is the astonishing increase to earned value outside the DoD by other agencies, commercial companies, and other countries (Abba, 1997; GAO, 1997). After decades of assertions that EVMS is too expensive and only appropriate for DoD projects, earned value is recognized more than ever as a necessary and effective management tool for projects of any size and risk. The key is the proper application of EVMS (Fleming and Koppelman, 1996).

David S. Christensen is an associate professor of accounting at the University of West Florida. After receiving a Ph.D. degree from the University of Nebraska in 1987, he joined the faculty at the Air Force Institute of Technology (AFIT), where he stayed for 10 years. While there, he taught courses on earned value, including earned value management control systems criteria. He also led a long-term research project on cost management topics related to earned value. Christensen holds several certifications and is active in several professional associations, including the Society of Cost Estimating and Analysis. He has published extensively in the area of earned value.

ENDNOTES

(1.) Value-added costs are the costs to convert raw material to a finished product. Total costs were not used as the base to determine the regulatory premium because a prime contractor's material purchases are "to a great extent the value-added costs of its subcontractors and suppliers." Using total costs in the denominator would double-count material costs and understate the regulatory cost impact (Coopers & Lybrand and TASC, 1994, p.4).

(2.) C&L/TASC did not indicate that 40 percent was based on the actual cost data from the sample. It may simply be an example used to explain the meaning of value-added costs.

(3.) Unfortunately, we could not locate most of these studies to evaluate the methodology.

(4.) C&LITASC used value-added costs to determine the 0.9 percent cost premium of EVMS. If the percentages in Table 1 were also based on value-added cost, they would be larger. For example, if the average value-added cost is 60 percent of contract cost, the range reported by Lampkin would be 0.67 percent to 2.71 percent.

(5.) As the government's on-site representative at the contractor's facility, DCMC provided DoD with assessments of the contractor's performance.

(6.) We have adapted this from a conceptual framework used by the Financial Accounting Standards Board (1980).

Abba, W. (1995, October). Earned value management rediscovered! Washington, DC: Office of the Under Secretary of Defense for Acquisition.

Abba, W. (1997, January-February). Earned value management--Reconciling government and commercial practices. Program Manager 26, 58-69.

Beach, C. P. Jr. (1990, November). A-12 administrative inquiry (Report to the Secretary of the Navy). Washington, DC: Department of the Navy.

Christensen, D. S. (1994). Using performance indices to evaluate the estimate at completion. Journal of Cost Analysis, 17-24.

Christensen, D. S. (1996, Spring). Project advocacy and the estimate at completion problem. Journal of Cost Analysis, 61-72.

Christensen, D. S., & Heise, S. (1993, Spring). Cost performance index stability. National Contract Management Journal 25, 7-15.

Christle, G. (1994). Implementation of earned value--A model program approach. Washington, DC: Office of the Under Secretary of Defense for Acquisition.

Christle, G. (1996). Earned value management--The cost/schedule control systems criteria and the DoD regulatory premium: A call to action. Washington, DC: Office of the Under Secretary of Defense for Acquisition.

Coopers & Lybrand and TASC. (1994). The DoD regulatory cost premium: A quantitative assessment. Washington, DC

Decision Planning Corporation. (1992). The cost of the criteria. Costa Mesa, CA: Author.

Department of Defense. (1987). C/SCSC white paper: DoD joint implementation guide. Washington, DC: Author.

Department of Defense. (1993). Use of contractor cost and schedule system data (Audit Report 93-067). Arlington, VA: Office of the Inspector General.

Department of Defense. (1996). DoD 5000.2-R: Management procedures for major defense acquisition programs. Washington, DC: Author.

Department of Defense. (1997). Cost driver No. 3-Cost/schedule control system criteria (C/SCSC). DoD updated compendium of office of primary responsibility reports. Washington, DC: DoD Regulatory Cost Premium Working Group.

Department of Defense and National Security Industrial Association. (1991). Joint DoD/industry total quality management team report for program management on the cost/schedule management process. Washington, DC: Author.

Financial Accounting Standards Board. (1980). Qualitative characteristics of accounting information. Statement of financial accounting concepts (No. 2). Norwalk, CT: Author.

Fleming, Q. M. & Koppelman, J. M. (1996). Earned value project management. Upper Darby, PA: Project Management Institute.

General Accounting Office. (1996). Efforts to reduce the cost to manage and oversee DoD contracts (NSIAD 96106). Washington, DC: Author.

General Accounting Office. (1997). Significant changes in DoD earned value management process (NSIAD 97-108). Washington, DC.

Lampkin, E. D. (1992). The marginal cost and practicality of cost/schedule control systems criteria implementation. Aeronautical systems center Dayton, OH: Wright-Patterson Air Force Base.

Little, A. D., Inc. (1983 and 1984). Survey relating to the implementation of cost/schedule control systems criteria within the Department of Defense. A report to the Assistant Secretary of Defense (Comptroller). Phases land II. Author.

National Security Industrial Association (NSIA). (1980). Cost schedule systems compendium. Washington, DC: Author.

Table 1. The Marginal Cost of EVMS Criteria
(Percent of Contract Cost)
Author (Year)                    Source of Estimate
Kouts (1978)                     Survey of industry
MITRE Corp. (1982)               Survey of industry
DoD IG (1984)                    Survey of DoD experts
Decision Planning Corp. (1992)   Industry cost estimation model
Humphreys and Associates (1992)  Consultant experience
Lampkin (1992)                   Average of five studies above
Author (Year)                    Cost Range (%)
Kouts (1978)                        0.5 to 5
MITRE Corp. (1982)                  0.1 to 0.2
DoD IG (1984)                         5.0
Decision Planning Corp. (1992)     0.6 to 1.0
Humphreys and Associates (1992)    0.5 to 4.0
Lampkin (1992)                     0.4 to 1.63
Lampkin (1992)
Table 2. Other Studies Related to the Cost of EVMS
Author (Year)       Research Method (Sample Size)
NSIA (1980)         Opinion survey (74 contractors
A.D. Little (1984)  Interview (56 managers)
DoD/NSIA (1991)     Interview (250 mangers)
Author (Year)               Cost Drivers Related
                           to Over-Implementation
NSIA (1980)         Excessive documentation
A.D. Little (1984)  Excessive levels of detail in the WBS
DoD/NSIA (1991)     Written variance analysis reports

Ten Benefits of EVMS

1. It is a single management control system that provides reliable data.

2. It integrates work, schedule, and cost using a work breakdown structure.

3. The associated database of completed projects is useful for comparative analysis.

4. The cumulative cost performance index (CPI) provides an early warning signal.

5. The schedule performance index provides an early warning signal.

6. The CPI is a predictor for the final cost of the project.

7. It uses an index-based method to forecast the final cost of the project.

8. The "to-complete" performance index allows evaluation of the forecasted final cost.

9. The periodic (e.g., weekly or monthly) CPI is a benchmark.

10. The management by exception principle can reduce information overload.

COPYRIGHT 1998 Defense Acquisition University Press
COPYRIGHT 2004 Gale Group