Logistics analyses: flyway costs versus individual components of aircraft: an analysis

Dan Ritschel

The 1990s saw the budgets for acquisition programs steadily shrink. Concurrently, an increased operations tempo and aging weapon systems and weapon system components put a great deal of pressure on Air Force operations and maintenance accounts. While acquisition budgets have improved, there are still significant issues to be resolved. One area at issue is the acquisition and cost analysis environment. That environment today is characterized by budget instability, time constraints, and changing objectives. With that as a backdrop, Captain Ritschel and his coauthors investigate and measure the risks associated with taking a macro versus a micro approach to aircraft cost estimation. By examining the fidelity of a cost estimate developed at the cost level versus the individual component level, they argue that guidelines can be developed for the allocation of cost analysis resources.

Background

A shrinking workforce, unstable budgets, and rapidly changing objectives, under stricter time constraints, characterize today's cost analysis and acquisition environment. The result is that today's cost community is being asked to do more with less. (1) This is driving the need for cost analysts to increase productivity or identify and concentrate on those areas that encompass the majority of estimation error risk in order to meet the demand.

Reductions in manpower have impacted operation-level organizations such as the Aeronautical Systems Center (ASC) cost analyst resources at Wright-Patterson AFB, Ohio. Since 1992, ASC's total authorized cost analyst slots have declined by 54 percent, from 136 authorizations to only 63 in 2001. This includes a 69-percent loss of military slots and a 44-percent drop in civilian authorizations. (2)

The current aircraft acquisitions environment presents several challenges to the cost analysis community. First, cost analysts must operate within the reality of a smaller workforce, while accomplishing their mission of providing the best possible cost analysis and estimating for their program. Second, cycle-time reduction goals require cost analysts to complete estimates in a compressed timeframe. Finally, in this unpredictable environment, cost analysts do not have the luxury of knowing estimation requirements in advance. Thus, the ability to accomplish data collection in support of developing low-level, grassroots estimates will be reduced greatly.

Despite these changes in time, manpower, and predictability, it is extremely important that weapon systems perform at optimal operating capabilities. Achieving this objective necessitates the highest quality of work from cost and acquisition personnel. "With budgets shrinking and requirements steadily growing, the Department of Defense (DoD) has focused logically on initiatives to increase efficiency." (3) Determining methods to meet these challenges is imperative for cost analysts in today's environment. To keep the quality of work high--with less personnel, increasing programs, and dynamic technology--analysts will be required to increase not only productivity but also efficiency. To achieve increased productivity and efficiency under these conditions, cost analysts must recognize the greatest estimation error risk in a new weapon system. Efforts must be concentrated in these high-risk areas when developing an aircraft cost estimate.

The purpose of this article is to investigate and measure the risks associated with taking a macro versus a micro approach to aircraft cost estimation. By analyzing the fidelity of a cost estimate developed at the flyaway cost level versus at the individual component level, this research provides guidelines for appropriate allocation of cost analyst resources in today's constrained environment. Flyaway costs for aircraft are defined as follows:

   It relates to production cost and includes the
   prime mission equipment (basic structure,
   propulsion, electronics), systems
   engineering, program management, and
   allowances for engineering changes and
   warranties. Flyaway costs include (all)
   recurring ... production costs (contractor
   and government-furnished equipment) that
   are incurred in the manufacture of a usable
   end-item. (4)

In particular, two categories of aircraft will be considered: fighters and intertheater airlift. Intertheater airlift is those aircraft used for supply and transportation. The following questions regarding each of these categories will be addressed:

* Which aircraft components have the most cost-estimation error risk, and what is that risk?

* What is the cost-estimation error risk associated with estimating at the flyaway cost level?

* Is there a statistically significant difference in estimating at the component level versus flyaway level?

* Given a constrained resource environment, where should cost analysts focus their attention when developing an aircraft cost estimate?

Previous Literature

This literature review focuses on the fundamental components and techniques used to develop an aircraft cost estimate. First, a discussion of the basic building block for any cost estimate, the work breakdown structure (WBS), is examined. Next, the role of cost-estimating relationships (CER) in aircraft estimation is explored to understand why and how they are used. Then an explanation of aircraft cost-estimation techniques, specifically the parametric and grassroots methods, are covered. Finally, an overview of research accomplished on comparisons of macro and micro aircraft estimation techniques is investigated.

Work Breakdown Structure

The work breakdown structure is a basic building block of all major defense acquisition programs. As such, DoD Regulation 5000.2-R mandates, "A program work breakdown structure shall be established that provides a framework for program and technical planning, cost estimating, resource allocation, performance measurement, and status reporting." (5) In addition to developing a work breakdown structure, every program office is required to tailor its work breakdown structure using the guidelines set forth in Military Handbook Standard 881 (MIL-HDBK-881).

This research focused its comparisons between level one and level two of the work breakdown structure to facilitate the macro versus micro properties. Level two is selected as the micro level because of data availability and the fact, "Level two of any work breakdown structure is the most critical, because at level two, the project manager will indicate the approach planned to manage the project." (6)

Level one of the work breakdown structure is the entire defense materiel item, represented in this research by a complete aircraft system. Level two of the work breakdown structure is the major elements that comprise the aircraft system. Level two includes equipment-specific elements and common elements found in all major weapon systems. These common elements include systems engineering and program management, training, data, system test and evaluation, and so on. The guidelines for the WBS structure of an aircraft system come from MIL-HDBK-881. (7)

WBS Terminology Clarification

While the suggested WBS structure is being followed for data collection and analysis purposes, there are some terminology differences between MIL-HDBK-881 and the subsequent language used to describe the data collected. Specifically, at WBS level one, the term flyaway cost is substituted for aircraft system. This change was made because program office costs and costs not directly related to the contractor are not being considered. At WBS level two, the term basic airframe was substituted for air vehicle. Also, the common elements of system engineering and program management, system test and evaluation, data, and training are reclassified into a single category called other air vehicle. The form of the available data for collection drives these changes.

Cost-Estimating Relationships

The CER is one of the fundamental techniques used to estimate aircraft cost. A CER is defined formally as a "technique used to estimate a particular cost or price by using an established relationship with an independent variable." (7) The dependent variable is the item of interest that the CER will estimate (for example, airframe cost). The independent variables are composed of a multitude of explanatory variables. The CER is a mathematical relationship that predicts the dependent variable as a function of the independent variables. This relationship typically is using a historical data set of variables and applying a statistical technique, usually regression, to find the parameter estimates of the independent variables. (8)

The selection of independent variables is extremely important. To ensure an accurate and meaningful CER is developed, the independent variables must be identified as cost drivers for the dependent variable. "Cost drivers are those characteristics of a product or item that have a major effect on the product or item cost." (9) Typically, performance parameters are the most useful and accurate independent variables; however physical and technical variables are common in CERs. Identification of cost drivers to include in the CER depends on the type of CER being developed. Depending on the life-cycle phase of the program, CERs can be categorized into three types: research and development, production, or operating and support. (10) This research focused on aircraft production CERs. Previous research identified conventional cost drivers for aircraft CERs, to include empty weight, speed, useful load, wing area, power, landing speed, and production quantity. (11)

CERs are prevalent in many different cost-estimation techniques. They are the cornerstones of the parametric estimation technique developed by the RAND Corporation in the 1950s to predict the cost of aircraft. (12) As such, it is now the primary component underlying most commonly used parametric software estimating suites. (13) The versatility of CERs can be shown by their cross utilization among other estimation techniques. The grassroots technique uses CERs to develop detailed labor and material estimates, which are then summed as components of the total estimate. (14) Because CERs are versatile and widespread, they can be found in virtually every cost analyst's toolbox.

There are several characteristics that make CERs desirable across these cost-estimation techniques. First, they are able to "provide quick estimates without a great deal of detailed information." (15) This is important since a CER can be used early in a program's life, before any actual data are available, to forecast and plan for future budgets. Second, because CERs are based on historical data, they incorporate the impacts of system growth, schedule changes, and engineering changes. (16) These changes are a fact of virtually every DoD program. Because these items are part of the historical data, the CER is able to give a more realistic picture of the future. Most important, CERs have proven to be good predictors, which is the goal of any cost-estimation technique.

Aircraft Estimation Techniques

A variety of techniques for developing aircraft cost estimates is available to the cost analyst. The two ends of the estimation technique spectrum are parametric estimation and grassroots estimation. The parametric estimation technique can be considered a macro approach to cost estimation, while the grassroots approach is consistent with a micro approach to cost estimation.

Parametric Estimation. In today's acquisition environment of doing more with less, parametric estimating has become a common tool for the cost analyst. Parametric estimation can be defined as:

   A technique employing one or more CERs and associated
   mathematical relationships and logic. The technique is used to
   measure and/or estimate the cost associated with the development,
   manufacture, or modification of a specified end item. The
   measurement is based on the technical, physical, or other end item
   characteristics. (17)

The CERs developed to populate the parametric cost model are typically derived through nonexperimental regression techniques. (18)

The parametric cost model represents the macro approach to estimation for several reasons. First, the focus is on high-level cost drivers and high-level data from which trends can be extracted. (20) Second, the parametric method often is used early in the acquisition cycle when program and technical definition is limited. At this point in the life cycle, the details needed to develop a comprehensive estimate are scarce, so the parametric estimate is a more useful estimation tool. Finally, capturing total program costs can be accomplished with a single parametric model. (21) This one-size-fits-all approach can be characterized as a macro technique.

Grassroots Estimation. The grassroots technique for cost estimation is synonymous with the phrases detailed, bottom-up, and engineering buildup. (22) As implied, the underlying crux of a grassroots estimate is to start at the lowest level of the work breakdown structure, estimate the components, and sum their parts. For this reason, the grassroots estimation technique is categorized as a micro approach to cost estimation.

Applicable Past Research

This research is the first of its kind to explore a statistical comparison of micro versus macro cost-estimating techniques. A critical component for this comparative analysis is the development of CERs for level one and level two WBS elements. The RAND Corporation is a leading organization in analyzing and hypothesizing aircraft CERs. (23) RAND studies on estimating aircraft airframe costs date back to the 1960s. Several components of these studies are relevant to this research effort. For example, while analyzing airframe components for a study, CERs were developed at the lowest level and compared to the aggregate level. In addition, RAND has examined the benefits and detriments to segregating CERs by aircraft categories. RAND also has completed extensive research in identifying those explanatory variables that are of most significance when developing regression models for aircraft airframes. This research examined elements of these studies, to include the segregation of aircraft by categories, identifying explanatory variables to derive CERs, and analyzing the validity of micro versus macro cost-estimation techniques.

Methodology Overview

The analysis began by segregating the aircraft cost data into the aircraft category subsets of fighter and intertheater airlift and by their macro and micro components of flyaway cost, basic airframe, and other air vehicle. Next, multiple regression equations were developed for each of these categories, six total. A Monte Carlo simulation then was applied to these regression equations. Specifically, the bootstrap technique is used to estimate the standard error of the equations. The resulting distribution from the differences of the standard error of the micro (basic airframe and other air vehicle) versus macro (flyaway cost) equations was analyzed to answer the original research questions.

Data

Total cost and component cost data for aircraft are required for this micro versus macro analysis. Two primary sources were used to gather data. The main source of data was the Cost Estimating System, Volume 2, Aircraft Cost Handbook, Book 1: Aircraft, November 1987, which was prepared for the Air Force Cost Analysis Agent by Delta Research Corporation. This data source provided information on the F-15, F-16, F-18, B-1, C-5, C-130, and C-141. The Delta Research Corporation generated the data for its study through interaction with the system program offices, contractor cost data reports, and their associated contractors. In addition to the data gathered through the Delta Research Corporation study, data were collected directly from the system program offices for aircraft under consideration that were not included in the study. This applies to data from the C-17.

The primary benefit of using data from the Delta Research Corporation is that they are normalized to constant year 1987 dollars. The C-17 data were adjusted manually through the use of Office of the Secretary of Defense inflation indices to normalize to constant year 1987 dollars. This normalization provides a homogeneous database for the purpose of analysis.

Although both recurring and nonrecurring cost data were available, only recurring data were used for this analysis. Recurring costs are incurred on an ongoing basis, such as final assembly, while nonrecurring costs are made up of one-time expenses such as initial tooling and production planning. Because these two categories are influenced by different sets of predictors, they typically are estimated separately by cost analysts. Not separating them for this analysis would add unnecessary variance to the results, hampering a comparison of the macro and micro techniques. (23)

To facilitate the analysis, the data were segregated into two distinct categories, based on aircraft type, to achieve homogeneity in the data sample. The two categories are fighters and intertheater airlift. The fighter category is composed of the F-15, F-16, F-18, and B-1. The intertheater airlift category consists of the C-17, C-5, C-130, and C-141. In addition to segregation by category, the data also will be subdivided by WBS level. This WBS breakdown will consist of flyaway cost, which is analogous to level one of the work breakdown structure. The two analogous components for WBS level two are the basic airframe costs and other air vehicle costs.

Data Limitations

There are two limitations with these data. The major limitation is that the majority of the data are from pre-1987. This is because of the limited availability of the Delta Research Corporation database. Since there are not much data available from newer systems such as the F-22 or joint strike fighter, this is not a debilitating limitation. However, research would benefit from obtaining additional data points from more recent history. The other limitation results from the WBS-level breakdown. Once again, because of the available data, a comparison between level one and level two of the work breakdown structure is analyzed. Practitioners may object that cost estimation normally does not occur at level one. Thus, future research may want to look at a different database that can be broken down for a level two versus level three comparison.

Variables

The development of high-fidelity CERs is crucial to making an accurate micro versus macro cost comparison. The variables, especially the independent variables selected, play a critical role in this CER development process. The dependent variable was cost since the goal of this research was to determine whether there is a difference in the resulting cost estimates based on the approach taken. Research has demonstrated that performance parameters are the most useful and accurate independent variables used for aircraft CERs. (24) Additionally, the RAND Corporation has published several studies that indicate weight and speed are the most important variables for aircraft CERs. (25) Therefore, performance parameters, physical characteristics, and technical variables all will be considered as independent variables in developing the aircraft CERs to ensure a robust model. The independent variables investigated for inclusion in the model are found in Table 1.

Regression

A multiple regression methodology will be used to develop the aircraft CERs. In total, six regression equations will be developed in the form of:

Y = [[beta].sub."] + [[beta].sub.+] [X.sub.+] + [[beta].sub.-] [X.sub.-] + [epsilon]

where Y is the dependent variable (cost), [beta] is the regression coefficient, X is the independent variables, and [epsilon] is the error term. The six regression equations consist of a flyaway cost, basic airframe, and other air vehicle equation for each of the two categories: fighters and intertheater airlift.

Monte Carlo Simulation

After the development of the regression equations is complete, the use of a Monte Carlo method is applied. The term Monte Carlo is very generic, as it can be applied to a multitude of differing methods. (26) "In a Monte Carlo method, the quantity to be calculated is interpreted in a stochastic model and, subsequently, estimated by random sampling." (27) Therefore, for an experiment to be considered a Monte Carlo experiment, it must involve the use of random numbers to examine a problem. This technique can be applied to a variety of problems.

The Monte Carlo simulation will generate multiple outcomes for the regression equations for basic airframe, other air vehicle, and flyaway costs. Commercially available software, Crystal Ball, is implemented to accomplish this simulation. The error terms from the regression equations are modeled as random variables with a probability distribution. These errors will follow a normal (0, [[sigma].sup.2]) distribution because of the underlying assumption of normality of the residuals from the regression.

   To perform simulation in a spreadsheet, we must first place a random
   number generator formula in each cell that represents a random, or
   uncertain, independent variable. Each random number generator
   provides a sample observation from an appropriate distribution that
   represents the range and frequency of possible values for the
   variable. (28)

Bootstrap

The bootstrap technique and resampling are intrinsically tied. The underlying construct behind bootstrap resampling is that the original sample is considered the best estimate of the population. The resampling occurs as one samples the sample. (29) Thus, the essence of the bootstrap technique is:

   That in many complex situations, where bootstrap statistics are
   awkward to compute, they may be approximated by Monte Carlo
   "resampling." That is, same-size resamples may be drawn repeatedly
   from the original sample, the value of a statistic computed for each
   individual resample, and the bootstrap statistic approximated by
   taking an average of an appropriate function of these numbers. (30)

Figure 1 illustrates how a simple bootstrap sample is constructed. It is important to note that sampling occurs with replacement.

The bootstrap technique is used widely with regression equations. Previous research on estimating the standard error of multiple regression equations found "model-based resampling will give adequate results for standard error calculations." (31) The specific regression resampling approach required for this research is the Fixed X, residual resampling. (32) This approach, as proposed by Stine, is a two-step process. First, a regression model must be fit and the residuals computed. Second, the bootstrap data are generated by

Y* = (Fit) + (BS sample of OLS residuals)

where Y* is the dependent variable and Fit is the Fixed X portion of the regression equation. It is important to note that, under this method, the "residual resampling keeps the same Xs in every bootstrap sample." (33)

Application of the Bootstrap, Monte Carlo, and Crystal Ball. The idea of a bootstrap is to estimate a characteristic (X*) of a population distribution, such as the standard deviation or mean, "by resampling from a distribution determined by the original sample X." (34) Monte Carlo techniques and Crystal Ball can be used in combination to apply this bootstrap technique.

The statistic of interest for the macro versus micro comparison in this research is the standard error of the regression equation. Using the bootstrap function in Crystal Ball, the regression equation as the forecast cell, and the residual normal (0, [[sigma].sup.2]) distribution as the assumption cell, the standard error can be calculated for each equation. "As a rule of thumb, about 200 samples are needed for finding a standard error." (35)

Drawing Conclusions. The distribution resulting from the pairing of the data points from the bootstrap results will be examined. An analysis of this distribution, to include the mean and a 95-percent confidence interval around the mean, will be used to determine if the mean is significantly different from zero. If it is not different from zero, it can be concluded that the error of the two equations is statistically equivalent. If the means are statistically different, it can be concluded that there are different risks from taking a macro versus micro approach to cost estimation. Analysis of these risks at the various WBS levels enables decisions to be made about appropriate allocation of resources. Specifically, it will be possible to determine whether more resources should be allocated to the basic airframe or to the other air vehicle category.

Analysis for Multiple Regression Models

Data for Fighter Category

The data for the fighter category come from the Delta Research Corporation's report. The four aircraft under consideration are the F-15, F-16, F-18, and B-1. These aircraft were chosen for three reasons. First, they are all operational aircraft currently used by their respective service. Second, multiple production data points are available for analysis. Multiple data points enhance the probability of generating a robust model, which is imperative for conducting the regression analysis. It is important to note that this condition eliminated next-generation aircraft such as the F-22 or joint strike fighter, which do not have production data. Third, the characteristics of these aircraft provide a natural grouping that allows for a homogeneous database. The final database consisting of the four aircraft has 47 data points.

Data for Intertheater Airlift Category

The data for the intertheater airlift category come from two sources, the Delta Research Corporation report and system program offices. The four aircraft under consideration are the C-130, C-141, C-5, and C-17. These aircraft were chosen for several reasons. First, they are all operational aircraft currently used by their service. Second, there are multiple data points available from which to conduct the analysis. Third, the characteristics of these aircraft provide a natural grouping that allows for a homogeneous database.

Development of the regression model for the basic airframe, other air vehicle, and flyaway cost all had one common result. The parameter estimates for the C-141 data were found to be insignificant in all models. As the C-141 had the least amount of data points, this is not a major limitation, and the C-141 data were discarded.

Preliminary Modeling Problem

Initial development of the regression models included consideration of all the independent variables listed in Table 1. As shown in Table 2 with a portion of the F-16 data, there is duplicity in many of the independent variables. For example, although the average lot cost decreases as subsequent lot buys occur, the wing area remains constant at 300 square feet. While the learning curve effect is captured with variables such as cumulative quantity and lot size, a bias is introduced into the regression by the duplicate independent variables.

There are two potential solutions to this problem. First, changes in the performance parameters and physical characteristics occur as the aircraft changes (that is, from an F-15 to an F-16) and as the aircraft model changes. For instance, when the F-15 was updated to the C model, the maximum internal fuel characteristic changed. Thus, one way to model the regression is to make a qualitative independent variable that represents an aircraft that has the same performance parameters and physical characteristics. The learning curve portion of the regression model still would be captured through independent quantity variables. A major benefit to this approach is that all 47 data points would remain in the model. The major detriment to this approach is that the independent variables may not be meaningful to the practitioner. However, it is important to note that the objective is not to have a practitioner use the regression equations but rather to achieve the best estimate of the standard error of the regression equation for comparative purposes.

The second option would be to use only one data point from each aircraft at a specific quantity, such as 100. This option would alleviate the bias found in the independent variables. However, this approach would result in a regression model with only four data points. Therefore, the number of independent variables would be limited to two because of the degrees of freedom in the regression model. The primary benefit of this approach is that the regression equation would be useful to a practitioner. However, there are some significant problems with this approach. Preliminary models using this technique found that the B-1 was a highly influential data point. Leaving this data point in the model may invalidate the results of the regression, including the p-values associated with the independent variables, the assumptions, and the regression coefficients. (38)

To achieve the objectives of a comparison of the micro and macro approaches to cost estimation, the validity of the errors resulting from the regression models must be of the highest quality. Therefore, the first solution of using qualitative variables is the preferred solution to this problem. This method provides a mathematical model that best estimates the errors.

Results

The residual term is the item of interest to perform the macro versus micro comparison. Table 3 shows the resultant residuals from each of the regression equations. The Kolmogorov-Smirnov Test returned a p-value of >0.15 for the residuals of each equation, validating their normal distributions. These distributions are critical as inputs to the bootstrap technique that will be used to perform the macro versus micro comparison.

Generation of the regression equations leads to the next step in the analysis: fixed X, residual resampling. Beginning with the fighter category, a comparison of the flyaway and basic airframe component is considered. Starting with the flyaway regression equation, Crystal Ball performs the bootstrap technique. Next, the bootstrap technique is replicated for the basic airframe category. The standard error of the resulting 200 bootstrap samples from the flyaway and basic airframe categories are then differenced. The differenced data distribution allows for a comparison of the macro versus micro techniques. The mean of the distribution is -0.0208 with a 95-percent confidence interval of -0.0195 to -0.0222

The bootstrap technique is applied in an identical manner for the other air vehicle data as it was for the flyaway and basic airframe components. The resulting 200 standard deviation samples from the flyaway data and other air vehicle were differenced. The mean of the distribution is -0.0445. The 95-percent confidence interval is -0.0429 to -0.046.

The same procedure is applied to the intertheater airlift category. First, the basic airframe versus flyaway is considered. The mean of the distribution is -0.027. The 95-percent confidence interval is -0.026 to -0.028. Next comes the other air vehicle versus flyaway. The mean of the distribution is -0.0732. The 95-percent confidence interval is -0.0722 to -0.0742. The four resulting distributions are the basis for the conclusions.

Importance of Findings

This research is important for several reasons. First, the cost-analysis career field is shrinking. As demonstrated by the ASC example, there has been a dramatic reduction in cost authorizations over the last decade. Cost analysts, therefore, are becoming a scarce resource. When confronted with the challenge of developing a cost estimate, program managers need to know how to optimize this resource. By understanding the advantages and disadvantages from an estimation error risk perspective of estimating at differing WBS levels, optimal allocation of cost analysis resources can be achieved. Second, to achieve cycle-time reduction goals, the time to develop a cost estimate is compressed. As a result, cost estimates need to be developed more quickly, while still maintaining a satisfactory level of fidelity. This lends to the conclusion that using the time-consuming grassroots techniques will not be possible. Rather, estimation will occur at the highest WBS level possible, while still achieving a satisfactory level of confidence in the estimate. This research provided the analysis necessary to understand the tradeoffs implicit in estimating at the differing WBS levels. When making resource allocation decisions under a constrained environment, program managers then can apply this information.

Limitations

There are several limitations to this research. First, only recurring data are considered in the analysis. The estimation error risk of nonrecurring data is not considered. Second, the weapon systems analyzed are limited to aircraft systems, specifically fighters and intertheater airlift aircraft. To extrapolate the results of the analysis to data outside aircraft weapon systems is inappropriate. Likewise, to extrapolate the results to other categories of aircraft, such as bombers, is inappropriate. Third, the WBS level comparison is limited to level one versus level two. Conclusions about lower WBS levels are not considered. Finally, the WBS level two breakdown is not a pure MIL-HDBK-881 breakout.

Conclusions can be drawn only about a level one versus level two comparison with regard to the breakout of WBS level two into the basic airframe and other air vehicle components.

Discussion of Results

Starting with the fighter category, there is a statistically significant difference between estimating at the flyaway cost level versus the basic airframe and other air vehicle level. This is confirmed by the 95-percent confidence intervals around the mean of the differenced distribution, which do not contain zero for either model comparison. For the flyaway cost versus basic airframe model, the mean of the distribution is -0.0208 with a 95-percent confidence interval of (-0.0195, -0.0222). For the flyaway cost versus other air vehicle model, the mean is -0.0445 with a 95-percent confidence interval of (-0.0429, -0.046). Several additional conclusions can be drawn from this. First, there is clearly more error risk in the estimation of the other air vehicle model than the basic airframe model. This indicates that program managers should allocate more time and resources to the development of the other air vehicle estimate than to the basic airframe estimate if the estimate is being developed at WBS level two. The second conclusion was one not anticipated when the research began. The differenced distributions are calculated by subtracting the WBS level two data from the WBS level one data. As shown above, the mean and resultant 95-percent confidence intervals of both these distributions are negative. This leads to the conclusion that estimating at WBS level one has less error risk than estimating at WBS level two. There are several possible reasons for this. It could be that when estimating at the lower levels, the details of the estimate cloud the bigger picture, leading to inaccurate or inappropriate model inputs from experts. In other words, it may be harder to break down an estimate to the individual components without adding additional error. Another possible explanation is that the positive and negative error risks in the individual components cancel each other out as they accumulate at higher levels. Although this research cannot conclude with any certainty why the WBS level one error risk is less than the WBS level two error risk, the above possibilities are reasonable explanations.

The results from the intertheater airlift category are similar. There is a statistically significant difference in the estimating error between estimating at WBS level one and level two. The mean of the distribution for the flyaway cost versus basic airframe is -0.027 with a 95-percent confidence interval of (-0.026, -0.028). The mean of the distribution for the flyaway cost versus other air vehicle is -0.0732 with a 95-percent confidence interval of (-0.0722, -0.0742). As neither confidence interval encompasses zero, it is appropriate to say that there is a statistical difference between the two. Like the fighter category conclusions, there is more estimation error risk in the other air vehicle model than the basic airframe. This indicates that program managers should allocate more resources to the other air vehicle portion of their estimates. Also, as with the fighter category results, it is determined that there is more estimation error risk when estimating at WBS level two than at WBS level one. The same rationale explained for the fighter category is applicable to the intertheater airlift results.

Practical Versus Statistical Significance

Despite the conclusions above regarding the statistically significant differences between estimating at the varying WBS levels, there is a practical application perspective to consider. The estimation errors from the models are extremely small considering the multimillion dollar costs of aircraft weapon systems. Quantitatively, the dollar amount differences are shown in Table 4.

These dollar amounts are so small that, although there is a statistically significant difference, there is little difference from a practical standpoint. In most cases, the error risk simply is not large enough for program managers to be concerned when allocating resources. As a result, it is anticipated that program managers will allocate resources based on other considerations, such as time constraints or desired level of visibility into the estimate.

Future Research

There are several areas related to the methodology of this research that can be explored in future research. First, an examination of the nonrecurring estimating error between differing WBS levels could be examined. This is a natural extension of the recurring estimation error analyzed in this research. Second, a comparison of the estimation error difference at WBS level two versus WBS level three could be explored. Although other variations of WBS level comparisons could be made, a level two versus level three would be most useful to the practitioner. Third, this methodology could be applied to different weapon systems than aircraft. These future research areas would be a natural bridge to the limitations described above.

Table 1. Independent Variables Considered for CER Development

Dimensions         Weight (Pounds)    Engines
(Feet)

Wingspan           Airframe Unit      Number of Engines
                   Weight

Wing Area          Empty Weight       Max Static Thrust Sea
(square feet)                         Level

Length Wingspan    Max Gross
                   Takeoff Weight

Height             Max Landing
                   Weight

Tread

Wheelbase

Dimensions         Performance        Fuel            Quantity
(Feet)

Wingspan           Takeoff Weight     Max Fuel        Lot Number
                                      Internal

Wing Area          Takeoff Run SL     Max Fuel        Cumulative
(square feet)      (feet)             External        Quantity

Length Wingspan    Max Speed SL
                   (knots)

Height             Max Altitude (knots)

Tread

Wheelbase

Table 2. Portion of F-16 Independent Variables Data

A/C        Lot #    Lot Qty    Avg Lot Cost    Wingspan

F-16         1          8           23.66        32.8
F-16         2         55            7.84        32.8
F-16         3        105            9.02        32.8
F-16         4        145             713        32.8
F-16         5         75            6.74        32.8
F-16         6        348            7.44        32.8
F-16         7        175            5.28        32.8
F-16         6        180            5.23        32.8
F-16         9        160            5.11        32.8
F-16        10        120            6.68        32.8
F-16        11        144            5.85        32.8
F-16        12        150            6.26        32.8

A/C      Wing Area                  Length       Height
       (Square Feet)

F-16       300                       49.4        16.4
F-16       300                       49.4        16.4
F-16       300                       49.4        16.4
F-16       300                       49.4        16.4
F-16       300                       49.4        16.4
F-16       300                       49.4        16.4
F-16       300                       49.4        16.4
F-16       300                       49.4        16.4
F-16       300                       49.4        16.4
F-16       300                       49.4        16.4
F-16       300                       49.4        16.4
F-16       300                       49.4        16.4

A/C      Tread    Wheelbase

F-16       7.8       13.1
F-16       7.8       13.1
F-16       7.8       13.1
F-16       7.8       13.1
F-16       7.8       13.1
F-16       7.8       13.1
F-16       7.8       13.1
F-16       7.8       13.1
F-16       7.8       13.1
F-16       7.8       13.1
F-16       7.8       13.1
F-16       7.8       13.1

Table 3. Residual Distribution Parameters
from Regression Equations

                                       Standard
Fighter Category               Mean    Deviation

Basic Airframe                  0        .334
Other Air Vehicle               0        .356
Flyaway                         0        .313
Intertheater Airlift
Category
Basic Airframe                  0        .225
Other Air Vehicle               0        .272
Flyaway                         0        .199

Table 4. Practical Significance of WBS Estimation Levels

Category                   Fighter      Intertheater Airlift

Flyaway vs Basic         $44,137.60          $68,682.90
Airframe

Flyaway vs Other         $124,097.40        $157,584.96
Air Vehicle

Figure 1. Constructing Bootstrap Samples (36)

Original process

Population [right arrow] ([X.sub.1], [X.sub.2], ..., [X.sub.n]) [right
arrow] [bar.X]

Resampling process

BS Sample 1: ([X.sub.3], [X.sub.7], ..., [X.sub.3]) [right arrow]
[bar.X.sub.1*]

BS Sample 2: ([X.sub.8], [X.sub.1], ..., [X.sub.1]) [right arrow]
[bar.*X.sub.2]

....

BS Sample B: ([X.sub.4], [X.sub.9], ..., [X.sub.11]) [right arrow]
[bar.X*.sub.B*]

Notes

(1.) George Cho, Hans Jerrel, and William Landley, Program Management 2000: Know the Way--How Knowledge Management Can Improve DoD Acquisitions, Ft Belvoir, Virginia: Defense Systems Management College Report of the Military Research Fellows, Jan 00, 1-1.

(2.) Kathy Ruffner, "Growing Cost Estimators," presentation to SAF/ FM, Wright-Patterson AFB, Ohio, 9 Jan 02.

(3.) Conrad S. Ciccotello, Steve G. Green, and Martin J. Hornyak, "Rethinking Twenty-First Century Acquisition: Emerging Trends for Efficiency Ends," Acquisition Review Quarterly, Winter 1997, 28.

(4.) AFSC Cost-Estimating Handbook Series, Reading, Massachusetts, prepared for the Air Force Systems Command, 1986 [Online] Available: https://fmfweb.wpafb.af.mil/fmweb/fmc/estimating handbooks/estimatinghandbooks.htm, 217.

(5.) Department of Defense, Department of Defense Handbook: Work Breakdown Structure, MIL-HDBK-88, Washington, Part 4, 1996.

(6.) Quentin W. Hemming and Joel M. Koppelman, Earned Value Project Management, 2d Edition, Newton Square, Pennsylvania: Project Management Institute, 2000, 54.

(7.) Office of the Deputy Director of Defense Procurement for Cost, Pricing, and Finance, Contract Pricing Reference Guides, Vol 2, Quantitative Techniques for Contract Pricing, Washington, 19 Jun 00, 1.

(8.) Obaid Younassi, Michael Kennedy, and John Graser, Military Aircraft Costs: The Effects of Advanced Materials and Manufacturing Processes, Santa Monica, California: RAND, 2001, 83.

(9.) John A. Long, "Parametric Cost Estimating in the New Millennium," ISPA SCEA Joint International Conference Papers, 2000, 2.

(10.) AFSC Cost-Estimating Handbook, 6:13:1.

(11.) Contract Pricing Reference Guides, 3.

(12.) Nhysoft, Inc, "A Brief History of Parametric Estimating" [Online] Available: www.nhysoft.com/history.htm, 13 Aug 02.

(13.) Air Force Materiel Command, AFMC--Price Estimating Suite 2000 [Online] Available: http://web2.deskbook.osd.mil/valhtml/2/2B/2B4/ 2B4S03.htm, Apr 01.

(14.) AFSC Cost Estimating Handbook 4:1.

(15.) Long, 2.

(16.) AFSC Cost-Estimating Handbook, 4:4:1.

(17.) Department of Defense (c), Parametric Cost Estimating Handbook, 14 Sep 95, 16.

(18.) J.L. Robbins and J.C. Daneman, "Parametric Estimating & the Stepwise Statistical Technique," National Estimator, Spring 1999, 14.

(19.) Nhysoft.

(20.) AFSC Cost-Estimating Handbook, 3:4:1.

(21.) AFSC Cost-Estimating Handbook, 3:4:3.

(22.) Mike Seibel, Chief, Research Section, Acquisition Cost Division, Aeronautical System Center, electronic message, Aug 02.

(23.) Seibel.

(24.) AFSC Cost-Estimating Handbook, 6:1.

(25.) Ronald Hess and H. P. Romanoff, Aircroft Airframe Cost-Estimating Relationships, Santa Monica, California: RAND, R-3255-AF, 1987.

(26.) Joy Woller, class note Chem 484, The Basics of Monte Carlo Simulations, University of Nebraska-Lincoln [Online] Available: http://www.chem.unl.edu/zeng/joy/mclab/mcintro.html, Spring 1996, 1.

(27.) Harold Niederreiter, Random Number Generation and Quasi-Monte Carlo Methods, Philadelphia: Society for Industrial and Applied Mathematics, 1992, 3.

(28.) Cliff T. Ragsdale, Spreadsheet Modeling and Decision Analysis 3d Edition, South-Western College Publishing, 2000, 567.

(29.) Robert Stine, class handout, "Bootstrap Resampling," ICPSR Blalock Lectures, Department of Statistics, Wharton School, 2002, 1:9.

(30.) Peter Hall, The Bootstrap and Edgeworth Expansion, New York: Springer-Verlag, 1992, 1.

(31.) Anthony Davison and D. Hinkle. Bootstrap Methods and Their Applications, New York: Cambridge University Press, 1997, 276.

(32.) Stine, Chap 3:11.

(33.) Stine, Chap 3:23.

(34.) Hall, 7.

(35.) Stine, Chap 1:6.

(36.) Stine, Chap 1:10.

(37.) Spyros Makridakis, Steven C. Wheelwright. and Rob J. Hyndman, Forecasting Methods and Applications 3'1 Edition, New York: John Wiley & Sons Inc. 1998, 213.

Captain Ritschel is deputy chief of Cost Analysis at the Standard Systems Group, Maxwell AFB, Gunter Annex. Alabama. This research, conducted while a master's student at the Air Force Institute of Technology (AFIT), Wright-Patterson AFB, Ohio, won him the SECA Outstanding Thesis award in 2003. Major Greiner is the director of the Graduate Cost Program at AFIT. Mr Reynolds is an assistant professor of mathematics at AFIT. Mr Seibel is the chief of Cost Research, Aeronautical Systems Center, Wright-Patterson.

COPYRIGHT 2003 U.S. Air Force, Logistics Management Agency
COPYRIGHT 2004 Gale Group