Selection and prioritization of projects--a Data Envelopment Analysis (DEA) approach.

Lall, Vinod ; Lumb, Ruth ; Moreno, Abel 等

Abstract

When an organization has multiple projects to be initiated the challenge it faces is the lack of a methodology that would select and prioritize projects that compete for limited resources. This research applies the Data Envelopment Analysis (DEA) approach to the selection and prioritization of projects. Published research will be used to identify factors used in the selection of projects, classify these factors into inputs and outputs, then develop and solve a DEA model for each project. Results from the solved DEA models will be analyzed to identify highly efficient projects and make recommendations on how to improve inefficient projects.

Keywords: DEA, DEA inputs, DEA outputs, project selection

I. INTRODUCTION

Decision making is at the core of all management functions. Managers are constantly called upon to make decisions in order to solve problems and/or to select one course of action from several possible alternative actions in order to obtain the goals and objectives of the organization. Since decisions direct actions, decisions regarding an organization's, resources, strengths, weaknesses, and future growth are all important factors that will have a considerable impact on the performance of a firm and which will determine the success or failure of the firm. For the past two decades, a factor that has had a significant impact on decision making in many firms is globalization. During the 1990's the forces of globalization (i.e., new demands of international competition and dramatic advances in technology) substantially changed the nature and operation of markets and organization of the production function in many industries throughout the world. As a result, today's highly competitive and demand driven market has put increased pressure on management to allocate and utilize resources appropriately in an effort to achieve optimal performance efficiently. Since decisions may be related to the allocation of scarce organizational resources, some of which involve substantial resources, may be difficult to reverse and can affect a company's business into the future, it is important that decisions are made that allow a firm to operate as efficiently and effectively as possible with the given resources.

One of the results of globalization is that it has had a huge impact on the way that organizations perform activities. In order for firms to keep pace with the fast changing environment there is a greater emphasis on project management. Project management was primarily driven by firms that realized the benefits not only of organizing work around projects, but also the need to communicate and coordinate tasks across departments and professions. It is an effective way of dealing with international projects. For project managers and their teams the decision making process often involves selecting one alternative project from several alternative projects. The decision making may be complicated since one or more projects selected from competing projects may be evaluated according to different criteria. Some projects require multiple decision makers and difficulties may arise due to different goals involved. For example, an important part of decision making for competing projects is to verify and validate alternatives. This may require input not only from the project manager but also from engineers or analysts. Even if decision makers share the same selection criteria, the importance level that is attached to each criterion is not necessarily the same, due to different budgets, time factors, alternative projects under consideration etc. At times several competing projects may be considered at the same time, with no interest a priori to one or more of the projects. The decision making may be further complicated because of a large number of attributes that must be considered. As a result, in order to arrive at a viable decision, managers at times must cope with an enormous amount of data relating to competing projects. Consequently, selecting the 'best' project from a potentially large number of different projects with varying levels of capability and potential is a complicated and time-consuming task. In summary, at any time a typical organization has multiple projects to be initiated and the challenge organizations face is the lack of a methodology that would help them select and prioritize projects that simultaneously compete for limited organizational resources. This paper presents an example of how Data Envelopment Analysis (DEA) may be used as a tool for selection and prioritization of projects. A review of the last 25 years of research involving applications of DEA methodology is summarized in Table 1. This table is not meant to be a comprehensive review but rather an overview of the different applications, inputs and outputs that have been utilized with DEA.

The next section summarizes the basics of DEA and its application in managerial decision-making. This is followed by a section that summarizes a DEA approach for selection and prioritization of projects. Next, a DEA model for project selection decision is developed and solved. Finally, model results are analyzed and interpreted to identify managerial implications of the DEA approach to project selection.

II. DATA ENVELOPMENT ANALYSIS (DEA)

Data Development Analysis (DEA) is an application of the linear programming technique and was developed by Charnes et al. (1978) to measure the relative efficiencies of options which involve multiple, incommensurate inputs and outputs. These options are referred to as decision-making units (DMUs). DEA has found a variety of applications in several areas and has been used to measure the performance of physician practices, component suppliers, school districts, banks hospitals, robots, courts etc. Several of these applications were summarized in Table 1 under section I. Lall and Teyarachakul (2006), Thanassoulis et al. (1978), Boussofiane et al. (1991) and several other papers addressed the fact that information obtained from DEA assessment can be used to discover which DMUs can be classified as efficient or inefficient, identify possible good operational practices and explore the possibility of setting targets for inefficient units. Banker and Morey (1986) presented the DEA formulation to evaluate the efficiency of DMUs when some of the inputs and outputs are exogenously fixed and beyond the control of the DMUs. Recently, DEA has been integrated with the multiple-objective linear programming (MOLP) as an interactive approach to a resource-allocation problem in organizations with a centralized decision-making environment. Golany (1988) proposed the use of preference information when setting the performance targets in the context of DEA. Sutton and Green (2002) used the DEA notion to evaluate decision choices. They suggested the modified DEA to find weights which show the performance of options and to provide a framework to elicit and use information exogenous to the decision alternatives. The efficiency score of each DMU is determined by the weighted sum of outputs divided by the weighted sum of inputs. Charnes et al. (1978) recognized the difficulty in seeking common weights because each DMU may value inputs and output differently; they proposed to use a set of weights that give the highest possible relative efficiency scores.

The fractional form of DEA, which maximize the efficiency h0 of the j0 DMU is defined as follows:

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (Model M1)

where

[y.sub.rj] = the amount of the [r.sup.th] output from unit j,

[u.sub.r] = the weight given to the [r.sup.th] output,

[x.sub.ij] = the amount of the [i.sup.th] input to the unit j,

[v.sub.i] = the weight given to the [i.sup.th] input, and

[epsilon] = a very small positive number

Charnes and Cooper (1962) provide approaches to convert Model M1 into a linear programming model by setting the denominator in the objective function to some arbitrary constant and moving the denominator in the first constraints to the right-hand side of the constraint. For computational convenience, the DEA linear programming model is converted into a dual model as follows:

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (Model M2)

where [[lambda].sub.j], [[bar.s].sub.i] [s.sup.+.sub.r] are the dual variables.

There are alternatives to measure the efficiency of a DMU. One may use either the input- reducing efficiency or an output-increasing efficiency measure. Both model M1 and M2 measure output-increasing efficiency. In measuring the input-reducing efficiency, the relative efficiency of a DMU (for example DMU j0) is evaluated by finding the best practice DMU's minimum effort required to produce the same amount of outputs as DMU j0 does. In other words, how much effort it takes for the best practice DMU (reference DMU) to produce as much outputs as DMU j0. We consider the application of DEA to project selection; the choices of DMU become project alternatives. For simplicity, we apply model M1 to select the best project candidate.

III. A DEA APPROACH FOR SELECTION AND PRIORITIZATION OF PROJECTS

DEA assesses the relative efficiency of DMUs by obtaining the maximum of a ratio of weighted outputs to weighted inputs. The selection criteria for competing projects will be the inputs and outputs in our study. Several selection criteria have been identified in the literature. Examples of these criteria include: Return on Investment, implementation time, clerical time, training time, net benefit, cost, efficiency, alignment with corporate strategy/goals, to name a few. Note that the units of measure of these criteria varies from $ to hours to percentages to subjective ratings. The DEA approach allows for the simultaneous use of data as it comes regardless of how different the units of measure of the output and input criteria under consideration are.

IV. DEA RESULTS

Relevant results from a DEA application are dependent upon the ratio of the number of input and output variables to the number of Decision Making Units. A rule of thumb for this ratio is given by Banker et al. (1984) as: s + m < n/3, where s is the number of inputs, m is the number of outputs and n is the number of DMUs. For illustration purposes and consistent with this rule of thumb, we will be considering 10 DMUs or projects and 4 project features. Return on investment and alignment with corporate strategy will be assumed to be outputs and implementation time and project cost will be assumed to be inputs. The data set used is included in Table 2. The original data set was obtained from McCain (2011) who applied the prioritization matrix technique to rank three alternative projects. To demonstrate the applicability of DEA to project selection and evaluation, seven additional projects with randomly assigned values of inputs and outputs were added to the original dataset. Alignment with corporate strategy is measured subjectively using a 1-5 score where 5 indicates perfect alignment. Implementation time is given in hours and cost in thousands of dollars.

Results from applying the DEA model are reported in Table 3. An examination of Table 3 indicates that projects 5, 6 and 10 exhibit a relative efficiency value of 1, meaning that for their individual return on investment percentage and alignment to corporate strategy score, no better implementation time and cost features could be offered by any of the competing projects under consideration.

The other seven projects under consideration exhibit a relative efficiency value of below 1, indicating that at least one other project in the sample offers better ROI and alignment to corporate strategy features for comparable levels (hours and $) of implementation time and cost features. As an illustration consider project 8. The DEA model suggests that project 8 is 42.4% less efficient than its reference set, namely, projects 6 and 10. An examination of the data associated with projects 8 and 6 reveals that a higher alignment score (5,4) and at least as high ROI (10,10) is attained with project 6 than with project 8 even when implementation time and cost features are higher for project 8 (6000, 1700) than for Project 6 (4000, 900). This indicates that one could expect at least as good of a return and better alignment with corporate strategy from project 6 even though it costs less and takes less time to implement than project 8. A consequence of this finding would be that in order for project 8 to be as attractive as project 6, the input variable cost would need to change, i.e. the cost of the project will have to be less and/or the implementation time feature will have to improve. As it can be seen then, the DEA results allow for an easier examination of why some projects are in fact better than others and thus provide an opportunity to determine what it would take for a given project to improve its standing relative to others in the sample. In addition, as indicated previously, the DEA approach allows for the use of various units of measure to be included simultaneously and in 'raw' form.

V. CONCLUSIONS, LIMITATIONS AND OPPORTUNITIES

In this paper, a DEA approach is proposed as an alternative procedure to assist decision-makers select the best project from several being considered. An actual data set available in the literature was modified by adding additional projects with corresponding inputs and outputs. This modified data set was used to illustrate how the DEA model works and to compare its features with those of an existing and fairly common procedure (use of informed weights and scores). In the data set used, subjective scores were assigned to the various features offered by the competing projects. Given that the DEA approach allows for the simultaneous consideration of inputs/outputs with different measurement units, a possible area of opportunity would be to replace the scores assigned to various inputs and outputs with actual raw data. For example, the output alignment to strategy could be replaced with another feature that used numerical data and not a rating. Sensitivity analysis may be performed on the results to determine what specific changes must occur in the input and output values of a project showing a relative efficiency of less than 1 in order for the package to attain a relative efficiency of 1.

References

Ahuja, G. and S. Majumdar (1998), "An Assessment of the Performance of Indian State Owned Enterprises", Journal of Productivity Analysis, Vol. 9, No. 2, 113-132.

Al-Shammari, M. (1999), "Optimization Modeling for Estimating and Enhancing Relative Efficiency with Application to Industrial Companies", European Journal of Operational Research, Vol. 115, No. 3,488-496.

Athanassopoulos, A. and J. Ballantine (1995), "Ratio and Frontier Analysis for Assessing Corporate Performance: Evidence From the Grocery Industry in the UK", Journal of Operational Research Society, Vol. 46, No.4, 427-440.

Athanassopoulos, A., N. Lambroukos, and L. Seiford (1999), "Data Envelopment Scenario Analysis for Setting Targets to Electricity Generating Plants", European Journal of Operational Research, Vol. 115, No. 3, 413-428.

Baker, R. and S. Talluri (1997), "A Closer Look at the Use of Data Envelopment Analysis for Technology Selection", Computers and Industrial Engineering, Vol. 32, No. 1, 101-108.

Banker, R. D. and R. C. Morey (1986), "Efficiency Analysis for Exogenously Fixed Inputs and Outputs", Operations Research, Vol. 34, No. 4, 513-521.

Banker, R. D., A. C. Charnes and W. W. Cooper (1984), "Some Models for Estimating Technical and Scale Inefficiencies in Data Analysis", Management Science, Vol. 39, 1078-1092.

Boussofiane, A., R. G., Dyson and Thanassoulis (1991), "Applied Data Envelopment Analysis", European Journal of Operational Research, Vol. 2, 429-444.

Bowlin, W (1987), "Evaluating the Efficiency of US Air Force Real-Property Maintenance Activities", Journal of Operational Research Society, Vol. 38, No. 2, 127-135.

Burgess, J. and P. Wilson (1993), "Technical Efficiency in Veterans Administration Hospitals", In The Measurement of Productive Efficiency: Techniques and Applications, H. Fried, C. Lovell, and S. Schmidt, eds, Oxford University Press, Inc., 335-351.

Chandra, P., W. Cooper., S. Li. and A. Rahman (1998), "Using DEA to Evaluate 29 Canadian Textile Companies--Considering Returns to Scale", Journal of Production Economics, Vol. 54, No. 2, 129-141.

Charnes, A., W. Cooper and S. Li (1988), "Using Data Envelopment Analysis to Evaluate Efficiency in the Economic Performance of Chinese Cities", Socio-Economic Planning Sciences, Vol. 23, No. 6, 325-344.

Charnes, A., W. W. Cooper and E. Rhodes (1978), "Measuring the Efficiency of the Decision Making Units", European Journal of Operational Research, Vol. 2, 429-444.

Charnes, A. and W.W. Cooper (1962), "Programming With Linear Fractional Functionals", Naval Research Logistics, Vol. 9, 181-185.

Cheng, E., Y. Chaing, and B. Tang (2007), "Alternative Approach to Credit Scoring by DEA: Evaluating Borrowers with Respect to PFI Projects", Building and Environment, Vol. 42, No. 4, 1752-1760.

Chu, S. and G. Lim (1998), "Share Performance and Profit Efficiency of Banks in an Oligopolistic Market: Evidence from Singapore", Journal of Multinational Financial Management, Vol. 8, No. 2-3, 155-168.

El-Maghary, S. and R. Lahdelma (1995), "Data Envelopment Analysis--Visualizing the Results", European Journal of Operational Research, Vol. 83, No. 3, 700-710.

El-Mashaleh, M., R. Minchin, and W. O'Brien (2007), "Management of Construction Firm Performance Using Benchmarking" Journal of Management Engineering, Vol. 23, No. 1, 10-17.

Farrell, M. (1957), "The Measurement of Productive Efficiency", Journal of the Royal Statistical Society (A, general), Vol. 120, part 3, 253-281.

Golany, B. (1988), "An Interactive MOLP Procedure for the Extension of DEA to Effectiveness Analysis", The Journal of the Operational Research Society, Vol. 39 No. 8.

Goto, M. and M. Tsutsui (1998), "Comparison of Productive and Cost Efficiencies Among Japanese and US Electric Utilities", Omega-International Journal of Management Science, Vol. 26, No. 2, 177-194.

Grosskopf, S. and V. Valdmanis (1987), "Measuring Hospital Performance: a Nonparametric Approach", Journal of Health Economics, Vol. 6, 89-107.

Hjalmarsson, L. and J. Odeck (1996), "Efficiency of Trucks in Road Construction and Maintenance: an Evaluation With Data Envelopment Analysis", Computers and Operations Research, Vol. 23, No. 4, 393-404.

Kirjavainen, T. and H. Loikkanen (1998), "Efficiency Differences of Finnish Senior Secondary Schools: an Application of DEA and Analysis", Economics of Education Review, Vol. 17, No. 4, 377-394.

Kozmetsky, G. (1998), "Comparative Performance of Global Semiconductor Companies," Omega-International Journal of Management Science, Vol. 26, No. 2, 153-175.

Lall, V., and S. Teyarachakul (2006), "Enterprise Resource Planning (ERP) Selection: A Data Envelopment Analysis (DEA) Approach", Journal of Computer Information Systems, Vol. 47, No. 1, 123-127.

Lee, H. and A. Somwaru (1993), "Share Tenancy and Efficiency in U.S. Agriculture", In The Measurement of Productive Efficiency: Techniques and Applications, H. Fried, C. Lovell, and S. Schmidt, eds., Oxford University Press, Inc., 288-299.

Lovell, P. and J. Turner (1995), "Measuring Macroeconomic Performance in the OECD a Comparison of European and non-European Countries", European Journal of Operational Research, Vol. 87, No. 3, 507-518.

McCabe, B., V. Tran and J. Ramani (2005), "Construction Prequalification Using Data Envelopment Analysis", Canadian Journal of Civil Engineering, Vol. 32, No. 1, 183-193.

McCain, C. (2011), "Use a Selection Matrix to Pick Projects, Evaluate Solutions", Quality Progress, 72.

McCarty, T. and S. Yaisawarng (1993), 'Technical Efficiency in New Jersey School Districts", In The measurement of Productive Efficiency: Techniques and Applications, H. Fried, C. Lovell, and S. Schmidt, eds., Oxford University Press, New York, 271-287.

Odeck, J. (1996), "Evaluating Efficiency of Rock Blasting Using Data Envelopment Analysis", Journal of Transportation Engineering, Vol. 122, No. 1, 41-49.

Ott, R. and M. Longnecker (2001), Statistical Methods and Data Analysis, Pacific Grove, CA.

Peck, M., C. Scheraga and R. Boisjoly (1998), "Assessing the Relative Efficiency of Aircraft Maintenance Technologies: an Application of Data Envelopment Analysis", Transportation Research Part A-Policy and Practice, Vol. 32, No. 4, 261-269.

Ray, S. and H. Kim (1995), "Cost Efficiency in the U. S. Steel Industry: a Nonparametric Analysis Using Data Envelopment Analysis", European Journal of Operational Research, Vol. 80, No. 3, 654-671.

Rouse, P., M. Putterill and D. Ryan (1997), "Towards a General Managerial Framework for Performance Measurement: A Comprehensive Highway Maintenance Application", Journal of Productivity Analysis, Vol. 8, No. 2, 127-149.

Russel, T., P. Dharmapala, L. Rothenberg and R. Thrall (1996), "DEA/AR Efficiency and Profitability of 14 Major Oil Companies in the U.S. Exploration and Production", Computers and Operation Research, Vol. 23, No. 4, 357-373.

Sutton, P. P. and R. H. Green (2002), "A Data Envelopment Approach to Decision Analysis", The Journal of the Operational Research Society, Vol. 53, 1215-1224.

Thanassoulis, E. (1995), "Assessing Police Forces in England and Wales Using Data Envelopment Analysis", European Journal of Operational Research, Vol. 87, No. 3, 641-657.

Thore, S., F. Phillips, T. Ruefli and P. Yue (1996), "DEA and The Management of The Product Cycle: the US Computer Industry, Computers and Operations Research, Vol. 23, No. 4, 341-356.

Vinter, G., S. Roznes and S. Spraggett (2006), "Using Data Envelope Analysis to Compare Project Efficiency in a Multi-Project Environment", International Journal of Project Management, Vol. 24, No. 4, 323-329.

VINOD LALL *, RUTH LUMB * AND ABEL MORENO **

* Minnesota State University-Moorhead, 1104 7th Avenue S, Moorhead, MN 56563, E-mails: lall@mnstate.edu, lumb@mnstate.edu

** School of Business, Metropolitan State College of Denver, Campus Box 45, P. O. Box 173362, Denver, CO, 80217, E-mail: morenoa@mscd.edu

Table 1
Inputs and Outputs for Different Applications of DEA

Source            Application               Inputs

Cheng             Provides banks with a     * Concession period
et al. (2007)     methodology to evaluate
                  concessionaires           * Financial risk to
                                            borrower

El-Mashaleh       Firm performance of       * Expenses
et al., (2007)    construction
                  contractors.                ** Project
                                                 management

                                              ** Safety

Vinter            Evaluating the            * Cost
et al. (2006)     performance of
                  several projects          * Work content

                                            * Level of

                                              ** Monitoring

                                              ** Uncertainty

McCabe            Pre-qualification of      * Safety record
et al. (2005)     construction
                  contractors               * Current capacity

                                            * Related work
                                            experience

Athanassopoulos   UK electricity            * Capital
et al.            generating                expenditures
(1999)
                                            * Controllable costs

                                            * Fuel (quantity)

Al-Shammari       Jordanian manufacturing   * Number of
(1999)            firms                     employees

                                            * Paid in capital

                                            * Fixed assets

Peck et al.       US aircraft               * Labor expenses
(1998)            maintenance
                                              ** Airframes/total
                                                 aircraft operating
                                                 expenses

                                              ** Aircraft engines/
                                                 total aircraft
                                                 operating expenses

                                            * Expenditures

                                              ** Airframe repairs/
                                                 total aircraft
                                                 operating expenses

                                              ** Engine repairs /
                                                 total aircraft
                                                 operating expenses

                                            * Material expenditures

                                              ** Airframes/total
                                                 aircraft operating
                                                 expenses

                                              ** Engines/total
                                                 aircraft operating
                                                 expenses

Kozmetsky         Global semiconductor      * Cost of goods sold
(1998)            companies
                                            * Selling, general,
                                            and administrative
                                            expenses

                                            * Total assets

Kirjavainen       Finnish secondary         * Hours per week *
and               schools
Loikkanen                                     ** Teaching
(1998)
                                              ** Non-teaching *

                                            * Teachers

                                              ** Experience

                                              ** Education

                                            * Admission level

                                            * Education level of
                                            students' parents

Goto and          US and Japanese           * Total number of *
Tsutsui           electric utilities        employees
(1998)
                                            * Generation capacity
                                            (mega watt)

                                            * Quantity of

                                             ** Fuel used
                                                (kilo calories)

                                             ** Power purchases
                                               (giga watt hours)

Chu and           Singapore                 * Shareholders fund *
Lim (1998)        banks
                                            * Interest expenses

                                            * Operating expenses

Chandra           Canadian textiles         * Number of
et al. (1998)     companies                 employees

                                            * Average annual
                                            investment

Ahuja and         Indian manufacturing      * Number of
Majumdar          enterprises               employees
(1998)
                                            * Net fixed assets

Rouse et al.      New Zealand               * Total expenditures
(1997)            highway                   on reseals,
                  maintenance               rehabilitation and
                                            general
                                            maintenance
                                            (contractor costs)

Baker and         Technology selection      * Cost
Talluri (1997)    (robots)
Thore et al.      US computer industry      * Repeatability (mm)
(1996)
                                            * Costs

                                             ** Raw material

                                              ** Labor

                                            * R&D expenditures

                                            * Capital investment

Russel et al.     US oil                    * Total costs incurred
(1996)            companies
                                            * Quantity

                                              ** Proved crude oil

                                              ** Proved gas

Ozcan and         US hospitals              * 1 (scalar or
McCue (1996)                                dummy variable)

Odeck (1996)      Rock blasting in          * Cost
                  Norway
                                              ** Labor

                                              ** Capital

                                              ** Commodity

Hjalmarsson       Trucks in road            * Make and model
and Odeck         construction and          year
(1996)            maintenance in
                  Norway                    * Region of operation

                                            * Capacity of the
                                            truck in tons

                                            * Costs

                                              ** Wage of driver
                                                 per year

                                              ** Fuel per year

                                              ** Rubber
                                                 accessories

                                              ** Maintenance

Thanassoulis      Police forces in          * Number of
(1995)            England and Wales
                                              ** Violent crimes

                                              ** Burglaries

                                              ** Other crimes

                                              ** Officers

Ray and           US steel industry         * Labor hours
Kim (1995)
                                            * Cost of material

Lovell et al.     Macroeconomic             * 1 (scalar or dummy
(1995)            performance of            variable)
                  European countries

El-Maghary        Finnish                   * Total expenditure
and Lahdelma      universities
(1995)                                      * Admission
                                            (acceptance rate)

Athanassopoulos   UK grocery industry       * Capital employed
and Ball (1995)
                                            * Fixed assets

                                            * Number of
                                            employees

                                            * Number of outlets

                                            * Sales area
                                            ([m.sup.2])

McCarty,          US school                 * Number of staff
Yaisawarng        districts                 per pupil
(1993)
                                            * Percentage of staff
                                            on M.S. Or PHD

                                            * Expenditure
                                            per pupil

Lee and           Share tenancy             * Fertilizers
Somwaru           in US
                                            * Pesticides
(1993)            agriculture
                                            * Seeds

                                            * Hired labor

                                            * Capital
                                            consumption

Eeckaut           Belgian municipalities    * Total operating
et al. (1993)                               expenses

Burgess and       US veterans hospitals     * Number of
Wilson (1993)
                                              ** Acute care
                                                 hospital beds

                                              ** Long term
                                                 hospital bids

                                            * Clinical labor

                                            * Non-clinical labor

                                            * Physician hours

Charnes           Chinese cities            * Number of staff
et al. (1988)                               and labor

                                            * Working fund

                                            * Investments in
                                            construction and
                                            acquisitions of
                                            machinery

Grosskopf and     US hospitals              * Number of
Valdmanis                                   physicians
(1987)
                                            * Non-physician labor

                                            * Admissions

                                            * Net plant asset

Bowlin (1987)     US Air Force              * Supply costs
                  real-property
                  maintenance               * Available direct
                                            labor hrs

                                            * Available passenger
                                            carrying vehicle
                                            (vehicles)

Source            Outputs

Cheng             * Toll setting up and
et al. (2007)     adjusting mechanism

                  * Total investment
                  schedule

                  * Attractiveness of main
                  loan

                    ** Financial

                    ** Analysis

                  * Strength of other
                  participants

                  * Net present value

                  * Internal rate of return

El-Mashaleh       * Performance
et al., (2007)
                    ** Schedule

                    ** Cost

                    ** Safety

                  * Customer satisfaction

                  * Profit

Vinter            * Performance
et al. (2006)
                    ** Schedule

                    ** Cost

                    ** Design

                  * Documentation

McCabe            * Sales history
et al. (2005)
                  * Employee experience

Athanassopoulos   * Electricity produced
et al.            (megawatt-hour)
(1999)
                  * Plant availability (%)

                  * l/ Number of accidents
                  incurred

                  * 1/ Generated pollution

Al-Shammari       * Market value per share
(1999)
                  * Net sales

                  * Net income after taxes

                  * Percentage of all
Peck et al.       scheduled flight arrivals
(1998)            not delayed for
                  mechanical reasons

Kozmetsky         * Net sales
(1998)

Kirjavainen       * Number of students who
and               passed their grade
Loikkanen
(1998)            * Number of graduates

                  * Score of students in
                  compulsory subjects in
                  matriculation
                  examination

                  * Score of students in
                  additional subjects in
                  matriculation
                  examination

Goto and          * Quantity of electricity
Tsutsui
(1998)            * Sold to residential
                  customers (giga watt
                  hours)

                  Sold to non-residential
                  customers (commercial,
                  industrial, others)

Chu and           * Annual increase in
Lim (1998)        average assets

                  * Total income

                  * Profits

Chandra           * Annual sales
et al. (1998)

Ahuja and         * Net value added
Majumdar
(1998)

Rouse et al.      * Kilometers of
(1997)
                    ** Highway resealed

                    ** Highway rehabilitated

                  * General maintenance as
                  measured by an index of
                  highway surface defects

                  * Level of service as
                  measured by annual
                  vehicle kilometers

                  * Roughness measures
                  combined for urban and
                  rural highways

                  * Categorical variable
                  (an assessment of
                  environmental difficulty
                  faced; geology and
                  climate)

Baker and         * Load capacity (kg)
Talluri (1997)
Thore et al.      * Velocity (m/s)
(1996)
                  * Sales revenues

                  * Profits

                  * Market capitalization
                  (number of shares
                  outstanding multiplied
                  by the stock price)

Russel et al.     * Quantity
(1996)
                    ** Crude oil

                    ** Gas

Ozcan and         * Return on assets
McCue (1996)
                  * Operating cash flow per
                  bed

                  * Operating margin

                  * Total asset turnover

Odeck (1996)      * Blasted rock volume
                  (m.sup.3])

Hjalmarsson       * Transportation work
and Odeck         in kilometers per year
(1996)

                  * Volume transported in
                  cubic per year

                  * Effective hours in
                  production per year

Thanassoulis      * Number of
(1995)
                    ** Violent crime clear
                       ups

                    ** Burglary crime clear
                       ups

                    ** Other crime clear ups

Ray and           * Quantity (weighted
Kim (1995)        index of quantities
                  shipped of 80 different
                  steel products)

Lovell et al.     * GPD per capita
(1995)
                  * 1/ inflation

                  * Employment rate

                  * Trade balance
                  (Exports/Imports)

                  * 1/ (carbon emissions
                  in millions of tons per
                  capita)

                  * 1/ (nitrogen emissions in
                  millions of tons per
                  capita)

El-Maghary        * Number of graduates
and Lahdelma
(1995)            * Number of post
                  graduates

                  * Graduation speed
                  (1/years)

                  * Completion

Athanassopoulos   * Total sales
and Ball (1995)

McCarty,          * Percentage of students
Yaisawarng
(1993)              ** Who pass HSPT test

                    ** Who pass MPCT test

                    ** Who pass RPCT test

Lee and           * Revenues
Somwaru
(1993)

Eeckaut           * Total population
et al. (1993)
                  * Length of roads to be
                  maintained

                  * Number of

                    ** Senior citizens

                    ** Crimes registered in
                       the municipality

                    ** Students enrolled in
                       primary schools

Burgess and       * Inpatient days
Wilson (1993)
                  * Number of

                    ** Inpatient discharges

                    ** Outpatient visits

                    ** Ambulatory surgical
                       procedures

                    ** Inpatient surgical
                  procedures

Charnes           * Gross industrial output
et al. (1988)     value

                  * Profit and taxes

                  * Retail sales

Grosskopf and     * Acute care (inpatient
Valdmanis         days)
(1987)
                  * Intensive care (inpatient
                  days)

                  * Surgeries (in-patient
                  and out-patient
                  surgeries)

                  * Ambulatory and
                  emergency care
                  (number of visits)

Bowlin (1987)     * Completed work orders

                  * Completed job orders

                  * Completed recurring
                  work actions

                  * Delinquent job orders

Table 2
Project Selection Data (Modified from data set in McCain (2011)

Project   ROI (%)    Alignment    Implementation     Cost
                    to Strategy       Time (Hrs)   ('000)

1               20             4             6000    2000
2               15             5             8000    1800
3               30             4             6500    1500
4               20             3             5000    2200
5               25             3             6500    1000
6               10             5             4000     900
7               35             3             7000    3000
8               10             4             6000    1700
9               40             4            10000    3500
10              30             3             5500    1200

Table 3
Project Efficiency Ratios

Project Number  Efficiency Ratios   Reference Set

1                           0.792   Projects 6,10
2                           0.581   Projects 6,10
3                           0.929   Projects 6,10
4                           0.842   Projects 6,10
5                           1.000   --
6                           1.000   --
7                           0.917   Project 10
8                           0.576   Projects 6,10
9                           0.733   Project 10
10                          1.000   --