Robustness of the multi objective MOORA method with a test for the facilities sector/ Daugiatikslio MOORA metodo patikimumo tikrinimas paslaugu sektoriuje.

Brauers, Willem K. ; Zavadskas, Edmundas K.

1. Definition of robustness

By 1953, which is quite recent for statistics (1), robust became a statistical term as "strong, healthy, sufficiently tough to withstand life's adversities" (Stigler 1973:872). Nevertheless, already in 1969 the statistician Huber considered robustness as purely cardinal as a compromise between a normal distribution and its light deviations (2). More recently the statisticians Casella and Berger call a robust alternative the Median Absolute Deviation for a sample [X.sub.1], ... [X.sub.n] (2002:509).

The error term in a linear equation is the starting point for the definition of robustness in econometrics (Darnell 1997: 355). In addition, robustness is not only linked to error terms or random variables but also to residual terms, slack and dummy variables, outliers etc. Darnell concludes: "given the somewhat arbitrary ad hoc nature of the robust estimators ... ... these approaches have had limited application in econometrics" (1997: 356). Kennedy recognizes the existence of robust estimators "an estimator whose properties while not quite best", he continues "the topic of robustness has become quite popular recently in econometrics, as researchers have become aware of the extreme sensitivity of some of their estimation procedures"(1998: 298). Other well-known textbooks on econometrics do not even mention the name of robustness, like Thomas (1985), Intriligator (1978), Madansky (1976), Walters (1973), Wonnacott, R., Wonnacott, T. (1970) and Johnston (1963). More specificity is found by authors who consider robustness in forms of the error distribution (Rhodes and Fomby 1988), whereas Mills (1992) presents a Bayesian prediction test which is robust to certain forms of non-normality in the error distribution. Moreover, from the beginning Bayesian Analysis has to be characterized as cardinal, nevertheless with a high grade of arbitrariness. This arbitrariness could be softened by considerations on robustness (3). Anyway, cardinal numbers form also the basis of robustness in the Poisson distribution, the t statistic and in sampling (Sarndal, Swensson and Wretman 1992).

However, even in econometrics, we observe a move to a more vague and qualitative definition of robustness, namely to the meaning of common language (4): from a cardinal towards a nominal scale: the most robust one, more robust than ..., as robust as ... ..., robust, weak robust, less robust than ..., not robust etc., comparable to so many other nominal scales in multi-objective analysis (for instance, mentioned by Brauers 2004: 97-99).

A debate between Frisch (1933) and Tinbergen (1930) ensued as whether or not Tinbergen had estimated structural form representations robust to changes in policy regimes or reduced form representations not robust to shifting policy regimes (Heckman 1992: 878). Kreps (1990) maintains that more robustness is more important for bargaining theory than for auction theory as more information is available in the latter case than in the former. He esteems that robust predictions are crucial although the meaning given to robustness may depend on the context. (also Vincke 1999: 186(2)). Edin and Ohlson (1991) examine that institutional arrangements in the political process affect budget deficits. Sensitivity Analysis indicates that the results are robust. Admati and Pfleiderer (1994) speak of robustness in financial contracting.

Another qualitative approach of robustness is related to subjective probability by Machina and Schmeidler (1992). Dasgupta and Maskin (2008) maintain that the simple majority rule is the most robust voting rule. Finally, the context will determine robustness in benchmarking, in scenario writing and in simulation (Brauers et al. 2009).

The remark that significance of robustness depends on the context can be specified in different ways. First, robustness can be considered as cardinal or as a nominal scale. Second, if robustness is indicated as vague or arbitrary, is it also not the case with inference statistics (Hoel 1971, 2 versus Hays, 1974: 47 and Casella et al. 2002, VII), probability theory (Hays 1974: 47) and statistical specification (Intriligator 1978: 2 and Matyas, Sevestre 1992, chapter 9 versus Thomas 1985: 71 and Wonnacott 1970: 312)i Third, robustness is characterized by completeness being present in the statistical population, when defined as covering events and opinions which are present, as well as in the statistical universe with events and opinions not only present but also possible (Brauers et al. 2009).

2. Conditions of robustness in multi-objective methods

The most robust multi-objective method has to satisfy the following conditions:

2.1. All stakeholders

The method of multiple objectives in which all stakeholders are included is more robust than this one in which only one decision-maker or different decision-makers defending only their own objectives are involved. All stakeholders mean everybody interested in a certain issue. Consequently, the method of multiple objectives has to take into consideration consumer sovereignty too. The method taking into consideration consumer sovereignty is more robust than this one which does not respect consumer sovereignty.

2.2. All objectives

The method of multiple objectives in which all non-correlated objectives are considered is more robust than this one in which only a limited number of objectives is considered.

2.3. All interrelations between objectives and alternatives

The method of multiple objectives in which all interrelations between objectives and alternatives is taken into consideration at the same time is more robust than this one in which the interrelations are only examined two by two.

2.4. Non-subjective

The method of multiple objectives which is non-subjective is more robust than this one which uses subjective approaches. Is this condition not purely theoreticali The difficulty lies in the many facets of multi-objective optimization in which excluding subjectivity seems to be impossible. Indeed, it concerns the subjectivity in the choice of the objectives, in the normalization procedure and in the importance given to an objective.

a. Non-subjectivity in the choice of the objectives

A creative and prospective thinking of all stakeholders interested in a certain issue has to generate a complete set of objectives. The Ameliorated Nominal Group Technique can assist for that purpose (see Appendix A). Non-subjectivity of the retained objectives results from unanimity or at least a convergence in the opinions of all the stakeholders concerned. Not only the Ameliorated Nominal Group Technique but in addition the Delphi Technique can bring support here (see Appendix B). Delphi will also assist in giving importance to the objectives, as explained underneath.

b. Non-subjectivity in the normalization procedure

Normalization affords a subjective solution for comparing the different units of the different objectives. Consequently, the method of multiple objectives which uses non-subjective dimensionless measures, meaning that normalization is not needed, like in the ratio systems approach, is more robust than this one which uses subjective weights (already introduced by Churchman et al. in 1954, 1957) or subjective non-additive scores like in the traditional Reference Point Theory (Brauers 2004: 158-159).

c. Non-subjectivity in the attribution of importance to an objective

Convergence in opinion between all stakeholders to give more importance to an objective results again from a Delphi exercise. Therefore its dimensionless numbers are multiplied by a Significance Coefficient. The Attribution of Sub-Objectives represents another solution. The Attribution Method is more refined than the Coefficient Method, as the attribution method succeeds in characterizing an objective better. For instance, instead of giving a significance coefficient of 3 to pollution abatement, the objective "pollution abatement" is divided into 3 sub-objectives: the Greenhouse Effect, Energy Consumption and Other Pollution, each with their own characteristics.

2.5. Cardinal

The method of multiple objectives based on cardinal numbers is more robust than this one based on ordinal numbers: "an ordinal number is one that indicates order or position in a series, like first, second, etc." (Kendall, Gibbons 1990: 1).

The robustness of cardinality is based first on the saying of Arrow (1974): "Obviously, a cardinal utility implies an ordinal preference but not vice versa" and, second, on the fact that the four essential operations of arithmetic: adding, subtracting, multiplication and division are only reserved for cardinal numbers (5).

2.6. Last available data

The method of multiple objectives which uses the last recent available data as a base in the response matrix is more robust than this one based on earlier data.

2.7. Different methods

Once the previous 6 conditions are fulfilled, the use of 2 different methods of multi-objective optimization is more robust than the use of a single method; the use of 3 methods is more robust than the use of 2, etc.

The Multi-Objective Optimization by Ratio Analysis Method (MOORA) satisfies the first 6 conditions if Non-Subjectivity in the choice of the objectives and Non-Subjectivity in the attribution of importance to an objective are solved. Therefore the Ameliorated Nominal Group Technique but also the Delphi Technique can bring support. In addition, MOORA satisfies partially the seventh condition by using 2 different methods of Multi-Objective Optimization. MOORA is the most robust method as no other method satisfies the 7 conditions better.

3. The MOORA method

The method starts with a matrix of responses of all alternative solutions on all objectives:

[X.sub.ij], (1)

with: [x.sub.ij] as the response of alternative j on objective or attribute i; i = 1,2, ... , n as the objective or the attribute; j = 1,2, ..., m as the alternatives.

In order to define objectives better, we have to focus on the notion of attribute. Keeney and Raiffa (1993: 32) present the example of the objective "reduce sulfur dioxide emissions" to be measured by the attribute "tons of sulfur dioxide emitted per year". An objective and a correspondent attribute always go together. Consequently, when the text mentions "objective", the correspondent attribute is meant as well.

The MOORA method consists of 2 parts: the ratio system and the reference point approach.

3.1. The ratio system as a part of MOORA

We go for a ratio system in which each response of an alternative on an objective is compared to a denominator, which is representative for all alternatives concerning that objective (6):

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII.] (2)

with: [x.sub.ij] = response of alternative j on objective i; j = 1,2, ... , m; m the number of alternatives; i = 1, 2, ... n; n the number of objectives; [x.sub.ij.sup.*] a dimensionless number representing the response of alternative j on objective i.

Dimensionless Numbers, having no specific unit of measurement, are obtained for instance by multiplication or division. The normalized responses of the alternatives on the objectives belong to the interval [0; 1]. However, sometimes the interval could be [-1; 1]. Indeed, for instance, in the case of productivity growth some sectors, regions or countries may show a decrease instead of an increase in productivity, i.e. a negative dimensionless number (7).

For optimization, these responses are added in case of maximization and subtracted in case of minimization:

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII.] (3)

with: i = 1, 2, ..., g as the objectives to be maximized; i = g + 1, g + 2, ..., n as the objectives to be minimized; [y.sub.j.sup.*] a dimensionless number representing the response of alternative j with respect to all objectives; [y.sub.j.sup.*] can be positive or negative depending of the totals of its maxima and minima.

An ordinal ranking of the [y.sub.ij.sup.*] shows the final preference. Indeed, cardinal scales can be compared in an ordinal ranking after Arrow (1974: 256): "Obviously, a cardinal utility implies an ordinal preference but not vice versa".

3.2. The reference point approach as a part of MOORA

Reference Point Theory will go out from the ratios found in formula (2), whereas a Maximal Objective Reference Point is also considered. The Maximal Objective Reference Point approach is called realistic and non-subjective as the co-ordinates ([r.sub.i]), which are selected for the reference point, are realized in one of the candidate alternatives. In the example, A (10;100), B (100;20) and C (50;50), the maximal objective reference point Rm results in: (100;100). The Maximal Objective Vector is self-evident, if the alternatives are well defined, as for projects in Project Analysis and Project Planning.

Given the dimensionless number representing the normalized response of alternative j on objective i, namely [x.sub.ij.sup.*] of formula (2) and in this way arriving to:

([r.sub.i] - [x.sub.ij.sup.*]), (4)

with: i = 1,2, ..., n as the attributes; j = 1,2, ..., m as the alternatives; [r.sub.i] = the ith co-ordinate of the reference point; [x.sub.ij.sup.*] a dimensionless number representing the response of alternative j on objective i, then this matrix is subject to the Min-Max Metric of Tchebycheff (Karlin and Studden 1966) (8):

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII.] (4)

[absolute value of [r.sub.i] - [x.sub.ij.sup.*]] means the absolute value if [x.sub.ij.sup.*] is larger than [r.sub.i], for instance, by minimization.

Concerning the use of the maximal objective reference point approach as a part of MOORA, some reserves can be made in connection with consumer sovereignty. Consumer sovereignty is measured with the community indifference locus map of the consumers (Brauers 2008b: 92-94). Given its definition, the maximal objective reference point can be pushed in the non-allowed non-convex zone of the highest community indifference locus and will try to pull the highest ranked alternatives in the non-allowed non-convex zone too (Brauers, Zavadskas 2006: 460-461). Therefore an aspiration objective vector can be preferred, which moderates the aspirations by choosing smaller co-ordinates than in the maximal objective vector and consequently can be situated in the convex zone of the highest community indifference locus. Indeed stakeholders may be more moderate in their expectations. The co-ordinates qi of an aspiration objective vector are formed as:

[q.sub.i] [less than or equal to] [r.sub.i]

([r.sub.i] - [q.sub.i]) being a subjective element we do not like to introduce subjectivity in that way again. Instead, a test shows that the min-max metric of Tchebycheff, even by using the maximal objective reference point, delivers points inside the convex zone of the highest community indifference locus (Brauers 2008b: 98-103).

3.3. The importance given to an objective

The normalized responses of the alternatives on the objectives belong to the interval [0; 1] (see formula 2). Nevertheless, it may turn out to be necessary to stress that some objectives are more important than others. In order to give more importance to an objective its normalized responses on an alternative could be multiplied with a Significance Coefficient:

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII.] (6)

with: i = 1, 2, ..., g as the objectives to be maximized; i = g+1, g+2, ..., n as the objectives to be minimized; [s.sub.i] = the significance coefficient of objective i; [[??].sub.j.sup.*] a dimensionless number representing the response of alternative j with respect to all objectives with significance coefficients.

The Attribution of Sub-Objectives represents another solution. Take the example of the purchase of fighter planes (Brauers 2002). For economics, the objectives concerning the fighter planes are threefold: price, employment and balance of payments, but there is also military effectiveness. In order to give more importance to military defense, effectiveness is broken down in, for instance, the maximum speed, the power of the engines and the maximum range of the plane. Anyway, the Attribution Method is more refined than that a significance coefficient method could be as the attribution method succeeds in characterizing an objective better. For instance, for employment two sub-objectives replace a significance coefficient of two and in this way characterize the direct and indirect side of employment.

Of course, at that moment the problem raises of the subjective choice of objectives in general, or could we call it robustness of a choicei The Ameliorated Nominal Group Technique will gather all stakeholders interested in the issue to determine the objectives in a non-subjective and anonymous way (see Appendix A).

The importance given to an objective results from convergence in the stakeholders' opinions, which will happen with the assistance in a robust way of the Delphi Technique (see Appendix B).

4. Application of the MOORA method for evaluating contractor's alternatives in the facilities sector in Lithuania

The facilities sector (Real Estate) in Lithuania provides the following services:

--acquisition, leasing and renting of existing buildings;

--management of buildings, which is a multifunctional service. This means that all supervision, maintenance and repairing is included in the sector.

The facilities sector is only a very small sector in Lithuania, composed of a small number of small firms, which even perform other tasks outside facilities management, such as waste management. The largest firm in the sector counts only 179 employees. Official statistics are not separately available for the facilities sector (Brauers, Lepkova 2003).

In theory the facilities sector could include the entire management of corporate real estate. This means the effective management, which is called the fifth resource. Indeed, in the report of "the Industrial Development Research Foundation of the United States", the corporate real estate assets are indicated as a fifth resource, after the resources of people, technology, information and capital (McGregor; Shiem-Shim Then 1999).

An application in facility management, namely on the choice of a contractor for the maintenance of private dwellings tests the definitions of robustness. The problem is multiobjective: like cost of building management, territory cleaning, experience, effectiveness, and size at the side of the contractors; quality of the construction, duration of the work and cost price at the side of the owners. Objectives are determined on basis of the opportunities of the contractors and of the wishes of the customers, here the owners of dwellings.

The 15 largest maintenance contractors for private dwellings of Vilnius, capital of Lithuania, agreed to fix and estimate their main objectives, namely 9 objectives, as given in Table 1. The full names of the contractors are not provided for the sake of confidentiality (9).

The management cost ([c.sub.min]/[c.sub.p]) evaluates the cost competitiveness of a contractor. [C.sub.min] represents the minimal cost in all offerings, [c.sub.p] is the cost offered by the contractor considered. Consequently:

[C.sub.min]/[C.sub.p] [less than or equal to] 1.

The nature of the construction industry involves that the total number of the minima is mostly larger than the total number of the maxima. Contractor's rating is performed according to the attributes (Table 2).

From information of the dwelling owners association, a panel of 30 owners of dwellings chosen at random agreed with these 9 objectives, but they increased the objectives with 11 other ones (10). However, these additional objectives were only expressed in qualitative points, showed some overlapping and after their rating represented only 25.9% importance of the total. If these opinions are only taken as indicative, these qualitative objectives can be dropped.

Is the owner's information perhaps useful to allot significance coefficientsi Therefore the sample is not enough representative. Indeed, a significance coefficient of importance was not possible to give to the 9 objectives as 30 interviews even chosen at random mean a confidence level of only:

Standard error = [square root of (pq/n)] = [square root of (25/30)] 0.09, which means 9% under or 9% above the real percentage.

Economics generally accept 100 interviews with a standard error of: [square root of (pq/n)] = [square root of (25/100)] = 0.05 which means 5% under or 5% above the real percentage (p = expected probability; q = 1 - p; in a symmetric distribution: p = q).

Instead of attributing significance coefficients the contractors and the small group of owners preferred the attribution of sub-objectives. Indeed, 5 objectives concern minimization of costs (efficiency). Even the last maximization forms a cost consideration. Furthermore, one objective is related to experience, as measured by length of time in maintenance business, one to size as measured by marked share and finally one objective measures effectiveness as expressed by the number of projects per executive.

Table 3 presents the results of the calculation process of MOORA. Appendix C gives the details of this calculation.

Both parts of MOORA rank more or less in the same way the first 4 positions after the quality of the contractors. In this way a double check is made on the robustness of the results, as shown in Table 4.

According to the results of table 4, we can find the priority between the contractors:

Contractor 6 P Contractor 10 P Contractor 1 P Contractor 4 (P preferred to)

In other words, 4 contractors are classified in a robust order, whereas contractor A15 is the very last one. The other 10 contractors are ranked low, but it is unclear in what position.

Contractor 6 is ranked first for size and experience and second for effectiveness. Contractor 10 is ranked first for effectiveness and second for size. Contractor 1 together with contractor 4 is second ranked for experience. All these strong contractors are not so good in efficiency (costs), which seems rather unusual. On the other side, the size of the enterprise seems to be very important. In this way the comments that from the beginning no small firms were considered are without any value.

5. Conclusions

The remark that significance of robustness depends on the context is specified in different ways. First, robustness can be considered either as cardinal robust or as a nominal scale. Second, if robustness is indicated as vague or arbitrary, perhaps it is also the case with inference statistics, probability theory and statistical specification. Third, robustness is characterized by completeness being present in the statistical population covering events and opinions which are present, as well as in the statistical universe with events and opinions not only present but also possible.

Concerning the most robust method of multi-objective optimization the following conditions are to be satisfied:

1) the method of multiple objectives in which all stakeholders are involved is more robust than one in which only one decision-maker or different decision-makers defending only a limited number of objectives are involved. All stakeholders mean everybody interested in a certain issue. Consequently, the method of multiple objectives which takes into consideration consumer sovereignty is more robust than this one which does not respect consumer sovereignty. Consumer sovereignty is measured with community indifference loci. Solutions have to deliver points inside the convex zone of the highest community indifference locus;

2) the method of multiple objectives in which all non-correlated objectives are considered is more robust than this one, in which only a limited number of objectives is considered;

3) the method of multiple objectives in which all interrelations between objectives and alternatives are taken into consideration, at the same time is more robust than this one in which the interrelations are only examined two by two;

4) the method of multiple objectives based on cardinal numbers is more robust than this one based on ordinal numbers.

The robustness of cardinality is based first on the saying of Arrow: "obviously, a cardinal utility implies an ordinal preference but not vice versa" and second on the fact that the four essential operations of arithmetic: adding, subtracting, multiplication and division are only reserved for cardinal numbers;

5) the method of multiple objectives which is non-subjective is more robust than this one using subjective estimations. Consequently, a method of multiple objectives which uses non-subjective dimensionless measures is more robust than this one which for normalization uses subjective weights or subjective non-additive scores like in the traditional Reference Point Theory.

A creative and prospective thinking of all stakeholders interested in a certain issue has to generate a complete set of objectives. The Ameliorated Nominal Group Technique can assist for that purpose (see Appendix A). Non-subjectivity of the retained objectives results from unanimity or at least a convergence in the opinions of all the stakeholders concerned. Not only the Ameliorated Nominal Group Technique but in addition the Delphi Technique can bring support here. Delphi will also assist in giving importance to the objectives (see Appendix B);

6) the method of multiple objectives which uses the last recent available data as a base in the response matrix is more robust than this one based on earlier data;

7) once the previous 6 conditions are fulfilled the use of 2 different methods of multiobjective optimization is more robust than the use of a single method; the use of 3 methods is more robust than the use of 2, etc.

The Multi-Objective Optimization by Ratio Analysis Method (MOORA) satisfies the 6 conditions with the assistance of the Ameliorated Nominal Group Technique and the Delphi Method. In addition, MOORA satisfies partially the seventh condition by using 2 different methods of Multi-Objective Optimization. MOORA is the most robust method as no other method satisfies the 7 conditions better.

The MOORA method, based on dimensionless measures, consists of 2 parts: the aggregation of dimensionless ratios and these ratios used as distances to a reference point. The 2 parts of MOORA control each other. Does the application of MOORA for the contractor's alternatives in the facilities sector of Lithuania respond to the seven conditions of robustnessi

1) First condition of robustness

All stakeholders, on the one side the contractors, on the other side the owners of private dwellings were involved. Indeed, MOORA was applied for the choice between the 15 main contractors of dwellings to satisfy the wishes of the owners of dwellings in the city of Vilnius, capital of Lithuania. Consequently, this condition also respects consumer sovereignty.

2) Second condition of robustness

All objectives were taken into consideration, though a part at the side of the owners was ignored as:

--the objectives were only qualitative

--there was overlapping with the contractor's objectives

--they were only a less significant part of the questioning.

3) Third condition of robustness

All interrelations between objectives and alternatives were involved at the same time under the form of a matrix of responses considered as a whole and as a starting point for the application of MOORA.

4) Fourth condition of robustness MOORA is based on cardinal numbers;

5) Fifth condition of robustness

The use of dimensionless measures is a more robust method than subjective methods of normalization. In the application MOORAs dimensionless ratios satisfied this condition.

6) Sixth condition of robustness The last available data were used.

7) Seventh condition of robustness

All the previous 6 conditions are fulfilled and also the 7th condition as 2 different methods of Multi-Objective Optimization are used. No other Multi-Objective Optimization Method exists which uses more than 2 Multi-Objective Optimization Methods and fulfill the previous six conditions.

The MOORA method came to the following results: 3 contractors take the first 3 positions. A fourth one has still to be mentioned with its favourable ranking position, whereas one contractor is classified the very last one. The other 10 contractors are ranked low, but it is unclear in what position. The best contractors are not the best in efficiency (costs), which seems rather unusual. On the other side, the size of the enterprise seems to be very important. In this way the comments that from the beginning no small firms were considered are without any value.

Even more, for contractors and their clients the firm quality as measured by size, experience and effectiveness seems to dominate the cost price of maintenance for a dwelling. A new research based on newer data, larger samples and a larger number of quantifiable objectives may verify this unexpected outcome and will rather increase the robustness of the outcome.

Is the robustness of the conclusion acceptable: quality chosen before cost pricei Probably after the well-known bad quality of the Soviet time, Lithuania was an integral part of the Soviet Union until 1990, the owners of dwellings prefer good quality for an acceptable price. Otherwise, it is possible that robustness of the research was rather weak. Indeed, the research dates from 2004. Data from 2008 would have a better robustness and the period 2004-2008 is even still better. Given that the robustness of the MOORA method itself was effectively proven, a new research based on the latest data, larger samples and a larger number of quantifiable objectives may increase the application robustness.

doi: 10.3846/1392-8619.2009.15.352-375

Appendix A

The assistance by the ameliorated nominal group technique

With experts representing all stakeholders for a certain issue the relation with robustness seems rather fuzzy, unless an Ameliorated Nominal Group Technique is used. Even better if with the produced information one can deduct complete scenario writing.

The ameliorated approach of the nominal group technique, which is explained here, was ameliorated by Brauers (1987, 2004: 44-64), but the Nominal Group Technique was first elaborated by Van de Ven and Delbecq (1971).

1. The Original Nominal Group Technique

The nominal group technique consists of a sequence of steps, each of which has been designed to achieve a specific purpose.

1) The steering group or the panel leader carefully phrases as a question the problem to be researched. Much of the success of the technique hinges around a well-phrased question. Otherwise the exercise can easily yield a collection of truisms and obvious statements. A successful question is quite specific and refers to real problems. The question has to have a singular meaning and a quantitative form as much as possible.

2) The steering group or the panel leader explains the technique to the audience. This group of participants is asked to generate and write down ideas about the problem under examination. These ideas too have to have a singular meaning and a quantitative form as much as possible. Participants do not discuss their ideas with each other at this stage. This stage lasts between five and twenty minutes.

3) Each person in round-robin fashion produces one idea from his own list and eventually gives further details. Other rounds are organized until all ideas are recorded.

4) The steering group or the panel leader will discuss with the participants the overlapping of the ideas and the final wording of the ideas.

5) The nominal voting consists of the selection of priorities, rating by each participant separately, while the outcome is the totality of the individual votes. A usual procedure consists of the choice by each participant of the n best ideas from his point of view, with the best idea receiving n points and the lowest one point. All the points of the group are added up. A ranking is the democratic result for the whole group.

The Original Nominal Group Technique can be characterized as weak robust as the participants expressed too much their personal feeling. For that reason amelioration was proposed.

2. The Ameliorated Nominal Group Technique

As there was too much wishful thinking even between experts, better results were obtained if the group was also questioned about the probability of occurrence of the event. In this way the experts became more critical even about their own ideas. The probability of the group is found as the median of the individual probabilities.

Finally, the group rating (R) is multiplied with the group probability (P) in order to obtain the effectiveness rate of the event (E):

E = R x P.

Once again, the effectiveness rates of the group are ordered by ranking. One may conclude that the Ameliorated Nominal Group Technique is more robust than the Original Nominal Group Technique.

3. Examples of the Ameliorated Nominal Group Technique

1) Which events will influence the most the economic aspect of the developing countries in the next 10 years to come (1987-1996)? 23 post-graduate students, non-experts, from French speaking Africa participated (Brauers 2004: 49-51).

Robustness of this examplei

An Evaluation ex-post showed an astonishing approximation of realization.

2) Which events (maximum 5 per participant) will influence the most the economic aspect of Lithuania in the next 10 years to come (2002-2011)? 18 post-graduate students, non-experts, from Lithuania participated (Brauers, Lepkova 2002).

Robustness of this examplei

A partial Evaluation ex-post showed a sufficient result.

3) Which events (economic, technical, political, social, medical and other events) will influence the most the business outlook of the Facilities Sector (Corporate Real Estate) of Lithuania in the next 10 years to come (2003-2012)i 15 experts as Lithuanian delegates from the facilities sector, the ministerial departments involved and from the academic world participated. In this way an attempt was made to involve all stakeholders (Brauers, Lepkova 2003).

22 events were recorded. Each participant could select 5 events with as points: 5 + 4 + 3 + 2 + 1 = 15. With 15 participants assisting the total arrived at 225 and, as all participants used their privileges, the total of 225 was maintained. The usefulness of the introduction of median probabilities was demonstrated by the fact that the total of 225 was reduced to 145.21.

Robustness of this example?

A partial evaluation ex-post showed a sufficient result.

The results of this nominal group technique inspired the panel leader to deduce 3 different scenarios for the future of the facilities sector in Lithuania: the fifth resource scenario, the status quo scenario and the cut throat competitive scenario (Brauers, Lepkova 2003: 7).

The fifth resource scenario

Foreign direct investments will come to Lithuania. They may find industrial zones with ready premises. However, they want more and they like to decide themselves on location. This is the moment that the fifth resource beside the resources of people, technology, information and capital, comes fully alive, namely the management of the corporate real estate assets (McGregor et al. 1999). New companies will respond to this demand. These new companies of real estate assets management will look after space (location), design of buildings, construction, reparations, maintenance, waste management and eventually demolition. They will look after the direct investments, so to say "from the cradle until the grave". This has to happen in the most effective way, i.e., with an optimal multiple objective utility. In this way, the foreign firms can look for an optimal multiple objective utility for people, technology, information and capital, but also for their corporate real assets. Synergy effects will play fully.

The status quo scenario

In this scenario the situation in the facilities sector does not change. A set of small firms will remain operative in the facilities sector of Lithuania. Some small firms will disappear by competition, aging, disagreement, etc., but new ones will take their places.

The cut-throat competitive scenario

In the cut-throat competitive scenario the consequences of the productivity effect of the European Union will fully play. The productivity in the new member countries of the European Union will rise in the internationally traded sectors. The result is an increase in wages. However, after the "Balassa-Samuelson Effect" the more national services have to raise their wages too, without an increase in productivity of the same size. This increase in wages will have an inflationary effect in the country (Balassa 1964; Balazs et al. 2002; Samuelson

1994, 1964).

The facilities sector in Lithuania will fully undergo this influence. It will remain only a nationally traded sector. If it increases its prices, together with the other not internationally traded services, inflation will go up in Lithuania and ipso facto the cost of living will rise.

Instead of increase in prices, diminution of quality of its services forms another alternative for the facilities sector. At that moment cut-throat competition between the facilities management companies will occur. In this struggle for life only the fittest will survive. It is also the moment that mala fide companies will appear, which will exploit the customers as much as possible.

Robustness of these scenarios?

Robustness of these scenarios is rather weak; the set of scenarios is not complete. Perhaps still many other scenarios are possible.

Appendix B

The assistance by the Delphi technique

In questionnaires, referenda and Delphi the face-to-face dialogue is absent in order to decrease subjective influence. Questionnaires fail if broad issues are involved. Indeed the steering group may influence the opinions by its phrasing of the questions. In addition, the steering group may make a too subjective summary when analyzing the questionnaires. With questionnaires, it is difficult to reach consensus. Opinions can be too divergent for a consensus to be reached, which is certainly the case with broad problems. Could the mean average be useful for this purposei Mueller et al. (1970: 140) remark that: "since it reflects every value in the array, it will be affected by the extremely high or low values that are always found in a skewed distribution and therefore it will lose its typicality and perhaps mislead the reader".

At that moment manipulation is possible. Suppose, for instance, in a jury a jury member can influence the voting by giving very high points to his protege (skewed to the right) and very low points to the other candidates (skewed to the left). These excesses can be left out too, but in a subjective way. Who can judge what is excessive and what noti Here the median is helpful. The median is defined as the middle measurement after the measurements have been arranged in order of magnitude. To measure skewness beside the median one could find two other values, one dividing the histogram at a point such that one fourth of the area to the left of it (first quartile) and the other such that one fourth of the area is to the right of it (third quartile). In fact the median itself is the second quartile. In this way one may speak of skewness to the left and skewness to the right. A task could be to find ways and means to decrease skewness by trying to bring the quartiles nearer to each other. This point is taken up later.

Referenda, as a direct vote of the population, may fail too, if broad issues are involved. Not only they will show the same disadvantages as the questionnaires, but they can also create opposition from a large part (eventually until 49.99%) of the population against the outcome. All depends if the referendum is simply indicative or really directive, like in direct democracy. In indirect democracy, democracy through representatives, majority voting would be the most robust voting rule (Dasgupta and Maskin, 2008).

Delphi tries to improve either the committee or the questionnaire approach. The Delphi method is a method for obtaining and processing judgmental data. It consists of a sequenced program of interrogation (in session or by mail) interspersed with feedback of persons interested in the issue, while everything is conducted through a steering group. We advocate most this method as it also takes care of:

--Quantitative treatment

--Expert knowledge

--Anonymity

--Convergence.

Dalkey and Helmer (1963) used Delphi in its present form for the first time around 1953. The essential features of it are:

1. A group of especially knowledgeable individuals (experts)

2. Inputs with a singular meaning and quantitative as much as possible

3. The opinions about the inputs are evaluated with statistical indexes

4. Feedback of the statistical indexes with request for re-estimation, also after consideration of reasons for extreme positions

5. The sources of each input are treated anonymously

6. Two developments: meeting and questionnaires. The organization of a meeting produces quicker results. However, the meeting has to be organized in such a way that communication between the panel members is impossible. Therefore, a central computer with desk terminals, television screen and computer controlled feedback is advisable.

As an example of Delphi, a music competition ended with 12 finalists (Brauers 2008a). Beside the personal preferences of the jury members, different music schools or tendencies exist. Total points and the medians were the same for the first four candidates but for the 5th and the 6th ranks, the laureates were reversed. However, the large diversion between the first and the third quartiles illustrated a possible frustration between the jury members for the laureates ranking 5 and 6 and the other finalists ranking 7, 8, 9, 10, 11 and 12. At that moment

Delphi interferes. The voting is repeated several times. In the beginning skewness is still too large, but then a new round may help. Delphi experiences a better convergence in opinion as the medians and quartiles approach more and more to one another in different rounds until convergence as much as possible is reached and automatically robustness is increased. At that moment, the ranking of the finalists in the positions 5 till 12 may be entirely reversed, but the members of the jury, like the public and the press, will be more satisfied.

Appendix C

MOORA: square root method (1a until 1c) and MOORA reference point theory (1d-1e)

Table 1. MOORA applied to 9 objectives for Lithuanian contractors
1a. Matrix of responses of alternatives on objectives: ([x.sub.ij])

 1 2 3 4

 min. min. min. min.

[A.sub.1] 0.064 0.11 0.18 0.31
[A.sub.2] 0.06 0.14 0.37 0.12
[A.sub.3] 0.057 0.11 0.18 0.15
[A.sub.4] 0.06 0.12 0.09 0.15
[A.sub.5] 0.058 0.1 0.18 0.2
[A.sub.6] 0.071 0.3 0.18 0.26
[A.sub.7] 0.11 0.14 0.18 0.12
[A.sub.8] 0.058 0.18 0.37 0.19
[A.sub.9] 0.053 0.14 0.16 0.23
[A.sub.10] 0.07 0.26 0.29 0.23
[A.sub.11] 0.12 0.2 0.09 0.2
[A.sub.12] 0.071 0.28 0.18 0.28
[A.sub.13] 0.078 0.2 0.18 0.3
[A.sub.14] 0.056 0.14 0.18 0.12
[A.sub.15] 0.12 0.14 0.09 0.21

 5 6 7 8 9

 min. max. max. max. max.

[A.sub.1] 0.67 12 11.75 4.6 0.83
[A.sub.2] 0.5 3 0.39 0.33 0.885
[A.sub.3] 0.69 12 5.25 1.47 0.935
[A.sub.4] 0.57 12 7.09 2.78 0.912
[A.sub.5] 0.45 12 5.56 1.39 0.912
[A.sub.6] 0.82 13 26.62 5.67 0.746
[A.sub.7] 0.55 5 2.82 1.2 0.483
[A.sub.8] 0.61 11 9.48 3.03 0.916
[A.sub.9] 0.8 11 2.23 0.76 1
[A.sub.10] 0.73 11 13.47 9.05 0.746
[A.sub.11] 0.81 4 4.7 1.5 0.443
[A.sub.12] 0.73 12 2.35 0.86 0.746
[A.sub.13] 0.76 8 5.6 3.25 0.681
[A.sub.14] 0.5 11 2.66 1.7 0.948
[A.sub.15] 0.56 3 0.04 0.03 0.531

1b. Sum of squares an d their square roots

[A.sub.1] 0.004096 0.0121 0.0324 0.0961
[A.sub.2] 0.0036 0.0196 0.1369 0.0144
[A.sub.3] 0.003249 0.0121 0.0324 0.0225
[A.sub.4] 0.0036 0.0144 0.01 0.0225
[A.sub.5] 0.003364 0.01 0.0324 0.04
[A.sub.6] 0.005041 0.09 0.0324 0.0676
[A.sub.7] 0.0121 0.0196 0.0324 0.0144
[A.sub.8] 0.003364 0.0324 0.1369 0.0361
[A.sub.9] 0.002809 0.0196 0.0256 0.0529
[A.sub.10] 0.0049 0.0676 0.0841 0.04
[A.sub.11] 0.0144 0.04 0.0081 0.04
[A.sub.12] 0.005041 0.0784 0.0324 0.0784
[A.sub.13] 0.006084 0.04 0.0324 0.09
[A.sub.14] 0.003136 0.0196 0.0324 0.0144
[A.sub.15] 0.0144 0.0196 0.0081 0.0441

sum 0.089184 0.495000 0.66890 0.6734

root 0.2986369 0.70356236 0.81786307 0.82060953

[A.sub.1] 0.4489 144 138.0625
[A.sub.2] 0.25 9 0.1521
[A.sub.3] 0.4761 144 27.5625
[A.sub.4] 0.3249 144 50.41
[A.sub.5] 0.2025 144 30.9136
[A.sub.6] 0.6724 169 708.6244
[A.sub.7] 0.3025 25 7.9524
[A.sub.8] 0.3721 121 89.8704
[A.sub.9] 0.64 121 4.9729
[A.sub.10] 0.49 121 182.25
[A.sub.11] 0.6561 16 22.09
[A.sub.12] 0.5329 144 5.5225
[A.sub.13] 0.5776 64 31.36
[A.sub.14] 0.25 121 7.0756
[A.sub.15] 0.3136 9 0.0016

sum 6.5096 1496 1306.8205

root 2.55139178 38.6781592 36.1499723

[A.sub.1] 21.16 0.6889
[A.sub.2] 0.1089 0.783225
[A.sub.3] 2.1609 0.874225
[A.sub.4] 7.7284 0.81
[A.sub.5] 1.9321 0.81
[A.sub.6] 32.1489 0.556516
[A.sub.7] 1.44 0.233289
[A.sub.8] 9.1809 0.839056
[A.sub.9] 0.64 1
[A.sub.10] 81.9025 0.5625
[A.sub.11] 2.25 0.196249
[A.sub.12] 0.7396 0.556516
[A.sub.13] 10.5625 0.463761
[A.sub.14] 2.89 0.898704
[A.sub.15] 0.0009 0.281961

sum 174.8456 9.554902

root 13.2229195 3.09110045

1c. Objectives divided by their square roots and MOORA (1)

 1 2 3 4

[A.sub.1] 0.21430707 0.15634719 0.22008574 0.37776797
[A.sub.2] 0.20091288 0.19898734 0.45239847 0.14623276
[A.sub.3] 0.19086724 0.15634719 0.22008574 0.18279096
[A.sub.4] 0.20091288 0.17056057 0.12226986 0.18279096
[A.sub.5] 0.19421578 0.14213381 0.22008574 0.24372127
[A.sub.6] 0.23774691 0.42640143 0.22008574 0.31683766
[A.sub.7] 0.36834028 0.19898734 0.22008574 0.14623276
[A.sub.8] 0.19421578 0.25584086 0.45239847 0.23153521
[A.sub.9] 0.17747304 0.19898734 0.19563177 0.28027947
[A.sub.10] 0.23439836 0.36954791 0.35458258 0.24372127
[A.sub.11] 0.40182576 0.28426762 0.1100429 0.2437213
[A.sub.12] 0.23774691 0.39797467 0.2200857 0.3412098
[A.sub.13] 0.26118674 0.28426762 0.2200857 0.3655819
[A.sub.14] 0.18751869 0.19898734 0.2200857 0.1462328
[A.sub.15] 0.40182576 0.19898734 0.1100429 0.2559073

 5 6 7 8

[A.sub.1] 0.26260177 0.31025261 0.325 0.348
[A.sub.2] 0.19597147 0.07756315 0.01078839 0.02495667
[A.sub.3] 0.27044063 0.31025261 0.14522833 0.11117061
[A.sub.4] 0.22340748 0.31025261 0.19640402 0.21024101
[A.sub.5] 0.17637432 0.31025261 0.15380371 0.10512051
[A.sub.6] 0.32139321 0.336107 0.73637677 0.42880092
[A.sub.7] 0.21556862 0.12927192 0.07800836 0.09075152
[A.sub.8] 0.23908519 0.28439823 0.26224086 0.22914758
[A.sub.9] 0.31355435 0.28439823 0.06168746 0.06050101
[A.sub.10] 0.27436006 0.28439823 0.37344427 0.68441769
[A.sub.11] 0.3174738 0.1034175 0.1300139 0.1134394
[A.sub.12] 0.2861183 0.3102526 0.0650070 0.0650386
[A.sub.13] 0.2978766 0.2068351 0.1549102 0.2457854
[A.sub.14] 0.1959715 0.2843982 0.0735824 0.1285646
[A.sub.15] 0.2194880 0.0775632 0.0011065 0.0022688

 9 Sum (2) 1.9335294
 (3) Rank

[A.sub.1] 0.269 0.0205713 1.9541007 4
[A.sub.2] 0.2863058 -0.7948889 1.1386405 12
[A.sub.3] 0.30248127 -0.1513989 1.7821305 6
[A.sub.4] 0.29115844 0.1081144 2.0416438 3
[A.sub.5] 0.29115844 -0.1161957 1.8173337 5
[A.sub.6] 0.241338 0.2201577 2.1536871 1
[A.sub.7] 0.15625503 -0.6949279 1.2386015 11
[A.sub.8] 0.29633459 -0.3009542 1.6325752 8
[A.sub.9] 0.32350938 -0.4358299 1.4976995 9
[A.sub.10] 0.24263204 0.1082820 2.0418114 2
[A.sub.11] 0.1433147 -0.8671458 1.0663836 14
[A.sub.12] 0.2413380 -0.8014993 1.1320301 13
[A.sub.13] 0.2203099 -0.6011581 1.3323713 10
[A.sub.14] 0.3066869 -0.1555639 1.7779655 7
[A.sub.15] 0.1717835 -0.9335294 1.0000000 15

(1) The nature of the construction industry involves that the total
number of the minima is mostly larger than the total number of the
maxima with very often negative sums as a result, which is the case
here. Therefore, to make the ranking more comprehensive, the
supplement to make the smallest total, here a15, equal to one, is
added to every total.

(2) Due to the many minima the majority of the alternatives shows a
negative total. On the contrary, alternatives [A.sub.6],
[A.sub.10], [A.sub.4] and [A.sub.1] finally classified in good
order, show positive totals.

(3) Supplement to make all totals positive and the total for
[A.sub.15] equal to one.

Reference to this paper should be made as follows: Brauers, W. K.; Zavadskas, E. K. 2009. Robustness of the multi-objective MOORA method with a test for the facilities sector, Technological and Economic Development of Economy 15(2): 352-375.

Received 21 January 2009; accepted 4 May 2009

References

Admati , A. R.; Pfleiderer, P. 1994. Robust financial contracting and the role ofventure capitalists, Journal of Finance 49(2): 371-402.

Arrow, K. J. 1974. General economic equilibrium: purpose, analytic techniques, collective choice, American Economic Review, June: 253-272.

Balassa, B. 1964. The purchasing power parity doctrine: a reappraisal, Journal of Political Economy, 584-596.

Balazs, E.; Drine, I.; Lommatzsch, K.; Rault, C. 2002. The Balassa-Samuelson Effect and Central and Eastern Europe: Myth or Realityi Working Paper 483, the William Davidson Institute, University of Michigan Business School.

Brauers, W. K. 2008a. Group decision making with multi-objective optimization, Foundations of Computing and Decision Science 33(2): 167-179. Poznan University of Technology.

Brauers, W. K. 2008b. Multi-objective decision making by reference point theory for a wellbeing economy, Operations Research International Journal 8: 89-104.

Brauers, W. K. 2007.What is meant by normalization in decision makingi In W K. Brauers, E. K. Zavadskas (Ed.). Special Issue: Normalization in Decision-Making Methods, International Journal of Management and Decision Making 8(5/6): 441-460. Geneve, Inderscience Publishers.

Brauers, W. K. 2004. Optimization Methods for a Stakeholder Society, a Revolution in Economic Thinking by Multi-Objective Optimization: Nonconvex Optimization and its Applications. Boston, U.S.A.: Kluwer Academic Publishers.

Brauers, W. K. 2002. The multiplicative representation for multiple objective optimization with an application for arms procurement, Naval Research Logistics 49: 327-340. Wiley.

Brauers, W. K. 1987. Nominal Methods in Group Multiple Decision Making. Research Paper No 3, Institute For Developing Countries, University of Antwerp, Antwerpen, Be, Ruca.

Brauers, W. K.; Zavadskas, E. K. 2009. Is Robustness Really Robusti Robustness from the Point of View of Statistics and Econometrics, in C. Zopounides (Ed.). Multi Criteria Decision Aid, Series "Advances in Operations Research", Nova Publishers, Hauppauge, New York (in press).

Brauers, W. K.; Zavadskas, E. K. 2006. The MOORA Method and its application to privatization in a transition economy, Control and Cybernetics 35(2): 443-468.

Brauers, W. K.; Lepkova, N. 2003.The application of the nominal group technique to the business outlook of the facilities sector of Lithuania over the period 2003-2012, International Journal of Strategic Property Management 7(1): 1-9.

Brauers, W. K.; Lepkova, N. 2002.The application of the nominal group technique to the economic outlook of Lithuania over the period 2002-2011, Technological and Economic Development of Economy [ikio technologinis ir ekonominis vystymas] 8(1): 19-24.

Casella, G.; Berger, R. L. 2002. Statistical Inference, second edition. Pacific Grove, CA, US, Duxbury, Thomson Learning.

Churchman, C. W.; Ackoff, R. L.; Arnoff, E. L. 1957. Introduction to Operations Research. New York, US, Wiley.

Churchman, C. W.; Ackoff, R. L. 1954. An approximate measure of value, Operations Research 2: 172-180.

Dalkey, N.; Helmer, O. 1963. An experimental application of the delphi method to the use of experts, Management Science, 458-467.

Darnell, A. C. 1997. Dictionary of Econometrics. Cheltenham, UK, Edward Elgar.

Dasgupta, P.; Maskin, E. 2008. On the robustness of majority rule, Journal of The European Economic Association 6(5): 949-973.

Edin, P. A.; Ohlson, H. 1991. political determinants of budget deficits: coalition effects versus minority effects, European Economic Review 35(8): 1597-1603.

Frisch, R. 1933. Pitfalls in the statistical construction of supply and demand curves, Veroffentlichungen der Frankfuther Gesellschaft fur Konjuncturforschung, New Series, V(II). Leipzig , Germany, Hans Buske.

Hays, W. L. 1974. Statistics for the Social Sciences. London GB, Holt, Rinehart and Winston.

Heckman, J. J. 1992. Havelmo and the birth of modern econometrics, Journal of Economic Literature XXX: 876-886.

Hoel, P. G. 1971. Elementary Statistics. New York, US, Wiley.

Huber, P. J. 1981. Robust Statistics. New York, US, Wiley.

Huber, P. J. 1969. Thiorie de l'infirence statistique robuste. Montreal, Canada, les Presses de l'Universite de Montreal.

Intriligator, M. D. 1978. Econometric Models, Techniques and Applications. Amsterdam, Nl, North Holland.

Johnston, J. 1963. Econometric Models. New York, US, McGraw Hill.

Karlin, S.; Studden, W. J. 1966. Tchebycheff Systems: with Applications in Analysis and Statistics. New York, Interscience Publishers.

Keeney, R. L.; Raiffa, H.1993. Decisions with Multiple Objectives, Preferences and Value Tradeoffs. Cambridge U.S., University Press.

Kendall, M. G. 1948. Rank Correlation Methods. London, Charles Griffin Cy.

Kendall, M. G.; Gibbons, J. D. 1990. Rank Correlation Methods. London, Edward Arnold.

Kennedy, P. 1998. A Guide to Econometrics. Oxford. GB, Blackwell.

Kreps, D. M. 1990. Game Theory and Economic Modelling. Oxford. GB, Oxford University Press and Clarendon Press.

Machina, M. J.; Schneider, D. A. 1992. More robust definition of subjective probability, Econometrica 60(4): 745-780.

Madansky, A. 1976. Foundations of Econometrics. Amsterdam Nl, North Holland.

Matyas, L.; Sevestre, P. 1992. The Econometrics of Panel Data. Handbook of Theory and Applications, Advanced Studies in Theoretical and Applied Econometrics, vol. 28, Dordrecht, Nl., Kluwer Academic Publishers.

McGregor, W.; Shiem-Shim Then, D. 1999. Facilities Management and the Business of Space. London, GB, Butterworth-Heinemann.

Mills, J. A. 1992. Bayesian prediction tests for structural stability, Journal of Econometrics 52(3): 381-388.

Mueller, J. H.; Schuessler, K. F.; Costner, H. L. 1970. Statistical Reasoning in Sociology 2nd Ed. Boston US, Houghton Mifflin.

Rhodes, G. F.; Fomby, T. B. (Eds.). 1988. Nonparametric and Robust Inference. Greenwich, Conn., US, JAI Press.

Rieder, H. (Ed.). 1996. Robust Statistics, Data Analysis and Computer Intensive Methods. New York, US, Springer.

Ruggeri, F. 2008. Bayesian robustness, European Working Group, Multiple Criteria Decision Aiding, Series 3, 17: 6-10.

Samuelson, P. A. 1994. Facets of Balassa-Samuelson thirty years later, Review of International Economics, 201-226.

Samuelson, P. A. 1964.Theoretical notes on trade problems, Review of Economics and Statistics, 145-154.

Sarndal, C.-E.; Swensson, B.; Wretman, J. 1992. Model Assisted Survey Sampling. New York, US, Springer.

Stigler, S. 1973. Simon Newcomb, Percy Daniell and the History of Robust Estimation 1885-1920, Journal of the American Statistical Association 68: 872-9.

Tinbergen, J. 1930. Bestimmung und Deutung von Angebotskurven, Zeitschrift fur Nationalokonomie 1(1): 669-679.

Thomas, R. L. 1985. Introductory Econometrics. London, GB, Longman.

Van De Ven A. H.; Delbecq, A. L. 1971. Nominal versus interacting group processes for committee decision-making effectiveness, Academy of Management Journal 14(2): 203 and fol.

Vilutieni, T.; Zavadskas, E. K. 2003. The application of multi-criteria analysis to decision support for the facility management of a city's residential district, Journal of Civil Engineering and Management 9(4): 241-252.

Vincke, P. 1999. Robust solutions and methods in decision aid, Journal of Multi-Criteria Decision Analysis, 181-87.

Walters, A. A. 1973. An Introduction to Econometrics. London, GB, Macmillan.

Wonnacott, R. J.; Wonnacott, T. H. 1970. Econometrics. New York, US, Wiley.

Zavadskas, E. K.; Vilutieni, T. 2006. A multiple criteria evaluation of multi-family apartment block's maintenance contractors: I-Model for maintenance contractor evaluation and the determination of its selection criteria, Building and Environment 41: 621-632. Elsevier.

Zavadskas, E. K.; Vilutieni, T. 2004. Multi-criteria analysis of multi-family apartment blocks maintenance service packages, Journal of Civil Engineering and Management 10(Suppl. 2): 143-152.

(1) As well known, statistics already existed in Roman times with the census of population.

(2) At a later time, namely in 1981, Huber wrote a more complete book on Robust Statistics. In 1994 at the occasion of Huber's birthday his colleagues edited a book on Robust Statistics (editor: Rieder).

(3) A good overview of this problem of robustness and Bayesian Analysis is brought by Ruggeri, 2008.

(4) Webster's new Universal Unabridged Dictionary: robust: strong; stronger, strongest.

(5) The pioneer of Rank Correlation, Kendall (1948: 1), pretends the contrary but without any proof. Therefore, the introduction of the notion of Correlation is of no use for multiple objectives (Brauers 2007: 450-451).

(6) Brauers and Zavadskas (2006) prove that the most robust choice for this denominator is the square root of the sum of squares of each alternative per objective.

(7) Instead of a normal increase in productivity growth, a decrease remains possible. At that moment the interval becomes [-1, 1]. Take the example of productivity, which has to increase (positive). Consequently, we look for a maximization of productivity, e.g. in European and American countries. What if the opposite does occuri For instance, take the original transition from the USSR to Russia. Contrary to the other European countries, productivity decreased. It means that in formula (2) the numerator for Russia was negative with the whole ratio becoming negative. Consequently, the interval changes to: [-1, +1] instead of [0, 1].

(8) Brauers 2008(b) proves that the Min-Max metric is the most robust choice between all the possible metrics of reference point theory.

(9) Dr. Tatjana Vilutiene took care of the field work. Previously, she already participated in a similar studies (Zavadskas; Vilutiene 2006, 2004; Vilutiene, Zavadskas 2003.)

(10) These additional objectives were: quality standard of management services, quality of maintenance of common property, work organization, the efficiency of information use, certification of company, range of services, reliability of company, company reputation, staff qualification and past experience, communication skills, and geographical market restrictions.

Willem K. Brauers [1], Edmundas K. Zavadskas [2]

[1] Faculty of Applied Economics and Institute for Development Policy and Management, University of Antwerp

[2] Department of Construction Technology and Management, Vilnius Gediminas Technical University, e-mail: edmundas.zavadskas@adm.vgtu.lt

Willem K. Brauers was graduated as Ph.D. in economics (Un. of Leuven), Master of Arts (in economics) of Columbia Un. (New York), Master in Management and Financial Sciences, in Political and Diplomatic Sciences and Bachelor in Philosophy (Un. of Leuven). He is professor at the Faculty of Applied Economics and at the Institute for Development Policy and Management of the University of Antwerp. Previously, he was professor at the University of Leuven, the Belgian War College, the School of Military Administrators, and the Antwerp Business School. He was a research fellow in several American institutions like Rand Corporation, the Pentagon, the Institute for the Future, the Futures Group and extraordinary

Table 1. Main attributes and objectives of contractors in Vilnius (2004)

No. Attributes Units of Max
 measurement Min

1 Cost of building management Lt*/[m.sup.2] Min [X.sub.1]
2 Cost of common assets Lt/[m.sup.2] Min [X.sub.2]
 management
3 Hvac system maintenance Lt/[m.sup.2] Min [X.sub.3]
 cost (mean)
4 Courtyard territory Lt/[m.sup.2] Min [X.sub.4]
 cleaning (in summer)
5 Total service cost Lt/[m.sup.2] Min [X.sub.5]
6 Length of time in maintenance Years Max [X.sub.6]
 business (experience)
7 Market share for % Max [X.sub.7]
 each contractor
8 Number of projects Units/person Max [X.sub.8]
 per executive
9 Evaluation of management -- Max [X.sub.9]
 cost ([c.sub.min]/[c.sub.p])

* Lt means Litas, the official currency of Lithuania

Table 2. Initial decision making matrix of 15 contractors in
Vilnius (2004)

Alternatives [X.sub.1] [X.sub.2] [X.sub.3]
[arrow down]

 [A.sub.1] 0.064 0.11 0.18
 [A.sub.2] 0.06 0.14 0.37
 [A.sub.3] 0.057 0.11 0.18
 [A.sub.4] 0.058 0.12 0.09
 [A.sub.5] 0.058 0.1 0.18
 [A.sub.6] 0.071 0.3 0.18
 [A.sub.7] 0.11 0.14 0.18
 [A.sub.8] 0.058 0.18 0.37
 [A.sub.9] 0.053 0.14 0.16
 [A.sub.10] 0.071 0.26 0.29
 [A.sub.11] 0.12 0.2 0.09
 [A.sub.12] 0.071 0.28 0.18
 [A.sub.13] 0.078 0.2 0.18
 [A.sub.14] 0.056 0.14 0.18
 [A.sub.15] 0.12 0.14 0.09

Alternatives [X.sub.4] [X.sub.5] [X.sub.6]
[arrow down]

 [A.sub.1] 0.31 0.67 12
 [A.sub.2] 0.12 0.5 3
 [A.sub.3] 0.15 0.69 12
 [A.sub.4] 0.15 0.57 12
 [A.sub.5] 0.2 0.45 12
 [A.sub.6] 0.26 0.82 13
 [A.sub.7] 0.12 0.55 5
 [A.sub.8] 0.19 0.61 11
 [A.sub.9] 0.23 0.8 11
 [A.sub.10] 0.23 0.73 11
 [A.sub.11] 0.2 0.81 4
 [A.sub.12] 0.28 0.73 12
 [A.sub.13] 0.3 0.76 8
 [A.sub.14] 0.12 0.5 11
 [A.sub.15] 0.21 0.56 3

Alternatives [X.sub.7] [X.sub.8] [X.sub.9]
[arrow down]

 [A.sub.1] 11.75 4.6 0.83
 [A.sub.2] 0.39 0.33 0.885
 [A.sub.3] 5.25 1.47 0.935
 [A.sub.4] 7.09 2.78 0.912
 [A.sub.5] 5.56 1.39 0.912
 [A.sub.6] 26.62 5.67 0.746
 [A.sub.7] 2.82 1.2 0.483
 [A.sub.8] 9.48 3.03 0.916
 [A.sub.9] 2.23 0.76 1
 [A.sub.10] 13.47 9.05 0.746
 [A.sub.11] 4.7 1.5 0.443
 [A.sub.12] 2.35 0.86 0.746
 [A.sub.13] 5.6 3.25 0.681
 [A.sub.14] 2.66 1.7 0.948
 [A.sub.15] 0.04 0.03 0.531

Table 3. Ranking of the 15 contractors by the 2 parts of MOORA

Contractors MOORA Square MOORA
 Root Part Reference Point Part

[A.sub.1] 4 3
[A.sub.2] 12 14
[A.sub.3] 6 8
[A.sub.4] 3 5
[A.sub.5] 5 7
[A.sub.6] 1 1
[A.sub.7] 11 10
[A.sub.8] 8 4
[A.sub.9] 9 13
[A.sub.10] 2 2
[A.sub.11] 14 9
[A.sub.12] 13 12
[A.sub.13] 10 6
[A.sub.14] 7 11
[A.sub.15] 15 15

Table 4. No discussion on the ranking of the first four positions
of the 15 contractors after the 2 parts of MOORA

Contractors MOORA MOORA
 Square Root Part Reference Point Part

 A6 1 1
 A10 2 2
 A1 4 3
 A4 3 5
A2, A3, A5; Not very clear Not very clear
A7-A9; A11-A14
 A15 15 15