Assessing the performance of supervisors: lessons for practice and insight into middle management resistance to change.

Daley, Dennis M. ; Lovrich, Nicholas P.

ABSTRACT

Data from the U.S. Merit Systems Protection Board's First-Line Supervisors survey conducted in 1991 is investigated to assess the relative power of ratings on twelve management functions (what managers do) and nine effectiveness characteristics (how the work is performed) in the prediction of performance appraisal outcomes. Findings from this secondary analysis of the MSPB survey data indicate that a subset of the Office of Personnel Management's Management Excellence Inventory (MEI) provides a broadly useful foundation for public sector supervisor performance appraisal. As importantly, however, the analysis reported here also reveals some troublesome findings regarding impediments to employee empowerment and the devolution of problem solving incentives to lower levels of bureaucratic authority. There is evidence that middle managers tended to view supervisors less favorably when they took a greater strategic view and exercised more leadership initiative in problem solving. The more intrepid supervisors were indeed taking career risks (receiving lower performance ratings) by engaging in more reinvention of government than their bosses were comfortable with at the dawn of the federal reinventing government effort.

INTRODUCTION

This study examines the perceptions of public sector second-line supervisors which come into play in assessing their own subordinate, first-line supervisors with respect to what factors contribute to their employees' success. The first-line supervisor nearly universally translates organizational goals and objectives into implementable instructions to operatives; the supervisor mediates "what is to be done" between the management and employees. Consequently, the accurate appraisal of supervisory performance is of considerable importance. Recent research in this area has focused on extending the conventional assessment of job-related tasks to include "organizational fit" competencies that enable the individual's job performance to be integrated successfully into an organization's overall mission (Bowen, Ledford, and Nathan, 1991; van Dyne and LePine, 1998).

The Federal First-Line Supervisor Survey (MSPB, 1992) is used for an examination of second-line supervisors' perceptions of the relative importance of a set of tasks and performance characteristics, and to assess how these perceptions relate to their evaluation of the job incumbent's overall performance. The job tasks and performance characteristics considered entail the major responsibilities or functions of a supervisory job; hence, judgments on those elements of a supervisor's performance and personal qualities should be strongly related to an overall assessment of how well a particular supervisory job is being performed. Because organizations selectively recruit (and orientate and train) to enhance success, performance appraisal ratings are understandably positively skewed. This study shows that the federal government workforce is typical in this regard. The vast bulk of the federal first-line supervisors studied here (about 80 percent) received "exceeds fully successful" and "outstanding" ratings; qualitative assessments were equally skewed to the "effective" and "very effective" choices (about 90 percent). Virtually no first-line supervisor (under half a percent) was formally classified as "unsatisfactory" or "minimally successful," or qualitatively judged as "not effective."

The idea of simplifying performance appraisal systems into two-point, "Pass-Fail" options is currently under wide discussion, especially in the federal government (Friel, 1998; U.S. Office of Personnel Management, 1996a, 1996b, 1997, 1998). While the existing five- (and seven-) point performance appraisal scales offer the illusion of fine distinctions in regard to determining performance ratings, the reality is that few supervisors actually use all the options available. Virtually no one is ever classified in any "failing" category. While a simpler, "Pass-Fail" performance appraisal would hardly be worth the effort and cost of administration (virtually everyone would "pass"), some form of "Low Pass-High Pass" might be feasible. Using data from federal supervisory appraisals, this paper examines whether various "Low Pass-High Pass" options would produce meaningful distinctions.

The analysis performed here focuses on which factors separate or distinguish "Low Pass" from "High Pass" individuals among first-line supervisors. Two separate options are examined here. First, the grouping of "Low Pass" performance ratings (1) "Unsatisfactory," (2) "Minimally Satisfactory," and (3) "Fully Satisfactory" are contrasted with "High Pass" supervisors rated (4) "Exceeds Fully Satisfactory" and (5) "Outstanding." Similarly, qualitative assessments of (1 and 2) "Not Effective" to (3) "Neither Effective nor Ineffective" are contrasted with (4 and 5) "Effective" assessments. Second, the "cut- point" is changed in these analyses to separate the STAR (4) "Exceeds" rating and (4) "Effective" assessment from the SUPERSTAR (5) "Outstanding" rating and (5) "Very Effective" assessment.

Since formal performance appraisals may serve a range of purposes, the core tasks and performance characteristics associated with supervisory positions are also analyzed in terms of informal supervisory assessments (Gabris and Giles, 1983; Longenecker, Sims, and Gioia, 1987).

Analysis is conducted on the responses of 1,779 matched first-line and second-line supervisors participating in a mail survey administered by the U.S. Merit Systems Protection Board (MSPB) in the Spring of 1991. That survey asked first-line supervisors to rate the importance of, and their own ability to do, a set of 118 supervisory and/or management tasks.Second-line supervisors (matched with those they supervised) were also asked to rate their subordinate first- line supervisor on these same job task items. These administrative tasks are grouped together into twelve functional indices (representing "what" managers do) and nine effectiveness indices (representing "how" they do it).

DATA AND METHODS

Management Excellence Inventory

The Merit Systems Protection Board conducted the survey of first-line supervisors as part of its effort to assess the quality of performance of employees managing federal programs (MSPB 1992). MSPB based its survey instrument on a Management Excellence Inventory (MEI) developed by the Office of Personnel Management (OPM) in the 1980s (Flanders and Utterback 1985). The OPM surveyed federal managers (GS/GM 13-15 and SES) about their job roles, conducted critical incident interviews, and performed comprehensive subject matter expert (i.e., job incumbent) job analyses to develop the MEI. Developers of the MEI identified 118 tasks grouped into 21 indices (see Appendix for index descriptions). Twelve of the indices assess the functions (or the "what ") of management (e.g., external awareness, interpretation, coordination, work unit planning, budgeting), and nine indices represent effectiveness characteristics (or "how") associated with effective management (e.g., broad perspective, flexibility, ability to communicate). The MSPB dropped a tenth MEI effectiveness characteristic, technical competence, from their survey of first-line supervisors because they believed it was too job-specific in its focus.

Because the MEI was constructed using information from job incumbents, it possesses both face and content validity. Additional independent studies were conducted to cross-validate the instrument. Cronbach-alpha tests of the relationship among items in each index indicated the presence of reasonable reliability, ranging from .6 to .8 (Flanders and Utterback 1985).

First-Line Supervisors Survey

A total of 2,533 first-line supervisor questionnaires were returned from the April 1991 First- Line Supervisors Survey, representing a 65% rate of response. When these respondents are matched with their second-line supervisors, a total of 1,779 matched sets of first- and second-line supervisors are retained in the study population. Since we are not concerned with the special supply clerk/technician and accountant aspects of the MSPB investigation, the matched first- line/second-line supervisor sample (n = 1,779) is weighted to reflect the appropriate general population proportions for the federal job series.

The 21 indices representing either managerial functions or effectiveness characteristics are made up of between 3 and 18 individual items. These items represent tasks which first-line supervisors were asked to rate along two distinct dimensions. The first dimension was the extent to which the supervisor believes each task is important for performing his or her job, rated on a scale ranging from 1 (to no extent) to 5 (to a very great extent). Responses indicating that they do not perform the task, or that they can't judge its importance, were coded as missing. The second dimension was the respondent's own rating of his or her ability to perform each task, using a scale ranging from 1 (cannot do this task at an acceptable level) to 5 (can do this task exceptionally well). Those marking a 6th choice (can't judge) were coded as missing. Second-line supervisors were presented the same items and asked to indicate how important they felt each task was for the supervisor in question, and how well they thought their first-line supervisor actually performed the tasks in question.

Managerial Function and Effectiveness Iindexes

Whereas the MSPB (1992) analysis of survey results focuses on the specific items, this research effort uses their broad-based, composite indices for more intensive study. Each of the twenty-one indices were constructed in two steps. First, the three to eighteen individual items associated with each of the managerial functions and effectiveness characteristics were summed (excluding missing values and "can't judge" responses). Second, for each of these the resultant additive score was divided by a count of the total number of items without missing values (to adjust for the specific number of items to which an individual had responded). Virtually identical results are obtained from analyses conducted using a more restrictive additive index which discarded all cases where one item has a missing value. It should be noted that many respondents have index scores featuring excluded "missing" items because the supervisors felt they did not perform that function or require that characteristic in their work.

In this analysis, the "importance" and "ability" index scores were multiplied to create weighted "performance" indexes (Weighting or "priority points" is a common approach in the performance rating process itself--see U.S. Office of Personnel Management, 2001). Performance indexes were constructed from second-line supervisor assessments of their subordinate, first-line supervisors. The twenty-one performance indexes are used as independent variables to predict second-line supervisor' performance assessments of their first-line supervisor

Logistic Regression Analysis

These sets of component performance indexes were asse0ssed in logistic regression models in which the second-line supervisor's recall of the first-line supervisor's performance RATING and a subjective, QUALITY assessment of that performance served as the dependent variables. Both the RATING and QUALITY measures are each based on a single item with an original five-point Likert scale response. Analysis was performed by converting (scores 1, 2, and 3 versus scores 4 and 5) or selecting a subset (score 4 versus score 5) of these responses into dichotomous categories Hence, there are two logistic regression models testing the MEI job analyses against both formal and informal performance perceptions. Because of the complications introduced into the regression analysis by missing values found among both dependent and independent variables, the weighted sample is further reduced such that the number of cases is 315 for RATING and the number of cases is 366 for

QUALITY ASSESSMENT.

Because all respondents in this study are supervisors (first-line and their second-line superiors), there is a built-in control for job understanding. The various misperceptions and misunderstandings concerning the specific job duties being rated and the task responsibilities properly associated with a position can introduce measurement error into the performance rating process. That common threat to fair and effective performance appraisal is considerably reduced here.

The proper interpretation of logistic regression requires a bit of discussion at this point. Indices that do not obtain statistical significance or indicate substantial variance explained can nonetheless be important; in fact, on the basis of the original MEI job analysis they are indeed assumed to be important factors in the supervisory job. However, the supervisory selection process assures us that virtually everyone already possesses them to some degree (and performs them relatively well). Hence, important as they are for performing the job well, these factors do not serve to distinguish one individual supervisor's performance from that of another. Those who do not perform these functions or exhibit these characteristics simply are not retained as supervisors The logistic regression models focus on those performance indices which enable one to distinguish among varying levels of performance.

FINDINGS AND DISCUSSION

What's Really Important in Determining Supervisory Performance

What managerial functions and effectiveness characteristics are critical to the job of supervisor? Which factors are substantively related to an assessment of supervisory performance? First, the "importance" attached by second-line supervisors to each managerial function and effectiveness characteristic is examined. Second, correlations are used to show the extent to which the individual MEI performance assessment criteria are linked to overall formal quantitative and to informal narrative performance appraisals.

In assessing the contribution that each management function or effectiveness characteristic makes toward a supervisor's formal and informal summative evaluations, it is important to remember that we are at this time assessing "marginal" differences. To be a supervisor in the first place requires a substantial "entry fee" in terms of talent and ability. Furthermore, once on duty, the vast majority of supervisory employees do their respective jobs well. Hence, the performance appraisals conducted in annual reviews focus on what separates those who "exceed expectations" from those who are judged to be "outstanding."

As a preliminary step in ascertaining what factors distinguish "outstanding" performance from commendable performance, it is important to document precisely what second-line supervisors believe are the more essential facets of the supervisory job. The second-line supervisor rankings of the 21 managerial functions and effectiveness indexes are presented in Table 1. The MEI indexes represent mean scores which can be interpreted on the same 1 (to no extent) to 5 (to very great extent) scale used for the survey's individual items. In essence, the results reported in Table 1 show all these functions and characteristics to be "important." Even the lowest ranked managerial function ("Budgeting" at 21) has a mean equivalent to something which is interpretable as more than the 3 (some extent) level on the single item response continuum. In fact, over two-thirds of the index means exceed the 4 (considerable extent) level of importance. In essence, the second-line supervisors add further content validity to the MEI's job analysis of the supervisory position. In general, second-line supervisors accord more importance to the several effectiveness characteristics than they do to the specific managerial functions. Six of the nine effectiveness characteristics are included in the top nine/ten ranks. While the internal role managerial functions are important to supervisory success, it is the external role effectiveness characteristics (competencies) that appear to have greater precedence.

Having established the "importance" of the core managerial functions and essential effectiveness characteristics to the supervisory job, a second requirement is to actually show that they are related to an assessment of success in a supervisory role. Do second-line supervisors make ultimate assessments of supervisorial performance in line with their evaluations of these specific dimensions of performance?

The correlations between the weighted performance measures of the managerial function and effectiveness characteristic assessments and the formal objective RATING and informal subjective QUALITY ASSESSMENT are presented in Table 2. Correlations running from r=.20 to r=.40 are associated with RATING, while QUALITY ASSESSMENT achieves higher levels of association, falling in the r=.28 to r=.62 range. These moderate to strong correlations are sufficient to merit the inclusion of all twenty-one managerial functions and effectiveness characteristics in the multivariate analysis to follow. It is likely that second-line supervisors do indeed "use" the MEI indexes in their determination of supervisory performance.

For the most part, the managerial functions and effectiveness characteristics have relatively similar correlations. There are slightly higher correlations with respect to the informal, subjective QUALITY ASSESSMENT. The effectiveness characteristics also tend to register higher correlations than are produced by the managerial functions. This latter finding reinforces the precedence accorded effectiveness characteristics (or competencies) indicated in the perceptions of the importance of the 118 supervisory performance scale items.

Logistic Regression Models.

Two dependent variables -- a formal, objective RATING and an informal, subjective QUALITY ASSESSMENT--are used in separate regression models. While the RATING and QUALITY ASSESSMENT measures are weakly correlated (r=.19) and the restricted (4) and (5) RATING and QUALITY ASSESSMENT measures are moderately correlated (r=.39), they are by no means identical. The measurement of performance encompasses many complexities that are clearly at work here. Because RATING and QUALITY ASSESSMENT both include degrees of measurement error, their joint use is an appropriate means for examining the effect of managerial functions and effectiveness characteristics on evaluations of supervisory performance.

Two sets of analyses are performed. First, a Low Pass-High Pass contrast is examined. The very few cases of failure are combined with those who are "Fully Satisfactory" or "Neither Effective nor Ineffective" and contrasted with the combined "Exceeds" and "Outstanding" RATINGS and "Effective" and "Very Effective" QUALITY ASSESSMENTS. Second, a restricted analysis is conducted focusing solely on the Stars and Superstars. The "Exceeds" RATING is contrasted with the "Outstanding" RATING; similarly, the "Effective" QUALITY ASSESSMENT is contrasted with the "Very Effective" QUALITY ASSESSMENT.

Low Pass vs. High Pass

The logistic regression analyses contrasting the (1), (2), and (3) "Fully Satisfactory" and lower with the combined (4) "Exceeds" and (5) "Outstanding" RATINGS and the (1) and (2) "Not Effective" and (3) "Neither Effective nor Ineffective" with the (4) and (5) "Effective" QUALITY ASSESSMENTS are displayed in Table 3. Analysis contrasting (3) "Fully Satisfactory"/"Neither Effective nor Ineffective" with (4) "Exceeds"/"Effective" responses produces similar results. The MEI job analysis managerial functions and effectiveness characteristics account for a limited amount of the variance in RATING (adjusted logistic R2 = .10) and QUALITY ASSESSMENT (adjusted logistic R2 = .26). However, in examining the percentage of correctly predicted cases for RATINGS (86%) and QUALITY ASSESSMENT (92%), serious concern is raised in regard to the model's overall fit. Because of the extremely skewed distribution, these are only marginal improvements over the modal category prediction of 84% for RATING (a 2% gain) and 92% for QUALITY ASSESSMENT (no change). Lamba-p statistics are .14 for RATING and .00 for QUALITY ASSESSMENT, respectively.

While due in part to the multiplicative index employed here, the individual unstandardized logistic regression coefficients (Bs) are relatively small (mostly under .1 for RATING, with some reaching .3 for QUALITY ASSESSMENT). The dependent variables both have only a 1-5 range, while the independent variables range from 1-25. In interpreting these unstandardized logistic regression coefficients, each index should be multiplied by the fifteen to twenty "units" of performance that second-line supervisors indicate for most respondents (see means in Table 2). A change in perceived importance or performance can easily translate into a four-to-five-point increase in the multiplicative measure. Hence, these independent variables can be seen theoretically to have a substantive impact on the five-point performance RATINGS and QUALITY ASSESSMENT scales. The log odds ratios, Exp (B), displayed in Table 3 are perhaps clearer indicators of this impact (adjusting for the multiplicative index's four to five unit increases).

With respect to RATING, no managerial functions or effectiveness characteristics are shown to be statistically significant factors. The formal performance appraisal process provides no information that can be used to distinguish those who are a "Low Pass" from those categorized as "High Pass." Differences in performing the managerial functions and exhibiting effectiveness characteristics fail to allocate supervisors to one or the other category of performance.

However, the QUALITY ASSESSMENT logistic regression highlights two managerial functions (Work Unit Guidance and Coordination) and two effectiveness characteristics (Action Orientation and Broad Perspective) as being statistically significant factors at play. Work Unit Guidance enhances the odds of obtaining a higher assessment by 32% per unit, while Coordination adds to the odds by 24%. Action Orientation is an effectiveness characteristic which can affect the odds of receiving the higher "Effective" assessments by 41% per unit increase in the index. Similarly, the characteristic Broad Perspective enhances the odds of a "High Pass" by 37%. The ability to facilitate communication between employees and upper level managers expressed in the Work Unit Guidance and Coordination functions is clearly seen as important. This is also reflected in the high regard for the Action Orientation and Broad Perspective characteristics that involve the balancing of long and short term interests. In contrasting the RATING and QUALITY ASSESSMENT logistic regression results, one witnesses a stark distinction. While the formal RATING process remains silent, the informal, subjective QUALITY ASSESSMENT almost shouts. It is unfortunate that the formal performance appraisal process fails to provide feedback as to what distinguishes supervisors who are barely making it (Low Pass) from those who are excelling in their work (High Pass). However, it is in the informal QUALITY ASSESSMENT that we find these insights. The ability to communicate "what is to be done" is clearly highly prized.

Stars vs. Superstars

The logistic regression analyses for the restricted (4) "Exceeds"/"Effective" and (5) "Outstanding"/"Very Effective" RATING and QUALITY ASSESSMENT are displayed in Table 4. Virtually identical results are obtained when the (3) "Fully Satisfactory"/"Neither Effective nor Ineffective" responses are combined with the (4) "Exceeds"/"Effective"responses in contrast with the (5) "Outstanding"/"Very Effective." The MEI job analysis managerial functions and effectiveness characteristics account for somewhat respectable amounts of the variance in RATING (adjusted logistic R2 = .12) and QUALITY ASSESSMENT (adjusted logistic R2 = .29). This is more evident in examining the percentage of correctly predicted cases for RATINGS (74%) and QUALITY ASSESSMENT (79%). These are improvements over the modal category prediction of 53% for RATING (a 21% gain) and 52% for QUALITY ASSESSMENT (a 27% gain). Lamba-p statistics are .45 for RATING and .56 for QUALITY ASSESSMENT, respectively.

In essence, the logistic regression models focus on distinguishing higher levels of success rather than demarcating a more complete path from failure to success. As many of the independent variables register no differences among respondents, they "drop out" of the model even though they are broadly viewed as essential components in supervisory performance.

It is notable that these MEI independent variables apparently have a greater impact on the assessment of QUALITY, a subjective assessment of performance, than they do on the more objective formal RATING. Free of the organizational and structural impediments that can influence the formal performance appraisal process (Gabris and Giles, 1983; Longenecker, Sims, and Gioia, 1987; Daley, 1992: 119-121), supervisors express their views with somewhat more candor in these informal assessments. These informal, qualitative assessments may account for much of the "unexplained" variance in promotional and developmental decisions.

The individual unstandardized logistic regression coefficients (Bs) are relatively small (.1 to .2); however, this is due in part to the multiplicative index employed here. As previously noted, the dependent variables both have only a 1 to 5 range, while the independent variables range from 1 to 25. In interpreting these unstandardized logistic regression coefficients, each index is properly multiplied by the fifteen to twenty "units" of performance that second-line supervisors indicate for most respondents (see means in Table 2). A change in perceived importance or performance can easily translate into a four- to five-point increase in the multiplicative measure. Hence, these independent variables can be seen to have a substantive impact on the five-point performance RATINGS and QUALITY ASSESSMENT scales. The log odds ratios, Exp (B), displayed in Table 4 are perhaps clearer indicators of this impact.

With respect to RATING, three managerial functions (Work Unit Guidance, Coordination, and Interpretation) and four effectiveness characteristics (Leadership, Results Focus, Broad Perspective, and Strategic Vision) are shown to be statistically significant factors at play. For Work Unit Guidance the odds of receiving an "Outstanding" over an "Exceeds Fully Satisfactory" increase 12% for each index unit; enhanced Coordination registers a 17% increase in odds per unit. Possessing a Results Focus and a Broad Perspective augment one's odds 22% and 18%, respectively. All these managerial functions and effectiveness characteristics share a forward looking, goal-directed approach to job performance. The managerial functions entail team-related implementation while the effectiveness characteristics foreshadow a "can do" attitude.

In contrast, first-line supervisors who are seen to excel at Interpretation and Leadership along with having a Strategic View are seen to decrease their chances of obtaining a higher rating! Since these factors are generally considered to be "good things" for a supervisor to do, these results are somewhat troubling. For Interpretation, the odds decline roughly 9% per unit while the Leadership odds decline 22%. A Strategic View diminishes one's chances by 15%. Interpretation, Leadership and Strategic View differ from some of the other, closely related managerial functions and effectiveness characteristics in their somewhat greater emphasis on responsibility and accountability. These indexes would seem to tap factors that involve keeping subordinates informed (beyond merely communicating what is to be done) and encouraging them to analyze priorities independently. They suggest that the concepts of employee empowerment and decisional delegation are not seen as truly useful for middle managers. For second-line supervisors the qualities of

Interpretation, Leadership, and Strategic View are seen as potential distractions from or hindrances to the goal achievement ethos. While these findings are not encouraging to those who advocate an employee empowerment approach to workplace design, they do correspond with the literature on "middle manager undermining" of such initiatives. Whether it be a matter of uncommon stress suffered by middle managers (Kahn, Wolfe, Quinn, Snoek, and Rosenthal, 1964: pp. 322-327), the need they feel for controlling information (Hummel, 1987: pp. 250-251), the fear they have of not being in control of ultimate outcomes (Bennis, 1989: pp. 81-85), or the concern they have that empowered employees are a potential threat to their status (Cotton, 1993: pp. 55-57; Wellins, Byham, and Wilson, 1991: pp. 92-93), there is indeed a strong basis in the literature on workplace employee involvement and empowerment that warns of middle management resistance.

The QUALITY ASSESSMENT logistic regression highlights three managerial functions (Work Unit Guidance, Budgeting, and Program Evaluation) and one effectiveness characteristic (Action Orientation) as being statistically significant factors at play. Work Unit Guidance enhances the odds of obtaining a higher assessment by 11% per unit, while Budgeting adds to the odds by 7%. However, effort devoted to Program Evaluation is seen to decrease the odds of being perceived as very effective by 10% per unit. Action Orientation is an effectiveness characteristic which can affect the odds of receiving the higher "Very Effective" assessment by 22% per unit increase in the index. Work Unit Guidance, Budgeting, and Action Orientation display the same goal-directed emphasis that was evident among many of the other factors with regard to RATING. Similarly, Program Evaluation shares in the analytic, priority assessment that contributed to the negative effects of Interpretation, Leadership, and Strategic View.

In contrasting the RATING and QUALITY ASSESSMENT logistic regression results, one can almost identify a "halo effect" from Action Orientation as the driving element in the qualitative assessment. On the other hand, the more formal RATING appears to actually focus attention on a number of factors. Work Unit Guidance and Coordination are relatively equal in importance, as are possessing a Results Focus and Broad Perspective (or placing too much emphasis upon a Strategic View).

However, the informal, subjective QUALITY ASSESSMENT (with its Action Orientation "halo") may hold more importance if it is the basis for later developmental assignments and promotional considerations.

CONCLUSION

Inasmuch as the MEI purports to be a job analysis of the supervisory position, it should fully capture the essential responsibilities of the job. With Low Pass/High Pass and Star/Superstar adjusted logistic R2s of .10 and .12 (RATING) and .26 and .29 (QUALITY ASSESSMENT), respectively, this "universality" is somewhat questioned, even for this restricted-range analysis. While the qualitative assessment measure indicates that supervisors do indeed identify factors that distinguish "Low Pass" from "High Pass" subordinates, they are most reluctant to incorporate these opinions in formal performance ratings. This finding seriously limits the utility of performance appraisal as a formal, judgmental instrument. In their "off-the-record" qualitative assessments supervisors focus on Work Unit Guidance and Coordination functions and Action Orientation and Broad Perspective characteristics to separate "Low Pass" from "High Pass" firstline supervisors. .

While supervisors are somewhat circumspect in calculating formal performance ratings for distinguishing between Stars and Superstars, they do tend to focus on measures that translate organizational goals into individual objectives (Work Unit Guidance and Coordination) or tap boundary spanning, inter-unit behaviors (Results Focus and Broad Perspective). This is even more clearly evident when informal, qualitative assessments (that are likely to influence promotional recommendations) are broached. The communication and implementation of organizational goals (Work Unit Guidance and Budgeting) by a dynamic, calculated risk taker (Action Orientation) is clearly the choice.

While the MEI is designed to be a job analysis outlining all the factors essential for the successful performance of the supervisory job, a number of indexes (Interpretation, Leadership, and Strategic View for RATING; Program Evaluation for QUALITY ASSESSMENT) produced negative results with respect to Star/Superstar designations. This is not only surprising, but rather disheartening. While supervisors most certainly desire to perform their jobs successfully, they are also strongly committed to their careers. These factors pose a dilemma potentially pitting job performance against career progress. Apparently, second-line supervisors in the federal government do not want their first-line, subordinate supervisors to dwell too long on questioning or analyzing options and priorities for themselves. While these are important activities, supervisors who are regarded as Superstars are not distracted by them. They stay focused on their main tasks.

It is clear from the findings reported here that in the federal management ranks in 1991 the preference was for a positive "Can Do" response to initiatives flowing from above. Highly rated first-line supervisors excelled at prioritizing and implementation. Second-line supervisors saw themselves as properly providing visionary leadership, while their first-line supervisors were seen as most properly devoting themselves to managing the programs designed to achieve those goals energetically and enthusiastically without exercising too much independence of judgment or action. To the extent that reinventing government entails the movement of decisional authority downward, as close as possible to where problems are being dealt with, it would appear that supervisors that took that message to heart in 1991 did so at some risk to their own supervisor's assessment of their performance. It is clear that the reinvention of government has encountered many obstacles over the past decade; it would appear that middle management resistance should be listed among those obstacles. It would be very instructive to compare these findings from the 1991 survey to a replication survey conducted among contemporary federal supervisors.

Appendix

Management Excellence Inventory Flanders and Utterback 1985: 405 (Figures 2 and 3)

The "What" of Management: Functions

1. External Awareness

Identifying and keeping up-to-date with key agency policies and priorities and/or external issues and trends (e.g., economic, political, social, technological) likely to affect the work unit.

2. Interpretation

Keeping subordinates informed about key agency and work unit policies, priorities, issues, and trends and how these are to be incorporated in work unit activities and products.

3. Representation

Presenting, explaining, selling, and defending the work unit's activities to the supervisor in the agency, and/or persons and groups outside the agency.

4. Coordination

Performing liaison functions and integrating work unit activities with the activities of other organizations.

5. Work Unit Planning

Developing and deciding upon longer-term goals, objectives, and priorities; and developing and deciding among alternative courses of action.

6. Work Unit Guidance

Converting plans to actions by setting short-term objectives and priorities; scheduling/sequencing activities; and establishing effectiveness and efficiency standards/guidelines.

7. Budgeting

Preparing, justifying and/or administering the work unit's budget.

8. Material Resources Administration

Assuring the availability of adequate supplies, equipment, facilities; overseeing procurement/contracting activities; and/or overseeing logistical operations.

9. Personnel Management

Projecting the number and type of staff needed by the work unit, and using various personnel management system components (e.g., recruitment, selection, promotion, performance appraisal) in managing the work unit.

10. Supervision

Providing day-to-day guidance and oversight of subordinates (e.g., work assignments, consultation, etc); and actively working to promote and recognize performance.

11. Work Unit Monitoring

Keeping up-to-date on the overall status of activities in the work unit, identifying problem areas, and taking corrective actions (e.g., rescheduling, reallocating resources, etc.).

12. Program Evaluation

Critically assessing the degree to which program/project goals are achieved and overall effectiveness/efficiency of work unit operations, to identify means for improving work unit performance.

The "How" of Management: Effectiveness Characteristics

13. Broad Perspective

Broad, long-term view; balancing short- and long-term considerations.

14. Strategic View

Collecting/assessing/analyzing information; diagnosis; anticipation; judgment.

15. Environmental Sensitivity

"Tuned into" agency and its environment; awareness of importance of non-technical factors.

16. Leadership

Individual; group; willingness to lead and manage, and accept responsibility.

17. Flexibility

Openness to new information; behavioral flexibility; tolerance for stress/ambiguity/change; innovativeness.

18. Action Orientation

Independence, proactivity; calculated risk-taking; problem solving; decisiveness.

19. Results Focus

Concern with goal achievement; follow through, tenacity.

20. Communication

Speaking; writing; listening

21. Interpersonal Sensitivity

Self-knowledge and awareness of impact on others; sensitivity to needs/strengths/weaknesses of others; negotiation; conflict resolution; persuasion.

REFERENCES

Bennis, Warren (1989) Why Leaders Can't Lead: The Unconscious Conspiracy Continues. San Francisco, CA: Jossey Bass.

Bowen, David E., Gerald E. Ledford, and Barry R. Nathan (1991) Hiring for the Organization, Not the Job. Academy of Management Executive, 5, 35-51.

Cotton, John L. (1993) Employee Involvement: Methods for Improving Performance and Work Attitudes. Newbury Park, CA: Sage.

Daley, Dennis M. (1992). Performance Appraisal in the Public Sector: Techniques and Applications. Westport, CT: Quorum.

Flanders, Loretta R., and David Utterback. (1985). The Management Excellence Inventory: A Tool for Management. Public Administration Review, 45 , 403-410.

Friel, Brian (1998). Pass/Fail Appraisal System Debated. Internet site: http://www.govexec.com/dailyfed/0698/062598b2.htm (June 25).

Gabris, Gerald, and William Giles. (1983). Level of Management, Performance Appraisal, and Productivity Reform in Complex Public Organizations. Review of Public Personnel Administration, 3, 45-61.

Hummel, Ralph P. (1987) The Bureaucratic Experience, 3rd ed. New York, NY: St. Martin's.

Kahn, Robert L., Donald M. Wolfe, Robert P. Quinn, J. Diedrick Snoek, and Robert A. Rosenthal. (1964) Organizational Stress: Studies in Role Conflict and Ambiguity. New York, NY: John Wiley & Sons.

Longenecker, Clinton O., Henry Sims, Jr., and Dennis Gioia. (1987). Behind The Mask: The Politics of Employee Appraisal. Academy of Management Executive, 1, 183-193.

Merit Systems Protection Board (1992). Federal First-Line Supervisors: How Good Are They? Washington, DC: U.S. Merit Systems Protection Board.

Van Dyne, Linn, and Jeffrey LePine (1998) Helping and Voice Extra-Role Behaviors: Evidence of Construct and Predictive Validity. Academy of Management Journal, 41, 108-119.

U.S. Office of Personnel Management (1996a) The Policy Perspective On: Pass/Fail Assessment. Workforce Performance Newsletter Reprint Internet site: http://www.opm.gov/perform/articles/089.htm (March).

U.S. Office of Personnel Management (1996b) Keys to Implementing Pass/Fail Programs. Workforce Performance Newsletter Reprint. Internet site: http://www.opm.gov/perform/articles/045.htm (October).

U.S. Office of Personnel Management (1997) Case Study: Leaving Pass/Fail Behind. Workforce Performance Newsletter Reprint. Internet site: http://www.opm.gov/perform/articles/012.htm (August).

U.S. Office of Personnel Management (1998) Pass/Fail Appraisals. Workforce Performance Newsletter Reprint. Internet site: http://www.opm.gov/perform/articles/061.htm (October).

U.S. Office of Personnel Management (2001) A Handbook for Measuring Employee Performance: Aligning Employee Performance Plans With Organizational Goals. Internet site: http://www.opm.gov/perform/wppdf/handbook.pdf (January).

Wellins, Richard S., William C. Byham, and Jeanne M. Wilson (1991). Empowered Teams: Creating Self-Directed Work Groups That Improve Quality, Productivity, and Participation. San Francisco, CA: Jossey-Bass.

DENNIS M. DALEY

North Carolina State University

NICHOLAS P. LOVRICH

Washington State University

Table 1
Percived Importance of Managerial Functions and
Effectiveness Characteristics

 Standard
Variable mean (rank) Deviation

Work Unit Planning 3.86 (18) .78
Work Unit Guidance 4.12 (6/7) .62
Budgeting 3.38 (21) 1.11
Material Resources 3.50 (20) .94
Personnel Management 4.05(9/10) .68
Super 4.03 (12) .59
External Awareness 4.02 (13) .73
Interpretation 4.04 (11) .69
Representation 3.85 (19) .72
Coordination 4.30 (2) .63
Work Unit Monitoring 4.05 (9/10) .65
Program Evaluation 3.87 (17) .73
Communication 4.32 (1) .61
Interpersonal 4.11 (8) .64
Sensitivity
Leadership 4.25 (3) .61
Flexibility 4.12 (6/7) .62
Action Orientation 4.20 (4) .64
Results Focus 4.15 (5) .64
Broad Perspective 3.92 (16) .73
Strategic View 4.01 (14) .68
Environmental 3.96 (15) .79
Sensitivity

Table 2
The Correlations of Managerial Functions and
Effectiveness Characteristics on Performance Ratings

 Mean Std Correlat Correlation
 Dev ion with with
 Formal Variable
 Rating Quality
 Assessment

Pay 11.45 2.33 -.01 -.00
Tenure 3.83 1.11 09 .07
Age 4.41 1.19 .05 .00
Sex 1.29 .45 .01 .03
Work Unit Planning 15.90 4.61 .31 .48
Work Unit Guidance 16.74 4.35 .39 .56
Budgeting 13.69 4.84 .21 .28
Material Resources 14.08 4.78 .29 .34
 Administration
Personnel 16.21 4.61 .37 .46
Supervision 16.19 4.28 .39 .53
External Awareness 16.25 5.08 .38 .48
Interpretation 16.29 4.90 .35 .51
Representation 15.70 4.87 .38 .52
Coordination 17.91 4.90 .37 .57
Work Unit Monitoring 16.08 4.75 .38 .56
Program Evaluation 14.88 5.06 .35 .50
Communication 17.61 5.06 .31 .54
Interpersonal 16.00 4.75 .34 .55
 Sensitivity
Leadership 17.32 4.78 .36 .60
Flexibility 16.47 4.71 .37 .60
Action Orientation 16.91 5.02 .39 .62
Results Focus 16.74 4.90 .40 .56
Broad Perspective 15.23 5.02 .38 .52
Strategic View 15.93 4.96 .31 .45
Environmental 15.97 5.20 .38 .55
 Sensitivity

Note: All correlations statistical significant at P [greater than or
equal to] .05 except Pay, Age, and Sex (N=667)

Table 3
Logistic Regression Of Managerial Functions And
Effectiveness Characteristics (Low Pass-High Pass)

 FORMAL RATING QUALITY
 (n=315) ASSESSMENT (n=366)

Variable B Exp(B) B Exp(B)

Constant -3.72 *** -6.91 ***
Work Unit Planning -.03 .97 .14 1.15
Work Unit Guidance .13 1.13 .28 ** 1.32
Budgeting -.01 .99 -.06 .94
Material Resources -.05 .95 -.02 .98
 Administration
Personnel Management .09 1.10 -.24 .79
Supervision -.07 .93 -.06 .95
External Awareness .05 1.06 .00 1.00
Interpretation .10 1.11 .16 1.17
Representation .07 1.07 .07 1.07
Coordination -.04 .96 .22 * 1.24
Work Unit Monitoring .07 1.07 -.04 .97
Program Evaluation -.07 .93 -.05 .95
Communication -.08 .93 .02 1.02
Interpersonal Sensitivity -.01 .99 -.11 .90
Leadership -.03 .97 .06 1.06
Flexibility .00 1.00 -.10 .90
Action Orientation .12 1.13 .34 ** 1.41
Results Focus .00 1.00 -.02 .98
Broad Perspective .05 1.06 .32 ** 1.37
Strategic View .13 1.13 -.07 .93
Environmental -.07 .94 -.13 .88
 Sensitivity
Model X2 68.161 94.890

Adjusted R2 .10 .26
Correctly Predicted 86% 92%

Note: * p [greater than or equal to] .10,
** p [greater than or equal to] .05,
*** p [greater than or equal to] .01

Table 4:
Logistic Regression Of Managerial Functions And
Effectiveness Characteristics (Exceeds- Outstanding)

 FORMAL QUALITY
 RATING ASSESSMENT
 (n=265) (n=337)

Variable B Exp(B) B Exp(B)

Constant -5.63 *** -9.59 ***

Work Unit Planning -.08 .92 .02 1.02
Work Unit Guidance .12 ** 1.12 .11 * 1.11
Budgeting .04 1.04 .06 * 1.07
Material Resources .03 1.03 .03 1.03
 Administration
Personnel Management -.02 .98 -.09 .92
Supervision .11 1.11 .02 1.02
External Awareness .07 1.07 .04 1.05
Interpretation -.09 * .91 .00 1.00
Representation .02 1.03 .04 1.04
Coordination .16 *** 1.17 .00 1.00
Work Unit Monitoring -.02 .98 .06 1.06
Program Evaluation .07 1.08 -.10 * .90
Communication .06 1.06 .07 1.07
Interpersonal Sensitivity .01 1.01 .08 1.09
Leadership -.25 ** .78 .04 1.04
Flexibility .14 .87 .06 1.07
Action Orientation .07 1.08 .20 *** 1.22
Results Focus .20 *** 1.22 -.01 .99
Broad Perspective .17 ** 1.18 -.09 .91
Strategic View -.16 ** .85 -.02 .98
Environmental Sensitivity -.03 .97 .03 1.04
Model X2 85.387 176.13
 1
Adjusted R2 .12 .29%
Correctly Predicted 7 7
 4% 9%

NOTE: P [greater than or equal to] .10,
*** P [greater than or equal to] .01