Agency-based evaluation and organizational change in the human services

Moxley, David P

Abstract

In this paper, evaluation is placed within the context of the pressures for change contemporary human-service organizations experience in their policy and operating environments. The authors discuss the proactive role evaluation can play in increasing an agency's ability to successfully cope with and master change. The authors identify three types of agency response to change-revitalization, renaissance, and recovery-and they describe the different environmental circumstances that call for each response. They then create a framework of six approaches to agency-based evaluation and discuss how agencies can use these evaluation approaches to help themselves successfully respond to different conditions of change. A central aim of this paper is to explicate the linkages between environment context and type of change, agency response to change, and evaluation approach.

INCREASINGLY, HUMAN-SERVICE AGENCIES are under considerable pressure to perform in new and demanding ways. Funders, consumers, and regulators require agencies not only to improve the services they offer but also to develop or adopt innovative approaches to social, mental-health, and rehabilitation services (Light, 1998). These stakeholders demand that contemporary human-service agencies improve the services they offer and even push the boundaries of traditional practice (Adams & Nelson, 1995). Changes in public and policy expectations have made the contemporary environment of human services even more turbulent than in the past (Gunther & Hawkins, 1996). This is forcing public, quasipublic, and nonprofit entities to consider making significant incremental changes in how they conduct business and practice; these organizations may need to consider more substantial changes in how they conceive of human need, identify social problems, develop social programs, and ultimately define the very purpose of their agencies (Brinckerhoff, 1994).

Contemporary agencies can face different forms of change (Eadie, 1997) reflecting the multiple practice and policy environments within which the agencies function. To be effective, human-service agencies must be sensitive to an increasingly broad array of demands from these environments. This sensitivity, or organizational permeability, can be both an asset and a potential threat. As an asset, organizational permeability enables an agency to scan and monitor emerging changes to respond to these changes in a timely and proactive manner. As a limitation, organizational permeability can overwhelm agencies with information, new performance expectations, and multiple priorities. Some environments demand that agencies increase their performance, becoming more effective or more efficient in how they function. Other environments demand that agencies adopt new missions, new purposes, and even new service technologies. Still other environments require a wholesale shift in how agencies conduct business, demanding totally new directions and conceptions of how to practice. Using agency-based evaluation, for building the knowledge human-service organizations need to advance human-- service practice in the face of these environmental demands, is more important than ever.

While some high-impact, high-performance organizations are taking the lead in building evaluation into the delivery of their services, other agencies resist incorporating knowledge building through evaluation into their organizations. These agencies dismiss evaluation as a luxury or as an add-on, to be undertaken only when external regulators or purchasers require it (Kanuft, Bergen & Gray, 1991). However, given the complexity of environments within which contemporary agencies operate, these agencies do need continuous processes of evaluation that foster organizational knowledge building. Based on indigenous agency knowledge about what works and what constitutes good practice, the utility of such knowledge cannot be readily dismissed by human-service agencies seeking to respond effectively to the changing dynamics of the market for their services (Nonaka, 1998; Nonaka & Takeuchi, 1995; Helgesen, 1995). Evaluation and knowledge building go hand-in-hand. Both are stepping-stones to improved agency performance, and both are central to meeting the expectations recipients, funders, and regulators hold for high performance (Weiss, 1998).

The effort to meet continuously increasing expectations for high performance is what drives agencies to change (Osborne & Gaebler, 1992). Some of the changes they must implement are incremental and can be easily accomplished by reforms to existing organizational structures and procedures. Other changes are more sweeping and can only be accomplished by the reorganization and redirection of agency values, mission, and purpose. Some organizational changes are defensive responses, during times when systemic and substantial changes take place at a societal level such that the very nature, meaning, and purpose of existing social institutions are altered, and basic assumptions about human services are forced to be reexamined, redesigned, and restructured (Fullan, 1991 ).

An organization's ability to meet the demands of these different kinds and levels of change rests, in part, on how successful it is at making knowledge building through evaluation a core competence. From this perspective, evaluation is not an end in itself; rather it is an instrumental and strategic step in the organization's implementation of and adaptation to change. To make evaluation this kind of core competence, contemporary human-service agencies must develop knowledge-- building and utilization systems, of which evaluation is a key component, and make these systems an integral part of agency infrastructure and culture (Sparrow, 1998). It is not enough for an agency to evaluate selected programs or services on an ad hoc basis (Wholey, Hatry, & Newcomer, 1994). The agency must incorporate knowledge building through evaluation into how it approaches organizational development and change.

The aim of this paper is to link agency-based evaluation to the conditions of organizational change that many human-service agencies experience today and to highlight how evaluation can contribute to the ongoing development and increased effectiveness of an agency as it faces different kinds of change.

Purpose of Agency-Based Evaluation

Effective human-service practice today requires agencies to make evaluation a central aspect of their operations (Manela & Moxley, 1999). Too often leaders of human-service agencies assign evaluation a secondary status within their organizations. They fail to see its importance in managing the changes their agencies must make to be successful in new policy and practice environments. Given the turbulence of their organizational environments, most agencies can benefit from an ongoing process of evaluation that operates as an "early warning system." Such a system enables the organization to anticipate the changes it will face and plan how to respond to these changes effectively by implementing purposeful actions.

When agency leaders see evaluation as a strategic activity, they focus on gaining the knowledge they need to meet the demands of change successfully. Evaluation within the context of a human-service agency is the capacity to judge what the agency does; how it does it; and the consequences, outcomes, and effectiveness of its programs, procedures, and products. Effectiveness is measured against criteria formed by the principal values of the organization (Stufflebeam & Shinkfield, 1989). Agency-based evaluation requires a human-service organization to use these judgements to correct, advance, or develop its practice. At its core, agency-based evaluation is the assignment of value through the systematic appraisal of the worth or merit of some object to which the human-service organization commits tangible or intangible resources (Stufflebeam & Shinkfield, 1989). Agency-based evaluation can illuminate the need to change, highlight how to approach change, and identify the effectiveness of the organization's substantive responses to change. It can support accountability, improvement, enlightenment, and substantial realignment of organizational programs, services, or paradigms (Worthen, Sanders, & Fitzpatrick, 1997). From this perspective, an aim of agency-based evaluation is to help organizational actors make informed judgments about what the agency does, based on what the agency desires and according to its values.

One of the ways the pressure on agencies to engage in continuous change and development expresses itself is in the need to make agency-based evaluation a strategic part of agency operations. As Manela and Moxley (1999) suggest, today the most effective forms of agency-based evaluation enable an agency to engage in high-performance activities in human-service markets that are increasingly competitive. This type of evaluation helps agencies develop and use knowledge by fostering the creation of a knowledge base of effective indigenous agency practices that support successful services, ongoing improvement, and, ultimately, effective transformation (Grasso & Epstein, 1992). Such knowledge building through evaluation links to research and development and enables agencies to adopt or even create new models, practices, and services that are both viable and relevant (Rothman, 1980). The strength of agency-based evaluation rests on its ability to get the agency to use the information and knowledge evaluation produces (Thomas, 1984). It is in this way that agency-based evaluation helps the organization meet changing environmental and internal expectations, standards, and requirements in an effective manner (Rothman & Thomas, 1994).

The professional standards that guide the appraisal of evaluation practice recognize that evaluations must be meaningful and relevant to the needs of those who use them (Worthen et al., 1997). For this to occur, they must provide enough sound data, collected through systematic means, to enable agency leaders to judge the merit of an object that is important enough to the agency to subject it to evaluative scrutiny. The evaluation must ultimately produce knowledge that is used to advance the practice of the agency (Paine, Bellamy, & Wilcox, 1984), but any evaluation activity an agency undertakes should contribute to the development of both its practices and its evaluation competencies. Evaluation for too long has been seen as a discrete function, limited in scope and time frame. The kind of agency-based evaluation being proposed here must be deeply rooted in the strategy of the human-service organization and, as a consequence, it must become a part of ongoing agency operations.

Evaluation within the Three Forms of Organizational Change

The value of agency-based evaluation lies in its utility. In this paper, we focus on those aspects of the utility of agency-based evaluation that rest on the extent to which evaluation helps the agency master change successfully. The response of human-service agencies to pressures for organizational change, like that of their counterparts in business, industry, and manufacturing, can take three forms. We can label one "revitalization" (McWhinney, 1997), another "renaissance" (McWhinney, 1997), and the third "recovery" (Fullan, 1991). All three are legitimate responses to the need for different kinds of change; each is suitable under the appropriate conditions.

Revitalization

This response to change is most often seen in contemporary approaches to organizational development, program management, and quality improvement (Cohen & Brand, 1993). It is most appropriate when reforms or revisions within an organization's existing structure and practices are sufficient to handle change. Revitalization is appropriate when principal agency actors, typically those in the upper tiers of management and at the strategic apex of the organization, are comfortable with and committed to the agency's existing framework of practice and operations (McWhinney, 1997). These actors have a vision of the organization, which may or may not be equally and enthusiastically shared by members of the lower echelon of the agency. Nonetheless, the vision and values of upper management remain intact and are used to advance the substantive outcomes that guide and direct how the agency achieves its organizational and performance objectives (Dykstra, 1995). Management's vision and values also define and specify the ideal circumstances under which the agency is to operate (Brinckerhoff, 1994; Nanus, 1992).

When agency performance is challenged and changes must be made, but the essential structure and function of the agency are not called into question, revitalization is the appropriate response. In revitalization, the scope of change undertaken by the agency is rather narrow, and changes are reformist rather than radical in nature. These kinds of changes do not require pervasive modification of the agency and of how it conducts business (Juran, 1992).

The demand for revitalization may come from internal or external sources. The organization's failure to satisfy its own ideals and achieve the desired outcomes it specifies for itself can prompt staff, supervisors, or consumers to become critical of agency's performance and seek changes in agency practice. The impetus for reform also can come from demands external agents place on the agency. These agents hold the agency responsible for meeting their expectations for performance.

Evaluation efforts undertaken during periods of revitalization usually fit comfortably within current agency structures. A program of quality improvement that incorporates an evaluative approach may emerge in response to critiques of agency performance by organizational actors (Gunther & Hawkins, 1996). External funders, regulators, or advocates may become dissatisfied or otherwise critical of agency performance, because they want more and better service or outcomes of higher magnitude or durability (Dykstra, 1995). External stakeholders may have their own perspectives on extending and improving existing services, and these perspectives may or may not be consistent with the perspectives of internal actors. Whatever the source of pressure for change, when the level of change necessary for the agency to respond is achievable within the context of the ideals, values, mission, organizational structure, and procedures of the agency as it is currently constituted, revitalization is the appropriate response.

The agency sets out to achieve revitalization through campaigns of program development, quality improvement, performance and productivity enhancements, and the introduction of innovations that are consistent with the cultural framework and content of the organization. These change projects are oriented to the reform of current organizational structures, procedures, and practices. They require the upper echelon of the agency to legitimate the necessary changes and rally organizational members around the need for change (McWhinney, 1997). In well-integrated organizations, agency staff will follow management's lead, accept the idea of reform, and actually reform the procedures, practices, and programs they carry out in their everyday work at the agency.

The role of evaluation within revitalization is to give a platform and a voice to the internal and external actors seeking to reform agency operations. Evaluation helps them articulate their needs and concerns. It provides these actors with information about the issues that they want to address, and it helps them judge the value of existing agency operations within the current framework of agency practice (Stufflebeam & Shinkfield, 1989). By doing these things, evaluation helps them identify what and how the agency needs to change to achieve the ideals and desired outcomes the agency identifies as important.

The agency's organizational vision, the ideals that flow from this vision, and the desired outcomes it seeks to achieve are the reference points and the rationale for the revitalization of the agency (Kennedy, 1991 ). The level of certainty agency actors assign to this vision gives it the power to move the agency in the direction of revitalization. Revitalization enables the agency to think of change in a segmented and incremental manner. It enables agency actors to think clearly about what and how to change without forcing them to question the underlying cultural, organizational, and programmatic framework of the agency. However, revitalization can be a dysfunctional form of change when it fails to recognize that escalating demands in the agency's environment may require the agency to expend considerable energy, employ considerable ingenuity, and implement extensive innovations (McWhinney, 1997). Simple reforms of existing programs and procedures may not be enough to satisfy the demand for significant changes, and, typically, revitalization is not an effective response to the kinds of changes required during high levels of environmental turbulence.

Renaissance

This form of change emerges when agency members at different levels perceive a loss of meaning within their organization. Members "sense" that the vision that empowers their agency and energizes their practices is eroding and is no longer relevant to the external or internal worlds of the agency (Handy, 1994). As the forces supporting organizational continuity begin to weaken and collapse, there is a growing sense of disenchantment with what the agency stands for, its direction, and what it seeks to achieve through the provision of human services. Yet, simultaneously, there may be considerable fear of change among agency members (Hirschhorn & Gilmore, 1993). The agency, as it had been conceived and constituted, is dying. Unless the agency transforms to a new and more vital form, the agency will cease to exist (Handy, 1995).

Renaissance is the form of change that comes into play when the agency's culture is in decline and, as Schein (1992) observes, it can no longer offer meaning to its members that is of sufficient relevance or importance to enable them to find unity of purpose. The loss of motivation and energy among agency members is an indicator of the need for renaissance. When agency members begin to think "our current organization cannot hold us together any longer," it is time for the agency to transform (Hirschhorn, 1997). The agency may try to implement efforts at revitalization, but they are not enough to foster transformation, because the core of the agency, its values, vision, and mission no longer capture the imagination or supply the unity to keep the agency together (McWhinney, 1997).

The crisis brought about by the loss of meaning is devastating because it undermines the criteria on which the agency bases choices about desired outcomes (McWhinney, 1997). The loss of unity within the agency can lead to depression, heightened levels of stress, and turnover among agency staff. Agency leadership is no longer able or is not motivated to generate enough meaning and purpose to engage the agency as a collective (Hirschhorn, 1997). Also, when the agency tries to initiate revitalization efforts, it discovers that they only create more problems than they solve.

A renaissance approach to change requires transcendence over uncertainty and can lead to the recreation of the basic framework of the agency (Handy, 1995). Whereas revitalization presupposes certainty about ideals, direction, and outcomes, renaissance presupposes uncertainty, anxiety, and ambiguity (Handy, 1994). The members of an agency that is in disarray must typically descend before they can emerge to find new meaning and direction. While revitalization legitimizes a segmented approach to change, renaissance requires a whole-systems approach (Senge, 1990).

Evaluation within a renaissance approach to change can help organizational members begin the search for meaning (Maehr & Midgley, 1996). Paradoxically, evaluation may initially help destabilize the agency by enabling agency members to question the meaning of existing agency structures and functions. Evaluation may raise questions about the legitimacy of existing agency values, the efficacy of current agency programs, and the functionality of current agency practices. In all of these ways, evaluation may spur agency staff to question and contest the current structure of the agency. Taking issue with and contesting the current form and activities of the agency within the renaissance approach is more vigorous and of more consequence than the critiquing that takes place during revitalization.

When agency staff embark on the transformation of their agency, evaluation can serve the process of emergence by illuminating new directions the agency can take, highlighting new practices and models of service, and orienting staff to new responses to operational and programmatic challenges. Evaluation also can support emergence by helping the agency identify effective ways to replace the old with the new. Here, evaluation can facilitate the process of replacement through the appraisal of implementation procedures and the development of new organizational practices, operations, and programs.

Recovery

When the underlying structure of the society in which an agency exists undergoes a dramatic redistribution of power and resources, human-service agencies, along with other social institutions, must consider their survival (Fox & Miller, 1995). Substantial and, perhaps, revolutionary changes in the distribution of power and resources are usually accompanied by changes in underlying social values and a reconsideration of basic questions about the nature, role, and organization of social institutions (Schon, 1970). Such changes usually have cataclysmic implications for all organizations in a society, human-service agencies among them.

In the wake of change of this magnitude, agencies must adapt to a new set of social values, a new distribution of power and influence, new formulations of social need, and new priorities for meeting those needs. They may find themselves off balance and without a clear position within the new or emergent environment. Some human-service agencies will be so grounded in the former social structure and its core values and procedures that they will be unable to respond effectively to these social changes. It is unlikely that these agencies will survive. Other agencies will be identified as part of the problem that brought about the revolution, and their demise will be seen as a necessary outcome of the successful alteration of the environment. It is unlikely that these agencies will survive. Agencies that cannot adapt or are not allowed to change should try to discontinue their services with some warning, close their operations in an orderly manner, and ensure that their clients find other sources of help.

Unfortunately, during revolutionary changes, the demise of existing organizations is not always orderly, and the disorderly closing of human-service agencies can have severe negative consequences for both agency staff and recipients. At such times, a human-service agency's best response is to try to minimize the damage.

Not all organizations are swept away by revolutionary changes, and for the agencies that remain, survival during and immediately after revolutionary changes is of paramount importance. Some organizations that supported the revolution may be catapulted to positions of prominence and power. Other organizations may be repositories of needed expertise. The need for their services may span system-wide revolutionary changes. If human-service agencies are to survive during and after a revolution, they must align their values and mission with the new structure of social values, and they must adapt to a vision that conceives of and organizes human services in new ways (Adams & Nelson, 1995).

In other words, the agency will find itself in a state of recovery. The agency will not necessarily adapt immediately to the new environment, but, more likely, it will struggle to make active accommodations that help it realign what it does and how it does it. Change here involves a conversion to new configurations of values and related policies. In part, an agency may work to justify its continued relevance, or it may abandon what it had done in the past and seek to reposition its internal culture, operations, and practices to align them with the new and vastly different requirements revolutionary change introduces. While both revitalization and renaissance are stimulated by changes in the expectations among external stakeholders for modifications within existing organizational forms and configurations, a new environment and new forms of policy induce the need for recovery. The essence of recovery lies in the search for balance, position, and survival amid the emergence of new and often unfamiliar organizational forms and configurations.

Evaluation can help an agency to achieve recovery by enabling it to identify and establish the ways it has embodied and acted on values that support and are supported by the revolution. In addition, evaluation can help an agency demonstrate that it provides essential services, and that the expertise for providing these services is not generally available outside the agency, thereby defining a relevant role for the agency in the new configuration. Evaluation within this form of response to change focuses on the context in which the agency exists, how it gains knowledge about how to position itself, what needs it must address, how it defines legitimate problems or issues, and how it engages in the process of service delivery. Unlike revitalization, where the organization faces a rather certain set of contingencies, and unlike renaissance, which requires the agency to deal with uncertainty, recovery demands that the agency deal with an assault on the relevance and appropriateness of its values, mission, and technologies and with an imminent threat to its continuation.

linkage of Recovery, Renaissance and Revitalization

Recovery during revolutionary change often requires reactive and defensive responses designed to give an agency time and space to sort out the implications of the social changes that have taken place around it. The agency needs time to adjust to new social values and the emerging configuration of social institutions. Once the dust from the revolution has settled, and the environment in which the agency must function is more stable, the agency can begin to undertake its own renaissance.

Successful renaissance is important after recovery in the wake of revolutionary change, and it is important when an agency has stagnated and is perceived internally to have lost relevance, meaning, and unity (McWhinney, 1997). Ideally, renaissance can result in a newly energized and rededicated agency, one with a unified core of values, strong leadership and administrative management, an integrated and committed staff, and a commitment to identify and implement a pattern of best practices and increased benefits for the recipients of agency services. After an agency has successfully made the transition to a new organizational form and framework, it can continue the process of change and improvement through revitalization. The three adaptations to change we have described-recovery during revolution, renaissance, and revitalization--each are appropriate in different circumstances. However, they all interlock to support an organization as it deals with varying kinds and levels of change.

Conceptual Framework of Evaluation in Organizational Change

Table 1 presents a conceptual framework that summarizes the contributions agency-based evaluation can make to the facilitation of organizational change in the human services. This framework incorporates two major dimensions. The first offers the three forms of change we have discussed previously. Revitalization is a preferred form of change in organizational conditions of certainty. Renaissance is a preferred form of change in organizational conditions of uncertainty. Recovery is a preferred form of change when societal conditions call the very nature and configuration of social institutions into question.

The second dimension of our framework assigns two major roles to evaluation. Examination/reflection relates to the role of evaluation in scrutinizing existing agency arrangements, objects, and operations. Implementation/action refers to the role of evaluation in helping the agency take the actions it deems necessary to address the challenges inherent in the changes it faces. The resulting framework yields six agency-- based evaluation alternatives, each of which offers specific responses to different aspects and contexts of change:

1. Critiquing. The agency's competence at evaluation is used to identify improvement issues, concerns, needs, and/or challenges that demand attention in the name of quality, performance, and/or effectiveness.

2. Extending and improving. This involves the direct involvement of evaluation in hands-on implementation activities that make a material but incremental improvement in those objects, activities, or operations that currently exist.

3. Contesting. The evaluation competence of the agency is used to help it challenge what exists, to debunk what exists, and to create a sense that the existing object needs to be replaced to ensure the viability, relevance, and/or vitality of the agency.

4. Illuminating and replacing. Evaluation brings data, information, and knowledge into the design and development of new alternatives and plays an important role in helping the agency replace existing objects with new, more viable, more relevant ones.

5. Converting. Evaluation is used to help an agency identify and adjust to new configurations of values and new organizational forms, and identify its role in the sweeping changes brought about by social revolution.

6. Adapting and positioning. Evaluation uncovers data to justify the continued existence of the agency by proving that it provides an essential set of services that embody and promote current social values and meet essential human needs.

Facilitating Organizational Change through Methodological Pluralism

This framework suggests that there is no one "right" methodological approach for evaluation to follow. Indeed, contemporary evaluation theory and practice have evolved beyond a strict prescriptive methodology to an appreciation of the many different forms of inquiry and investigation that human-service agencies can incorporate into the evaluation enterprise. This move from methodological prescription to investigative pluralism is a healthy one. It favors a flexible, eclectic stance toward evaluation that can systematically address client needs, even when those needs change during the course of an evaluation project (Maddus, Scriven, & Stufflebeam, 1983). Such an approach is especially useful when it is necessary for an agency's evaluation system to adapt quickly to contingencies and unexpected events that arise during the process of change. A flexible, eclectic orientation to evaluation method offers the evaluator and the agency opportunities to match evaluation projects to the client information and knowledge needs change creates.

Within the evaluation profession itself, there has been a long debate concerning the relative value of positivist and naturalistic methods, with advocates on each side of the debate arguing for the legitimacy and dominance of their own methodological perspective. However, the emergence of investigative pluralism within agency-based evaluation indicates the need for the differential application of method to the change situation facing an agency. Multimethod approaches integrate both structured and unstructured methodologies, offering opportunities to blend quantitative and qualitative techniques into the same project, thereby creating databases that feature and link both forms of data. On a continuum of investigative approaches ranging from highly structured quantitative approaches to highly unstructured qualitative approaches, multimethod approaches are a middle-range option. This blend of structured and unstructured methods is accompanied by a combination of normative and idiographic approaches to the analysis and interpretation of data. Despite the flexibility inherent in the multimethod approach, "nuts and bolts" decisions about the implementation of investigative methods are still guided by meta-evaluation principles and criteria that require accuracy, quality, and relevance during the critical steps of data collection, analysis, and interpretation.

Methodological pluralism also can incorporate diversification of perspective into agency-based evaluation. Agency-based evaluation is not and does not have to be a top-down enterprise (McLagan & Nel, 1997). It can create participatory opportunities that democratize the design of evaluation, its implementation, and the analysis and use of the information that evaluation produces (Sclove, 1995). Action research methods (Argyris, 1980), participatory research models (Stringer, 1996), and empowerment-based evaluation (Fetterman, 1996) all help lower-level actors gain knowledge through action and offer agencies opportunities to broaden participation in evaluation projects. They also enable agencies to diversify the responsibility for the utilization of evaluation findings and extend the involvement of agency personnel into the preparation for and response to change.

Organizational actors who are logical users of evaluation to meet the demands of change but who, nonetheless, are kept out of the organizational venues in which utilization decisions are made can become involved through participatory approaches to evaluation (Fetterman, 1996). Through the use of methodological pluralism, their perspectives and expectations about change and evaluation and their information or knowledge needs can become part of agency-based evaluation.

Methodological pluralism incorporates three principal properties in relationship to agency-based evaluation. First, there is the perspective and organizational positions of those who define, frame, and design the evaluation project. This property recognizes that most formal human-service agencies, like other modern or postmodern organizations, embody a collection of perspectives, and these perspectives are conditioned by the position of actors within the agency's social system (Berquist, 1993). Second, there is the scope and extent of participation of agency actors in the evaluation, including its design, planning, implementation, and utilization (Stringer, 1996). It is this property that underscores how involvement in evaluation and the actions participants take during evaluation can shape knowledge of the object under scrutiny. Third, there is the actual approach to data capturing and analysis the evaluation project takes, as expressed through the design and use of instruments, measures, and data sources, and through the organization, reduction, and interpretation of evaluative data. Here, methodological pluralism suggests that there are numerous ways to capture and analyze data.

An eclectic approach to method provides an opportunity for a great deal of creativity, freedom, and independence for those involved in agency-based evaluation, conditions essential to an effective agency response to change. While pluralism is the most appropriate methodological approach for agencybased evaluation, method does not drive evaluation; rather, the agency's need for evaluation, formed by the pressure for change the agency faces, drives the selection of method and its use. Diversification of perspective, participation, and inquiry are essential to ensure that utility is woven into all aspects of the evaluation, remembering that utility emerges from the ability of the evaluation to help the agency meet the organizational requirements, expectations, and opportunities change creates.

Using Evaluation: An Agency Example

Background

The Workplace (a pseudonym) was founded in 1968 as one of the earliest efforts to facilitate the work performance of persons with developmental disabilities. The mission of the agency was to foster the vocational development of persons with developmental disabilities, as they moved into community residential alternatives from state developmental centers. The agency adopted what is now known as a facility approach to vocational development. In this approach, the agency offers sheltered work opportunities at a facility that receives work contracts from government and business. "Workers" perform various assembly duties at their workstations, where they are supervised by someone who helps them acquire the necessary assembly skills, sustain their motivation, and perform at adequate levels of production. Behavioral specialists are available to help each worker address behavioral concerns that emerge at the workstations or at the facility.

Need for Revitalization

By the early 1970s, the Workplace was recognized as an exemplary habilitation facility by the state department of mental retardation and developmental disabilities. It also received considerable media attention for facilitating community living by persons with developmental disabilities.

Yet, by the late 1970s, there was growing dissatisfaction with the performance of the Workplace. Deinstitutionalization was escalating in the state, and the emergence of county-based developmental disabilities service systems introduced new expectations about the habilitation of persons with developmental disabilities. Increasingly, both state and local funders wanted the Workplace to offer its vocational development services in community settings. The force of these expectations was strengthened by the voices of advocates for people with developmental disabilities. They called for normalizing of vocational alternatives, so that people with developmental disabilities could become more active in their communities. The agency's evaluation system enabled the Workplace to identify and give voice to these criticisms, highlighting the issues these various constituencies articulated and focusing on their concerns about the performance of the agency. Thus, the evaluation system helped the agency critique its operations. We have identified this stage of critiquing as a fundamental step toward embarking on revitalization.

The Workplace did not see the expectations and pressures for change as requiring them to implement a new paradigm of habilitation and, therefore, overlooked the revolutionary changes coming to the field of developmental disabilities. Service personnel and administrators agreed that the agency needed to reform and extend its model of habilitation into various industrial sites located in the community and to give up its facility-based approach to vocational development. The agency was able to replicate its bench work assembly in various manufacturing settings. A work supervisor accompanied a small group of people with developmental disabilities into the industrial work setting and they completed the work at the business site that formerly had been sent for assembly at the Workplace facility. A behavioral specialist was assigned to the enclave to address behavioral and work performance issues that emerged among clients on the job.

Advocates and funders endorsed this change to the Workplace. The evaluation system demonstrated that people with developmental disabilities were involved in learning and using public transportation, they were more productive on the job in a real work setting than they had been in the agency's facility, and they were learning new skills. However, Workplace personnel were increasingly dissatisfied with this approach to habilitation. They observed that the so-called normal workers often ridiculed the members of the enclave, and there was very little positive interaction between the regular employees and enclave members. Staff obtained assistance from the agency evaluator to document these issues, determine how serious they were, and identify their negative consequences. The evaluation of six industrial work settings employing 72 persons with developmental disabilities demonstrated that the enclave members experienced considerable stigma and had to endure negative social reactions from other employees.

Recovery and the Call for Renaissance

The 1980s saw a commitment on the part of the agency to quality improvement. Agency personnel focused an increasing amount of attention on refining and retooling their industrial approach to the habilitation of persons with developmental disabilities. They experimented with various approaches to facilitate the social participation of enclave members in the larger social system of the industrial locations in which they were placed. However, the field of developmental disabilities was changing quite dramatically, as old frameworks of deinstitutionalization gave way and new approaches to community support of people with developmental disabilities evolved.

By the late 1980s, the developmental disabilities movement was advocating inclusive community living and service approaches that sought to tailor employment, educational, and community-involvement opportunities to the needs of each individual, instead of applying uniform interventions to groups of people. The idea that persons with developmental disabilities would only be passive members in communities gave way to the idea that they should be active participants in and members of their communities. Advocates of supported employment derided work enclaves and sheltered employment. They demonstrated how a person could be individually supported in an employment situation of their own choosing. Person-centered planning was identified as a preferred approach to identifying a person's work preferences and designing the supports they desired in the work setting. Natural supports, in which friends, co-workers, and family members supported a person with a developmental disability in their employment, were preferred as an alternative to supports offered by specialized human-service personnel.

The Workplace did not keep up with these changes in its environment. Its approach, which had been groundbreaking when it was first developed, was now considered passe and was no longer among the best practices endorsed by the field of developmental disabilities. Because the agency-based evaluation system was focused only on revitalizing the agency's existing services, it failed to identify emerging practices, examine new practice models, and extend the agency's knowledge base to include what were currently considered to be best practices in the employment of persons with developmental disabilities. The agency was caught off balance and failed to secure its position within a policy environment that endorsed new values.

As older staff left the agency, the new employees brought this emergent knowledge into the agency When they saw that the agency did not make use of these best practices, their personal discomfort and dissatisfaction grew and led to internal discord in the agency.

Agency personnel began to contest the use of enclave models of habilitation. County and state funders considered withdrawing key contracts in the early 1990s. Turnover among agency personnel began to escalate, and remaining staff members felt that the agency's approach to habilitation did not hold much meaning for them.

A new executive director came to the agency in 1993. She had previously led an agency that worked with industrial enclaves, but she also recognized the need to move on. The mandate of the board was to see the agency through a successful renaissance. The executive director gave the agency's evaluation system a central role in this renaissance. The system was used to scrutinize and monitor the status of the agency's practice and document ways external stakeholders viewed the agency and its practice. The evaluator's efforts to scrutinize the current practice base of the agency and to appraise the agency's use of state of the art practices had broad-based support and gained the participation of both agency staff and external constituencies.

Early in the agency's change process, the evaluation contributed strategies for how the Workplace could negotiate new policy values and expectations and find a new position within the community as a provider of supported and inclusive employment. Subsequently, the evaluation produced a renaissance agenda that outlined the substantive changes the agency could make in specific programs so as to reconstitute itself, and again become relevant, vital, and viable in an environment in which consumers, advocates, funders, and regulators held new expectations and had established new standards.

One of the important tools the agency produced was a structured self-evaluation instrument that helped the agency profile the extent to which it was incorporating state-of-the-art practices. By taking issue with and "contesting" the efficacy of what the agency was doing, agency staff and external constituencies pushed the agency to develop a new vision and a new ideal of agency performance. These, in turn, led to the production and implementation of a new set of desired outcomes, which were made explicit by the agency's creation of an instrument to monitor the extent to which it was implementing state-of-theart practices.

By 1996, the agency negotiated recovery successfully, and it was making progress towards renaissance. The evaluation system took on an increasingly important role in "illuminating" best practices, promoting their adoption by the agency, and supporting the development of how best practices could be installed to supplant or replace old program models. The evaluation system was central in codifying the best-practice knowledge base and in creating tools and information to guide implementation efforts. The evaluation system was instrumental in assessing the integrity of various programs. Integrity here was defined as the "extent to which a program adopted and implemented inclusive, person-centered procedures of habilitation practice." The agency's reporting system was modified to serve a monitoring and feedback function. It illuminated the extent to which agency programs successfully adopted and implemented what the agency now knew were best practices. The agency regularly carried out individual program case studies to illuminate the barriers and issues each program faced in adopting and implementing best practices.

While this internal evaluation was being carried out, the evaluation system also monitored the attitudes of external stakeholders. In this effort, the evaluation methodology integrated surveys of large samples, interviews with selected key informants, and focus groups to identify how the changes in the practice base of the agency were perceived among advocates, regulators, and funders. This multimethod approach fed a rich stream of information to the agency about additional changes that were needed in the agency's practice base.

The consolidation of the changes that were taking place at the agency led to a new paradigm of habilitation, which was endorsed and used by the Workplace and which was a milestone in its renaissance. By 1998, the agency had a new vision in place, supported by a new set of ideals and a desired set of outcomes. In addition, the agency had developed new competencies in information management, program development, and outcome funding and management. A striking result of this whole process was the realization on the part of key agency personnel and leaders that all the changes the agency had successfully negotiated were instigated by threats to the agency's survival that emerged during the early 1990s. This led to a process of change that began by being reactive and gained momentum as it became more and more proactive, leading the agency to take steps to recreate itself. In this process of change, during which the agency recreated itself, the agency learned that by using evaluation as a core competency to integrate revitalization and renaissance it could become the agent of its own self-renewal.

Today, the biggest challenge facing the agency is to integrate self-renewal into its organizational culture, so the organization will continue to use evaluation to perpetually learn and appropriately use recovery, revitalization, and renaissance to keep the agency at the leading edge of its field. The centrality of evaluation in this case required organizational leadership to prioritize evaluation and its use as a tool to negotiate the change process. As the agency learned, evaluation is not a one-shot event, undertaken when the agency finds itself in new funding situations or in distress. Ideally, as the content of this paper underscores, evaluation is a continuous and ongoing process in which there is a commitment to making evaluation part of the implementation of the agency's strategic plan for building organizational effectiveness and competence.

Conclusion

The theme of this paper underscores the central role evaluation can play in contemporary human-- service agencies. It is not uncommon for agency leaders to feel overwhelmed by the need to keep up with the escalating pace and complexity of the changes they face on a daily basis (Handy, 1994). It is especially difficult for an agency to keep up with changes that emerge from expectations, values, and standards that are themselves influenced by dynamics in the greater society, over which the agency has little or no control and which are in a constant state of flux. In the environment of contemporary human services, change is now a way of life, and, as we have discussed here, evaluation and the collection, creation, analysis, and application of knowledge is one of the best ways to deal with the demands change places on agencies.

Some may argue that the pressure to engage in evaluation really comes from a general attitude of societal mistrust. From this perspective, the motives and work of nonprofit, quasipublic, and public human-- service organizations are held suspect by large segments of the polity, and, therefore, evaluation is introduced as one way to discipline and control organizations that engage in social welfare (Meenaghan & Kilty, 1994).

We do not dispute that this view of evaluation may describe some of the impetus and motivation to evaluate human-service agencies and the programs and services they provide. However, as we point out, evaluation need not just oversee and constrain agency activities. It can be a much more proactive and relevant activity, which can help protect an agency during times of crisis, complement an agency's commitment to its own development, and advance and renew the purpose, operations, and technologies of an agency. Given the pace of change in the organizational environment of the human services, many, if not all, human-service agencies are threatened with the possibility that they may not be able to keep up with the demands, expectations, and emerging requirements of a society that increasingly wants more for less, i.e., higher levels of performance achieved with fewer resources. Agency-based evaluation is one system that can help agencies counter a possible loss of relevance, vitality, and viability. The promise of agency-based evaluation lies in its ability to guide an agency along a path of self-renewal. This is what infuses agency-based evaluation with meaning and relevance in contemporary human services.

References

Adams, P, & Nelson, K. (Eds.) (1995). Reinventing human services: Community- and family-centered practice. Hawthorne, NY: Aldine De Gruyter.

Argyris, C. (1980). Inner contradictions of rigorous research. New York: Academic Press.

Bergquist, W. (1993). The postmodern organization: Mastering the art of irreversible change. San Francisco: Jossey-Bass. Brinckerhoff, P C. (1994). Mission-based management: Leading your

not-for-profit into the 21st century. Dillon, CO: Alpine Guild. Cohen, S., & Brand, R. (1993). Total quality management in government. San Francisco: Jossey-Bass.

Dykstra, A. (1995). Outcome management: Achieving outcomes for people with disabilities. New Lenox, IL: High Tide Press. Eadie, D. (1997). Changing by design: A practical approach to

leading innovation in nonprofit organizations. San Francisco: Jossey-Bass.

Fetterman, D. M. (1996). Empowerment evaluation: An introduction to theory and practice. In D. M. Fetterman, S. Kaftarian, & A. Wandersman (Eds.). Empowerment evaluation: Knowledge and tools for self assessment and accountability (pp. 3-46). Thousand Oaks, CA: Sage,

Fox, C. J., & Miller, H. T (1995). Postmodern public administration. Newbury Park, CA: Sage.

Fullan, M. G. (1991). The new meaning of educational change (2nd ed.). New York: Teachers College Press.

Grasso, A. J., & Epstein, I. (Eds.) (1992). Research utilization in the social services. New York: Haworth Press.

Gunther, J., & Hawkins, F (1996). Total quality management in human service organizations. New York: Springer.

Handy, C. (1994). Beyond paradox. Boston: Harvard Business School Press.

Handy, C. (1995). Beyond certainty. Boston: Harvard Business School Press.

Helgesen, S. (1995). The web of inclusion. New York: Doubleday. Hirschhorn, L. ( 1997). Reworking authority Leading and following in the post-modern organization. Cambridge, MA: MIT Press.

Hirschhorn, L., & Gilmore, T. N. (1993). The psychodynamics of a cultural change: Learnings from a factory. In L. Hirschhorn & C. Barnett (Eds.). The psychodynamics of organizations. Philadelphia: Temple University Press.

Juran, J. M. (1992). Juran on quality by design. New York: Free Press.

Kanuft, E., Berger, R., & Gray, S. ( 1991 ). Pro files of excellence: Achieving success in the nonprofit sector. San Francisco: jossey-Bass.

Kennedy, L. W (1991). Quality management in the nonprofit world. San Francisco:Jossey-Bass.

Light, P C. (1998). Sustaining innovation: Creating nonprofit and government organizations that innovate naturally San Francisco: Jossey-Bass.

Maddus, G. F., Scriven, M., & Stufflebeam, D. L. (1983). Evaluation models: Viewpoints on educational and human services evaluation. Boston: Kluwer.

Maehr, M. L., & Midgley, C. (1996). Transforming school cultures. Boulder CO: Westview.

Manela, R., & Moxley, D. (1999). The new pillars of agency-based evaluation. Applied Behavioral Science Review, 7(1), 1-17.

McLagan, P, & Nel, C. (1997). The age of participation: New governance for the workplace and the world. San Francisco: Berrett-Koehler.

McWhinney W (1997). Paths of change: Strategic choices for organizations and society (Rev. ed.). Thousand Oaks, CA: Sage. Meenaghan, T., & Kilty, K. (1994). Policy analysis and research

technology: Political and ethical considerations. Chicago: Lyceum Books.

Nanus, B. (1992). Visionary leadership. San Francisco: Jossey-Bass. Nonaka, I. (1998). The knowledge creating company. In Harvard Business Review (Ed.), Harvard Business Review on knowledge management. Boston: Author.

Nonaka, L, & Takeuchi, H. (1995). The knowledge-creating company. New York: Oxford University Press.

Osborne, D., & Gaebler, T (1992). Reinventing government. Reading, MA: Addision-Wesley.

Paine, S. C., Bellamy, G., & Wilcox, B. (1984). Human services that work: From innovation to standard practice. Baltimore: Paul H. Brookes.

Rothman, J. (1980). Social R & D; Research and development in the human services. Englewood Cliffs, NJ: Prentice-Hall. Rothman, J., & Thomas, E. J. (Eds.). (1994). Intervention research:

Design and development for human service. New York: Haworth.

Schein, E. H. (1992). Organizational culture and leadership (2nd ed.). San Francisco: Jossey-Bass.

Schon, D. (1970). Beyond the stable state. New York: Norton. Sclove, R. E. (1995). Democracy and technology New York: Guilford.

Senge, P (1990). The fifth discipline: The art and practice of the learning organization. New York: Doubleday.

Sparrow, J. ( 1998). Knowledge In organizations: Access to thinking at work. Thousand Oaks, CA: Sage.

Stringer, E. T. (1996). Action research: A handbook for practitioners. Thousand Oaks, CA: Sage.

Stufflebeam, D., & Shinkfield, A. (1989). Systematic evaluation. Boston: Kluwer.

Thomas, E. J. (1984). Designing interventions for the helping professions. Beverly Hills, CA: Sage.

Weiss, C. H. (1998). Evaluation (2nd ed.). Englewood Cliffs, NJ: Prentice-Hall.

Wholey, J. S., Hatry, H. P., & Newcomer, K. E. (1994). Handbook of practical program evaluation. San Francisco: Jossey-Bass. Worthen, B. R., Sanders, J. R., & Fitzpatrick J[. L. (1997). Program

evaluation: Alternative approaches and practical guidelines. New York: Addison Wesley Longman.

David P. Moxley is professor, School of Social Work, Wayne State University, 321 Thompson Home, Detroit, MI, 48202. Roger W. Manela is a school social worker, Detroit Public Schools, and adjunct faculty, School of Social Work, Wayne State University, Detroit MI.

Original manuscript received: May 4, 1999 Accepted: November 4, 1999

Copyright Manticore Publishers May/Jun 2000
Provided by ProQuest Information and Learning Company. All rights Reserved