A user-friendly approach to program evaluation and effective community interventions for families at risk of homelessness.

Mulroy, Elizabeth A. ; Lauber, Helenann

Practitioners who administer social programs seek usable knowledge from academic researchers that encompasses the critical issues of the times and the problem of hands-on management (Schuman & Abramson, 2000). Practitioners in small nonprofit organizations in particular face a multitude of issues in the implementation of public sector grants and contracts that affect program management and evaluation. Particularly hard hit are prevention programs in local community-based nonprofit organizations that serve very low-income children and families (Weil, 2000).

The purpose of this article is to demonstrate how a Family Center's program evaluation--an example of coproduction that used logic modeling as an analytic framework--can help social work practitioners and local community workers evaluate their own programs. A logic model is a one-page "graphic representation of a program that describes the program's essential components and expected accomplishments and conveys the logical relationship between these components and their outcomes" (Conrad, Randolph, Kirby, & Bebout, 1999, p. 18).

As Wolch (1999) contended: "The real burden is on nonprofit agencies suddenly faced with rising demands for services, reduced public funding, and mandates to monitor clients and enforce sanctions including benefit terminations and evictions, on behalf of their partner the state" (p. 28). In the face of these developments, to what extent are program managers in community-based nonprofits capable of responding to local needs? This is a timely question for the evaluation of social work practice. First, at the turn of the 20th century, small community-based nonprofit organizations such as settlement houses historically served poor neighborhoods as mediating institutions to help immigrant newcomers move out of poverty (Jansson, 1994). In the new millennium, immigrant issues are controversial social policy concerns. Today, with privatization generating role shifts of government as funder and nonprofit as provider, we need to know more about how services are configured in low-income neighborhoods and how targeted beneficiaries use such services.

Second, social workers are in positions of responsibility for the development, administration, and evaluation of program initiatives that implement new federal and state social policies intended to reform public welfare, child welfare, and public housing (Mulroy & Lauber, 1999). Many of these programs are small and focus on the coordination of community-based services intended to increase resident and community empowerment. The philanthropic community, also invested in strengthening families and communities, encourages local projects to be comprehensive community initiatives that use collaborations and partnerships to achieve community-building goals (Annie E. Casey Foundation, 1997; Leventhal, Brooks-Gunn, & Kamerman, 1997; O'Connor, 1995).

Third, public and philanthropic funders expect program managers to perform these functions in a context of heightened accountability for program efficiency and effectiveness (Forbes, 1998; Schalock, 2001). An important contextual factor is that in the era of privatization most program evaluations have a political context; the research questions may come from federal, state, or local government funding agencies, and findings are intended to provide feedback to legislators to inform their future resource allocation decisions (Bickman & Rog, 1998; Yegidis & Weinbach, 1996). The dilemma faced by many program managers in community-based nonprofit organizations is their lack of training in evaluation research with preferred experimental or quasi-experimental designs and control processes and limited budgets that prevent hiring consultants to carry out the evaluations. Many practitioners seek knowledge from evaluation research that can help them improve their programs, not just respond to the call for externally conducted outcome evaluations. To solve the practice dilemmas for service accountability, program improvement, and community-based solutions, a movement has emerged to identify "best practices" and in the process make program evaluation more usable for many audiences (Connell, Kubisch, Schorr, & Weiss, 1995; Fetterman, Kaftarian, & Wandersman, 1996; Marquart & Konrad, 1996; Quinn Patton, 1997; Resource Coalition of America, 1996). One commonality is the coproduction of the evaluation by evaluators and practitioners. The developmental approach expects that the evaluation process will result in improvement of internal program and agency operations--often referred to as organizational capacity--as the lessons learned become institutionalized (Quinn Patton).

Background: The Family Center

Several characteristics of Parents and Children Together (PACT), the Family Center, and the public housing project are relevant to professional practitioners, indigenous community workers, funders, and academics concerned with program planning, management, and evaluation of community-based programs. These include neighborhood characteristics, organizational structure, and program goals.

Neighborhood Characteristics

PACT and its programs were located on-site at Kuhio Park Terrace (KPT) public housing project in urban Honolulu, Hawai'i, a development of about 2,500 very low-income people, largely immigrants. The demographic profile of residents was similar to other public housing projects in large cities: 94 percent were people of color; 68 percent of families were headed by single parents; 80 percent received public assistance; and the average annual family income was $11,412 in a city where the median income for a family of four was $60,400. KPT, with two high-rise towers and low-rise garden apartments, was located close to the city's main highway in a densely populated neighborhood surrounded by single-family homes, apartment complexes, an elementary school, Honolulu Community College, and commercial firms. Other high-density public and subsidized housing complexes were in close proximity so that gang activity and turf wars were not uncommon. The neighborhood had an unemployment rate of 11.5 percent, whereas the state experienced a 5.7 percent unemployment rate as the national rate dipped to 4.1 percent (Mulroy & Lauber, 1999). Recreational opportunities and nonprofit social services agencies were resources in the neighborhood that was well served with public bus transportation to a nearby strip shopping mall just a few blocks away.

Organizational Structure

The Family Center was one of 15 programs of PACT, a large, private, community-based, nonprofit family services agency that targeted most of its programs in the KPT public housing development but had service sites across the state. Its programs were clustered into five service areas: (1) early childhood education, (2) mental health support, (3) community economic development, (4) child abuse prevention and treatment, and (5) domestic violence prevention and treatment. PACT's mission was to promote and support healthy individuals, families, and communities by creating opportunities for them to identify and address their own strengths, needs, and concerns and to realize their potential. PACT was an umbrella organization that pioneered the integration of community-based services and the partnership concept of service delivery. Founded in 1968, the agency's revenues were increasingly generated from public grants and contracts.

Program Goals

In 1995 PACT received a three-year demonstration grant from the U.S. Department of Health and Human Services [HHS] to help prevent homelessness among at-risk, very low-income families living in a large public housing development. The $218,000-a-year grant was administered through the Family Center to implement the social policy goal of "preventing homelessness and moving families to self-sufficiency." The definition of moving to self-sufficiency in federal terms meant moving to independence from government subsidies, such as Temporary Assistance to Needy Families (TANF) and public housing. The target population was current residents of KPT who were previously homeless or at risk of homelessness, primarily through eviction.

Family Center staff reframed the definition of self-sufficiency to mean movement toward independence. Such movement was expected to be different among their target population depending on life circumstance; would not happen swiftly, and would include backslides and steps forward. Staff's definition of preventing homelessness meant helping those at risk of homelessness stabilize their tenancy by remaining in the public housing complex in good standing or moving out of KPT in good standing to another apartment if that was their choice.

Potential program participants were believed to have multiple barriers to personal and material independence. They were less likely to move out of public housing in good standing or off TANF roles and into wage work than people with fewer barriers who were more easily served by traditional job training programs offered by private industry councils through community economic development programs (Strawn & Martinson, 2000).

At the inception of the grant, the program was designed to prevent homelessness through the provision of intensive and comprehensive support services to previously homeless families and families at risk of becoming homeless. By the end of the funding period, the center expected a majority of participants to have stable housing and move toward a job, and for the entire public housing community to experience an increase in civic pride and in resident participation. Staff included professionally trained managers and local community workers familiar with the multiple cultures and languages represented by resident groups. The budget provided a modest amount for program evaluation conducted by an external evaluator.

Method

Many government agencies that fund social programs require logic modeling in grantee program planning and evaluation. A logic model can be useful to practitioners and evaluators: It offers a clear conceptualization of a chain of events that, when developed at the beginning of a project, serves as a program planning tool with guideposts for program implementation (Yin, 1998); the conceptualization reflects the theory of program intervention (Mulroy & Shay, 1997; Weiss, 1995; Weiss, 2000); it is sufficiently detailed to guide formulation of the research design (Yin, 1998); and it can be used developmentally as a project typically shifts and changes as it moves forward in time (Alter & Murty, 1997).

HHS determined the purpose and boundaries of the evaluation of the KPT project. First, this was a study of the center's second year of operations only, setting the timeline to activities developed and outcomes attained up to that point. Second, it asked for innovation in programming and in evaluation design. It was made clear at the outset that a major purpose of the evaluation was to provide policymakers in Congress and in HHS, Office of Community Services, with lessons learned to help them make sound decisions for future homeless prevention programs. Knowing the purpose of the evaluation at the outset facilitated decision making about the type of information needed, the best way to measure the target variables, and the appropriate units of analysis (Logan & Royce, 2001).

Before the evaluation began, a user-friendly approach to program evaluation was put in place: HHS required all practitioners who applied for homelessness prevention funds to complete a logic model as the framework for their proposed program and to include it in their grant application. HHS also required grant recipients to refine and expand their logic models into real-world program plans. Among the materials provided to grantees was a one-page outline of a logic model that required the identification of clearly stated theories, principles, or assumptions that guided their activities and interventions; activities required to achieve the intended outcomes; anticipated outcomes defined as immediate and long-term, and the ultimate goal. With a small budget and a tight one-year time frame, the center contracted with a professor of management, community planning, and social policy at a local school of social work to conduct the external evaluation, with the promise of involving graduate students.

An action research approach was used because it was consistent with the evaluator's approach to knowledge building; afforded site-level analysis for an in-depth examination of the social context; was compatible with staffs interest in participation; and facilitated the use of multiple methods that best fit the research questions and the complexity of the context with the resources available (Greenwood & Levin, 1998; Knapp, 1996; Quinn Patton, 1997; Witkin, 1994). The evaluator had three front-end goals: (1) be a partner with staff by over time conducting work face-to-face and on-site; (2) improve the quality of data collection by creating a management information system, testing it out with staff, redesigning it as needed, then leaving it behind; and (3) improve the program's organizational capacity (Quinn Patton, 1997; Stevenson, Mitchell, & Florian, 1996; Yin, Kaftarian, & Jacobs, 1996).

Logic modeling was selected as a framework for analysis in an embedded single case study (Yin, 1994, 1998). Staff were familiar with logic modeling as a program planning tool. Therefore, its continued use had the potential to enhance staffs ability to think critically, that is, to define assumptions and line of reasoning among assumptions, activities, and outcomes that help link theory to practice (Alter & Murty, 1997). Logic modeling also provided a framework for conducting a process evaluation needed by legislators for their interest in replication and helped focus the design of the outcome evaluation to account for multilevel factors--individual, neighborhood, and societal (Conrad et al., 1999; Knapp, 1996). Finally, findings from the data could be compared with the original logic model in a pattern-matching mode to determine the viability of the original conceptualization (Yin, 1998).

Data were gathered from multiple and diverse sources: review of the center's case files, archival records, program documents, focus groups, interviews, participant observation, a "physical artifact" timeline (Yin, 1994), and housing authority tenant rent role records. New instruments for collecting data on program activities were codesigned with staff, pretested, and then revised for easier use by staff. Three well-attended formal focus groups were held with 17 highly involved program participants to gauge their participation in the center's activities. Group meetings, each facilitated by the evaluator and staffed by two graduate research assistants and not attended by staff, were taperecorded and transcribed (Krueger, 1994).

A physical artifact (Yin, 1994, 1998) Resident Participation Time Line (RPTL) was created in the final focus group as one source of data to determine residents' level and form of participation in the center's activities. A 10-foot-long roll of paper was taped to a wall in a familiar, comfortable activities room in the housing project and marked with the months and years of the center's operations. Residents were supplied with color-coded stickers, markers, and symbols and asked to identify when and through which activity they first entered the program, which services were used and when, and what critical events, turning points, or milestones were most important to them in their relationship with the center.

Multiple methods of data analysis were used (Bickman & Rog, 1998; Maxwell, 1998; Yin, 1998). Focus group data and the RPTL were analyzed using traditional focus group methods (Krueger, 1994). Three evaluators conducted an independent analysis of each focus group transcript, after which the evaluating team convened to discover common themes and patterns. Qualitative data were analyzed using an interactive model consisting of data reduction, data display, and conclusion drawing and verification (Huberman & Miles, 1994; Miles & Huberman, 1994). Conclusions were drawn only after all evaluators concurred on emerging themes and patterns. Quantitative methods were used to analyze tenant housing histories, rent payment schedules, and evictions. Verification and a control for evaluator biases were achieved by triangulating the multiple methods used to gather and analyze data and by incorporating informant feedback (Bickman & Rog, 1998; Guba & Lincoln, 1989; Heineman-Pieper, 1994; Maxwell; Quinn Patton, 1997; Tyson, 1995; Yin, 1994, 1998). Center staff were involved in the iterative process as data collectors and as a feedback loop for purposes of confirmability. Regularly scheduled meetings were held between evaluators and staff for this purpose, as well as to comanage the evaluation.

Findings

Four phases unfolded sequentially that informed the way Family Center staff understood and operated its demonstration program.

Phase 1: Logic Modeling as Program Planning

The first product of the evaluation was a revised logic model that streamlined and systematized the originally conceived plan into a road map through the complex community-based program (Figure 1). There is a logical flow from a belief structure to related interventions, to outcomes, and then to a goal. The model consisted of six assumptions or principles targeted to multiple systems--families, communities, neighborhoods, and institutions--that underlie five domains of program activities and interventions. These activities were intended to help participants achieve one or all of the four immediate outcomes as indicated by their circumstances. We hoped that residents could then attain the appropriate intermediate outcomes, and ultimately the long-term goal of stable housing and movement toward self-sufficiency.

[FIGURE 1 OMITTED]

It took several months of intensive discussion, analysis, and decision making between the practitioners and the evaluators (faculty member and graduate research assistants) to revise the logic model. In the end the number of principles proposed by staff was reduced from 28 to six (as shown in Figure 1). The program design was changed from 72 interventions to five domains of activities, and staff downsized the number of people it intended to serve in this demonstration project from all 2,500 public-housing residents to the 93 most at risk of homelessness. The logic model revision had three effects. First, it helped the staff scale back its own scope of work by better understanding what was realistic and doable by a very small staff in a time-limited demonstration project. Second, it helped the evaluators understand the complexity of the interventions, the organizational culture of the center, and the social environment of the public housing project. Third, it offered an opportunity for practitioners and evaluators to get to know each other and learn to work collaboratively and respectfully.

Phase 2: Conceptualize the Intervention

The revision process highlighted the need to clarify to staff, local leaders, and funders what the family support intervention "looked like" and how it differed from traditional case management interventions with the familiar one-on-one client--social worker relationship. The intervention was found to be a holistic, interactive, and preventive approach that took account of the multiple factors affecting homelessness and movement toward independence for the very low-income target population (Figure 2). The model had three distinct characteristics: (1) It was based on a family support and community building approach; (2) services were universal and accessible through four domains; and (3) services were provided in parallel, that is, simultaneously.

[FIGURE 2 OMITTED]

Family and Community Support Services. The activities focused on family and community development simultaneously, a theoretical approach drawn from the ecological perspective (Coulton, 1996; Weil, 2000) that sees the family at the center of a ring of concentric circles of support (Leventhal et al., 1997; Mulroy & Shay, 1997). Informal supportive activities were offered to surround and nurture the family using local community institutions and resources.

Accessibility through Four Domains. The intervention operated as four interactive domains in support of an overarching community-building goal (Bruner & Parachini, 1997; Chaskin, Joseph, & Chipenda-Dansokho, 1997; Mulroy & Lauber, 2002): (1) family strengthening (that is, provision of basic services and resources such as emergency food, rent, furniture, parenting classes, crisis counseling, information and referral); (2) education and life skills activities (that is, literacy programs, money management, computer classes); (3) worker development (that is, job readiness workshops, pre-employment training, resume preparation, career counseling); and (4) community improvement (that is, promotion of multicultural understanding, family fun, community pride, and safety).

Parallel Provision of Activities. The ecological approach to programming facilitated the parallel provision of activities (Cohen & Phillips, 1997; Mulroy, 1997) so that activities were offered as a seamless array of opportunities. Action among the domains was fluid for residents and staff (as depicted by the bidirectional arrows in Figure 2). There was no prescribed order for obtaining services and no specific staff person who performed an intake "gatekeeper" function.

The multiple directions of the arrows in Figure 2 show how activities in one domain were intended to influence outcomes in other domains. This acknowledged staff's understanding of the complexity and interdependence of elements considered to be building blocks of a social support network.

Phase 3: Delineate Implementation Processes

Interorganizational relationships were developed with other on-site PACT programs and with offsite agencies that resulted in an increased number of Family Center offerings. For example, from October 1997 through September 1998, a total of 122 classes, workshops, or events were coproduced with on-site and off-site partner programs and organizations. Evidence shows that every activity in each domain was coproduced in cooperation with at least one other organization or program. Thus, each participating organization played a key role in the provision of resources or services. The center leveraged its resources by finding organizational partners who were willing to participate in community building and then "packaged" diverse teams of partners for different activities.

Public--Private--Nonprofit Partnerships. These interventions represented multiparty and multisector partnerships with diverse organizational types. On site these included the resident management office of the private for-profit firm under contract with the Public Housing Authority to manage this housing project, other PACT programs with similar goals and objectives, and the Residents Association. Off-site linkages included the neighborhood public schools, the Girl Scouts, United Way, a private security guard firm, the State Department of Employment, Hawai'i Literacy, Hawai'i Community Action Program, Honolulu Community College, the Cooperative Extension Service at the University of Hawai'i, Manoa, and a myriad of local cultural and civic associations.

Shared Resources to Increase Capacity. Compatible programs in the PACT organization shared resources from their new federal funding streams. Between March and June 47 meetings, classes, and workshops were offered in all four domains. Of these, nearly 60 percent were worker development activities (Figure 3). This surge in offerings was the result of new partnerships that coalesced at that time from a newly released source of federal funding combined with a private-sector job training initiative. The time-limited job readiness classes ended in June but resumed in the fall, eventually developing tighter linkages to another newly funded PACT program in community economic development.

[FIGURE 3 OMITTED]

This finding confirmed partnership formation as a key element of community building (Chaskin & Ogletree, 1995) that builds on traditional intervention strategies to enable neighborhoods "to acquire, develop, and use human, economic, and institutional resources for the benefit of residents" (Naparstek & Dooley, 1997, p. 79). The effect of partner formation both among PACT programs and with community agencies and firms enabled residents to participate in diverse activities at the same time and over an extended period, implementing one key principle of the logic model.

Phase 4: Client Outcomes with Intensive, Customized Help

Evidence suggests that residents took different paths through these activities and services depending on their needs, life circumstances, and assets, confirming another underlying principle of the logic model. When new resources were accessible, residents used them. However, rather than follow a prescribed order, highly involved residents participated in activities in all four domains simultaneously.

lob Readiness Outcomes. Worker development activities were available to all residents of KPT, not just clients of the center. A total of 124 heads of household (16.5 percent of all resident heads of household in KPT) attended structured job readiness workshops, including security guard training offered in partnership with another PACT program funded with a federal Drug Elimination Grant. A total of 108 heads of household (87 percent of those enrolled) completed the trainings, achieving one key program outcome. Resident participation in these worker development programs peaked in May and June, when the most courses were offered (Figure 3).

Although data were not available on all participants in the worker development trainings, a subsample of 24 of 31 clients of the center who completed job readiness workshops was available (seven no responses) (Table 1).

Sixteen heads of household (67 percent) were engaged in some form of employment: seven (29 percent) had full-time employment in one job; five (21 percent) were employed in multiple jobs working both full-time and part-time; three (13 percent) were engaged in part-time work; one (4 percent) worked part-time, while also attending community college and volunteering. Of those not yet working, five (21 percent) were engaged in volunteering; 2 (8 percent) were enrolled in community college and volunteering, and one (4 percent) was in late-term pregnancy and out of the labor market.

Although there were limitations to these data because of the number of no responses, the range of outcomes experienced by these participants suggests that many definitions of "successful" job-related outcomes are required. The job-related outcomes found in this study were not small achievements in this neighborhood. There was interest and motivation among these public housing residents to prepare for employment when opportunities were accessible on site or in the neighborhood.

Intensive, Customized Help: Case Studies. Residents selected their own paths through the center's activities and customized their services use to meet their individual needs. Evaluators adapted the logic model (Table 2) to trace the paths of three highly involved residents engaged in the worker development activities: "Tomas," an American Samoan immigrant father of four; "Livia" and "Silah," Native American Pacific Islander mothers, each with four children. Path tracings and the case stories suggest that the center's programs in supportive counseling and volunteerism played important roles in helping residents move toward the program goal and that flexibility and access to activities on site facilitated program completion in hard times.

The reason for program entry for Tomas, a recent immigrant to Hawai'i, was to obtain furniture; for Livia, to participate in an aerobics class; and for Silah, to attend a job readiness workshop. From that point of entry, however, Tomas went on to participate in 17 activities; Livia and Silah participated in nine and seven, respectively.

Supportive counseling was used "on demand" and provided a safety net in times of crisis. Supportive counseling was generally sought only after trusting relationships were built with staff through other activities. All three residents used supportive counseling as their second activity, and the common theme was a family crisis. For example, Tomas was unsettled because he was separated from his wife and one child who were still in Samoa, and he was, in effect, a single parent of three small children. Livia's family situation changed dramatically when her husband was sent to prison for a second time. Silah, with sporadic attendance in job readiness training classes, arrived late to the class graduation with a noticeable black eye--the victim of domestic violence. Supportive counseling during these periods of crisis helped to stabilize each of these residents. This facilitated movement to a range of other activities that, over time, resulted in attainment of immediate and intermediate outcomes (Table 2).

Structured volunteer training and volunteer activities were key elements woven into the program that helped build self-confidence and define a social support network. One focus group participant said, "You know things are getting better when you are able to give back to others." Tomas, Livia, and Silah participated in volunteer training. This training gave them knowledge of neighborhood services and legitimacy to serve as resource experts for other residents. Over time, they volunteered in the resident civic association and eventually assumed leadership roles.

The critical juncture for most residents was found to be moving from immediate to intermediate outcomes--a hurdle made difficult because of multiple and unanticipated crises and setbacks. Use of an enlarging social support network and flexibility in Family Center procedures buffered these crises by helping to build personal resilience. For example, after completing two eight-week sessions of job readiness training and one eight-week session of security guard training, Tomas expected to get a job. However, when he applied for work he was rejected. Instead of remaining despondent and discouraged, he went back to the center to improve his job-seeking skills. Eventually he got a full-time job on-site at $8.05 per hour with benefits.

Livia was moving toward greater economic independence by becoming involved in center recreation activities, applying to Community College, applying for financial aid, and attending volunteer training. Then her husband, Keoni, got out of prison and came home to try and put the pieces of family life back together. Keoni enrolled in the center's job readiness classes. Then unexpectedly, he got seriously ill and was homebound. In a dramatic role reversal, he assumed a homemaker role, and the family tried to adjust to a new balance. The couple participated in center activities in all four domains, and their four young sons, ages five to 12, were involved in PACT's Community Teen Program. With new priorities in his life, Keoni's family became important to him. Livia continued to juggle numerous responsibilities necessary to comply with Welfare-to-Work requirements. She secured part-time work at $8.00 per hour, 19 hours a week at the on-site Teen Video Store; attended Community College on financial aid; and continued to volunteer on-site. The couple developed a renewed will to keep the family together.

Silah eventually and quietly returned to worker-development activities to receive individual career counseling, assistance with a resume, and eventually to complete her job readiness training. She kept her history of abuse very private. She then received concrete services through the family-strengthening domain, because she needed food and school supplies for the children. Her next step was to complete volunteer training and then to participate in volunteer projects onsite. Eventually, Silah got a part-time job that paid $15.92 per hour for 20 hours a week at the local elementary school.

Silah still lives in an abusive domestic relationship. Her increased economic independence may again be threatened by battering. Family Center staff observed a pattern of domestic violence as women stepped out of traditional, homebound, subservient roles to become more self-reliant.

Discussion

Several implications can be drawn from this study that inform policymakers, community-oriented practitioners, and academics.

Social Policy

Policymakers concerned with improving program outcomes should note the extent to which the federal agency's investment in technical assistance facilitated local-level organizational learning. Findings suggest that staff originally reached beyond their capacity, setting themselves up for program failure. HHS's investment in both national training and on-site technical assistance facilitated midcourse corrections and improvements, while also providing insightful comments regarding the evaluation methodology. At the local level a fear of failure was transformed into a spirit of pioneering. This view from the field suggests that federal demonstration grants can have positive effects on building the capacity of small programs when much-needed technical assistance is built in.

Social Work Practice

The articulation of a few clear and compelling principles in the logic model served as the guideposts for program design and determined the form of practice (Connell et al., 1995; Weiss, 1995; Weiss, 2000). Family Center staff brought the values and principles of the family support movement to the demonstration project because it existed as a Family Support Center first. The positive affirming statements reflected the principles of family strengthening and asset building derived from family support practice (Resource Coalition of America, 1996). The concept of practice did not conform to the traditional view of social work with individuals, groups, and communities--an approach staff considered unnecessarily narrow and unrealistic. Rather, to help residents reach the goal of "stable housing and movement toward self-sufficiency," practitioners needed to use a wide lens of working toward systems change, partnership formation, community development, and family strengthening simultaneously, an approach that is consistent with community building (Mulroy & Lauber, 2002; Naparstek & Dooley, 1997; Weil, 1996, 1997).

Research

Information Management. Practitioners involved in complex community-based initiatives need to systematically collect and manage appropriate data. The Family Center's evaluation was hindered by a lack of baseline data concerning client participation in their myriad activities. Although the family support model may be ideologically compatible with community building, practitioners must pay careful attention to documentation to tease out the diversity of client pathways followed in complex, comprehensive programs and to determine whether intended outcomes were achieved.

Researcher--Practitioner Collaboration. Practitioners can develop researcher--practitioner partnerships when certain criteria are met: the program manager and staff help formulate the research questions; the final report is coauthored by the researcher and practitioners; practitioners require that findings flow back to them, not only to a funder; a cooperative relationship is developed and sustained throughout the study; the researchers stay around to provide technical assistance with the implementation of findings (Schuman & Abramson, 2000).

Collaboration in this case was facilitated by four factors: (1) the goals, role, attitude, and actions of technical support personnel at the funding agency; (2) the resources of seven social work graduate students who served as research assistants in every phase of this labor-intensive study; (3) an egalitarian attitude and open process that conferred respect on participants as knowledgeable in their own fields, and (4) mutual benefits that accrued to practitioners and evaluators.

Language and the maintenance of professional boundaries were critical to sustaining a collegial relationship and required sensitivity and vigilance (Quinn Patton, 1997). The term "external" evaluator was always used with two effects: It clarified roles, and it set clear expectations for behavior. The term external evaluator conveyed to all the responsibility and commitment to adhere to established ethical standards of conduct in research methods (NASW, 2000). In a mutually beneficial internal--external exchange, practitioners permitted the evaluators to get a much-needed "insider view," and in return, the faculty member was expected to share cumulative knowledge and skills (Quinn Patton) gained from previous experience evaluating community-based programs in other parts of the country.

This resulted in a two-way transfer of knowledge. The evaluators were catalysts who offered theoretical insights that helped practitioners examine their work in a reflective, systematic way and who left behind evaluation tools and conceptual frameworks used by staff in subsequent grant applications. The staff offered evaluators deep access to the public housing community and an organizational setting for a prolonged period. This perspective from the field stimulated evaluators to critically examine the relevance of existing theories intended to inform practice, and equally important to consider how practice might better inform theory.

Conclusion

Community-based programs can increase their organizational capacity when they engage in the coproduction of evaluation that offers an ongoing opportunity to critically examine and improve work processes and products systematically. Logic modeling offers promise as an analytic framework to help practitioners and evaluators develop baselines, move toward better outcomes, and monitor program management in a funding environment that demands increased accountability and in a community environment that continues to pose challenges, risks, and rewards.

Table 1

Outcomes for Family Center Clients
Completing Job Readiness Workshops

 Working Not Working

Outcome No. % No. %

Full-time job 7 29
Full- and part-time jobs 5 21
Part-time job 3 13
Part-time job, school,
 and volunteering 1 4
School and volunteering 2 8
Volunteering 5 24
At home 1 4
All 16 67 8 33

NOTE: n = 24 is a subsample of 31 center clients who
completed workshops with 7 no responses.

Table 2

Adapted Logic Model: Customized Paths of Client Service Use and
Movement toward the Program Goals

Variable Tomas Livia

Recruitment Concrete services-needed Aerobics class
 furniture (Family (Community
 Strengthening) Improvement)
Initial point of Referred by Housing FSC Staff
access (domain) Management
Second service Crisis counseling Crisis counseling
used (domain) (Family Strengthening) (Family Strengthening)
Third service Job-related services Volunteer training
used (domain) (Worker Development) (Education/Life
 Skills)
Total number of 17 9
services used
Immediate out- Used emergency services, Used worker develop-
comes support groups, to meet ment services; applied
 family's basic needs; to community college;
 particpated in educa- applied for financial
 tional, worker develop- aid; spouse used
 ment courses. Social worker development
 support network services. Social
 expanded. support network
 expanded.
Intermediate Completed classes; Increased family
outcomes volunteered; found full- activities; increased
 time job on-site: $8.05/ volunteer participa-
 hr., 40 hrs./week, tion; attended
 benefits. community college;
 incompliance with
 welfare reform regula-
 tions; part-time job
 on-site: $8/hr., 19
 hrs./week.
Goals Stable housing; movement Stable housing;
 toward family stability movement toward family
 and independence. stability and inde-
 pendence.

Variable Silah

Recruitment Job-readiness training
 (Worker Development)
Initial point of Self
access (domain)
Second service Crisis counseling (Family
used (domain) Strengthening)
Third service Budgeting/money manage-
used (domain) ment (Education/Life Skills)
Total number of 7
services used
Immediate out- Attended worker development
comes classes; used family strength-
 ening services. Social support
 network expanded.
Intermediate Completed job-readiness class;
outcomes completed volunteer training;
 volunteered; increased inde-
 pendence and self-confidence;
 part-time job in neighborhood
 schools: $15.92/hr., 20 hrs./
 week.
Goals Stable housing; movement
 toward family stability and
 independence.

References

Annie E. Casey Foundation. (1997). Evaluating comprehensive community change. Baltimore: Author.

Alter, C., & Murty, S. (1997). Logic modeling: A tool for teaching practice evaluation. Journal of Social Work Education, 33, 103-117.

Bickman, L., & Rog, D. (1998). Handbook of applied social research. Thousand Oaks, CA: Sage Publications.

Bruner, C., & Parachini, L. (1997). Building community. Washington, DC: Together We Can Initiative.

Chaskin, R. J., Joseph, M. L., & Chipenda-Dansokho, S. (1997). Implementing comprehensive community development: Possibilities and limitations. Social Work 42, 435-444.

Chaskin, R., & Ogletree, R. (1995). The Ford Foundation's family and neighborhood initiative: Building collaboration. Chicago: University of Chicago, Chapin Hall Center for Children.

Cohen, C. S., & Phillips, M. H. (1997). Building community: Principles for social work practice in housing settings. Social Work, 42, 471-481.

Connell, J., Kubisch, A., Schorr, L., & Weiss, C. (1995). New approaches W evaluating community initiatives. Washington, DC: Aspen Institute.

Conrad, K., Randolph, F., Kirby, M., & Bebout, R. (1999). Creating and using logic models: Four perspectives. Alcoholism Treatment Quarterly, 17(1 /2), 17-31.

Coulton, C. J. (1996). Poverty, work, and community: A research agenda for an era of diminishing federal responsibility. Social Work, 41, 509-519.

Fetterman, D., Kaftarian, S., & Wandersman, A. (1996). Empowerment evaluation: Knowledge and tools for self-assessment and accountability. Thousand Oaks, CA: Sage Publications.

Forbes, D. (1998). Measuring the unmeasurable: Empirical studies of nonprofit organizational effectiveness from 1977-1997. Nonprofit and Voluntary Sector Quarterly, 27, 183-202.

Greenwood, D. J., & Levin, M. (1998). An introduction to action research Thousand Oaks, CA: Sage Publications.

Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage Publications.

Heineman-Pieper, M. (1994). Science not scientism: The robustness of naturalistic clinical research. In E. Sherman & W. J. Reid (Eds.), Qualitative research in social work (pp. 71-88). New York: Columbia University Press.

Huberman, M., & Miles, M. (1994). Data management and analysis methods. In N. Denzin & Y. Lincoln (Eds.), Handbook of qualitative research (pp. 428-444). Thousand Oaks, CA: Sage Publications.

Jansson, B. (1994). Social policy: From theory to practice (2nd ed.). Pacific Grove, CA: Brooks/Cole.

Knapp, M. (1996). Methodological issues in evaluating integrated services initiatives. In J. Marquart & E. Konrad (Eds.), Evaluating initiatives to integrate human services (pp. 21-34). San Francisco: Jossey-Bass.

Krueger, R. (1994). Focus groups: A practical guide for applied research (3rd ed.). Thousand Oaks, CA: Sage Publication.

Leventhal, T., Brooks-Gunn, J., Kamerman, S. (1997). Communities as place, face, and space: Provision of services to poor urban children and their families. In J. Brooks-Gunn, G. Duncan, & L. Aber (Eds.), Neighborhood poverty (Vol. 1, pp. 182-204). New York: Russell Sage Foundation.

Logan, T., & Royce, D. (2001). Program evaluation. In B. Thyer (Ed.), Handbook of social work research methods (pp. 193-206). Thousand Oaks, CA: Sage Publications.

Marquart, J., & Konrad, E. (Eds.). (1996). Evaluating initiatives to integrate human services. San Francisco: Jossey-Bass.

Maxwell, J. (1998). Designing a qualitative study. In L. Bickman & D. Rog (Eds.), Handbook of applied social research (pp. 69-100). Thousand Oaks, CA: Sage Publications.

Miles, M., & Huberman, M. (1994). Qualitative data analysis. Thousand Oaks, CA: Sage Publications.

Mulroy, E. A. (1997). Building a neighborhood network: Interorganizational collaboration to prevent child abuse and neglect. Social Work, 42, 255-264.

Mulroy, E. A., & Lauber, H. (1999). Evaluation of the second year Federal Demonstration Grant Preventing Homelessness and Moving Families toward Self Sufficiency: Final report. (grant no. 90EXO057). Submitted to the U.S. Department. of Health and Human Services, Office of Community Services. June 7, 1999).

Mulroy, E. A., & Lauber, H. (2002). Community building in hard times: A post-welfare view from the streets. Journal of Community Practice, 10(1) 1-16.

Mulroy, E. A., & Shay, S. (1997). Nonprofit organizations and innovation: A model of neighborhood-based collaboration to prevent child maltreatment. Social Work, 42, 515-524.

Naparstek, A. J., & Dooley, D. (1997). Countering urban disinvestment through community-building activities. Social Work, 42, 506-514.

National Association of Social Workers. (2000). Code of ethics of the National Association of Social Workers. Washington, DC: Author.

O'Connor, A. (1995). Evaluating comprehensive initiatives: A view from history. In J. Connell, A. Kubisch, L. Schorr, & C. Weiss (Eds.), New approaches to evaluating community initiatives (pp. 23-63). Washington, DC: Aspen Institute.

Quinn Patton, M. (1997). Utilization-focused evaluation (3rd ed.). Thousand Oaks, CA: Sage Publications.

Resource Coalition of America. (1996). Guidelines for family support practice. Chicago: Author.

Schalock, R. (2001). Outcomes-based evaluation (2nd ed.). New York: KluwerAcademic/Plenum Publications.

Schuman, C., & Abramson, A. (2000). Collaborating for usable knowledge: A Work in progress by the Nonprofit Sector Research Fund. Nonprofit and Voluntary Sector Quarterly, 29(1), 11-23.

Stevenson, J., Mitchell, R., & Florian, P. (1996). Evaluation and self-direction in community prevention coalitions. In D. Fetterman, S. Kaftarian, & A. Wandersman (Eds.), Empowerment evaluation: Knowledge and tools for self-assessment and accountability (pp. 208-233) Thousand Oaks, CA: Sage Publications.

Strawn, J., & Martinson, K. (2000). Steady work and better jobs. Washington, DC: Manpower Demonstration Research Corporation.

Tyson, K. (1995). New foundations for scientific social and behavioral research: The heuristic paradigm. Boston: Allyn & Bacon.

Weil, M. (1996). Model development in community practice: An historical perspective. Journal of Community Practice, 3(3/4) 5-68.

Weil, M. (1997). Community building: Building community practice. In P. L. Ewalt, E. M. Freeman, S. A. Kirk, & D. L. Poole (Eds.), Social policy: Reform, research, and practice (pp. 35-61). Washington, DC: NASW Press.

Weil, M. (2000). Services for families and children: The changing context and new challenges. In R. Patti (Ed.), The handbook of social welfare management (pp. 481-510). Thousand Oaks, CA: Sage Publications.

Weiss, C. (1995). Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families. In J. Connell, A. Kubisch, L. Schorr, & C. Weiss (Eds.), New approaches to evaluating community initiatives (65-92). Washington, DC: Aspen Institute.

Weiss, J. (2000). From research to social improvement: Understanding theories of intervention. Nonprofit and Voluntary Sector Quarterly, 29(1), 81-110.

Witkin, S. (1994). Commentary: Evaluation in context: Is there any other way? In E. Sherman & W. Reid (Eds.), Qualitative research in social work (pp. 328-332). New York: Columbia University Press.

Wolch, J. (1999). Decentering America's nonprofit sector: Reflections on Salamon's Crises Analysis. Voluntas, 10(1), 25-35.

Yegidis, B., & Weinbach, R. (1996). Research methods for social workers. Boston: Allyn & Bacon.

Yin, R. (1994). Case study research. Thousand Oaks, CA: Sage Publications.

Yin, R. (1998). The abridged version of case study research: Design and method. In L. Backman & D. Rog (Eds.), Handbook of applied social research (pp. 229-260). Thousand Oaks, CA: Sage Publications.

Yin, R., Kaftarian, S., & Jacobs, N. (1996). Empowerment evaluation at federal and local levels: Dealing with quality. In D. Fetterman, S. Kaftarian, & A. Wandersman (Eds.), Empowerment evaluation: Knowledge and tools for self-assessment and accountability (pp. 188-207). Thousand Oaks, CA: Sage Publications.

Elizabeth A. Mulroy, PhD, is associate professor, University of Maryland, Baltimore, 525 West Redwood Street, Baltimore, MD 21201; e-mail: emulroy@ssw.umaryland.edu. Helenann Lauber, BS, is program director, Family Support Center, Parents and Children Together, Honolulu, Hawaii. An earlier version of this article was presented at the Council on Social Work Education Annual Program Meeting, February 28, 2000, New York.

Original manuscript received August 24, 2001 Final revision received September 1, 2003 Accepted November 18, 2003