A user-friendly approach to program evaluation and effective community interventions for families at risk of homelessness.
Mulroy, Elizabeth A. ; Lauber, Helenann
Practitioners who administer social programs seek usable knowledge
from academic researchers that encompasses the critical issues of the
times and the problem of hands-on management (Schuman & Abramson,
2000). Practitioners in small nonprofit organizations in particular face
a multitude of issues in the implementation of public sector grants and
contracts that affect program management and evaluation. Particularly
hard hit are prevention programs in local community-based nonprofit
organizations that serve very low-income children and families (Weil,
2000).
The purpose of this article is to demonstrate how a Family
Center's program evaluation--an example of coproduction that used
logic modeling as an analytic framework--can help social work
practitioners and local community workers evaluate their own programs. A
logic model is a one-page "graphic representation of a program that
describes the program's essential components and expected
accomplishments and conveys the logical relationship between these
components and their outcomes" (Conrad, Randolph, Kirby, &
Bebout, 1999, p. 18).
As Wolch (1999) contended: "The real burden is on nonprofit
agencies suddenly faced with rising demands for services, reduced public
funding, and mandates to monitor clients and enforce sanctions including
benefit terminations and evictions, on behalf of their partner the
state" (p. 28). In the face of these developments, to what extent
are program managers in community-based nonprofits capable of responding
to local needs? This is a timely question for the evaluation of social
work practice. First, at the turn of the 20th century, small
community-based nonprofit organizations such as settlement houses
historically served poor neighborhoods as mediating institutions to help
immigrant newcomers move out of poverty (Jansson, 1994). In the new
millennium, immigrant issues are controversial social policy concerns.
Today, with privatization generating role shifts of government as funder
and nonprofit as provider, we need to know more about how services are
configured in low-income neighborhoods and how targeted beneficiaries
use such services.
Second, social workers are in positions of responsibility for the
development, administration, and evaluation of program initiatives that
implement new federal and state social policies intended to reform
public welfare, child welfare, and public housing (Mulroy & Lauber,
1999). Many of these programs are small and focus on the coordination of
community-based services intended to increase resident and community
empowerment. The philanthropic community, also invested in strengthening
families and communities, encourages local projects to be comprehensive
community initiatives that use collaborations and partnerships to
achieve community-building goals (Annie E. Casey Foundation, 1997;
Leventhal, Brooks-Gunn, & Kamerman, 1997; O'Connor, 1995).
Third, public and philanthropic funders expect program managers to
perform these functions in a context of heightened accountability for
program efficiency and effectiveness (Forbes, 1998; Schalock, 2001). An
important contextual factor is that in the era of privatization most
program evaluations have a political context; the research questions may
come from federal, state, or local government funding agencies, and
findings are intended to provide feedback to legislators to inform their
future resource allocation decisions (Bickman & Rog, 1998; Yegidis
& Weinbach, 1996). The dilemma faced by many program managers in
community-based nonprofit organizations is their lack of training in
evaluation research with preferred experimental or quasi-experimental
designs and control processes and limited budgets that prevent hiring
consultants to carry out the evaluations. Many practitioners seek
knowledge from evaluation research that can help them improve their
programs, not just respond to the call for externally conducted outcome
evaluations. To solve the practice dilemmas for service accountability,
program improvement, and community-based solutions, a movement has
emerged to identify "best practices" and in the process make
program evaluation more usable for many audiences (Connell, Kubisch,
Schorr, & Weiss, 1995; Fetterman, Kaftarian, & Wandersman, 1996;
Marquart & Konrad, 1996; Quinn Patton, 1997; Resource Coalition of
America, 1996). One commonality is the coproduction of the evaluation by
evaluators and practitioners. The developmental approach expects that
the evaluation process will result in improvement of internal program
and agency operations--often referred to as organizational capacity--as
the lessons learned become institutionalized (Quinn Patton).
Background: The Family Center
Several characteristics of Parents and Children Together (PACT),
the Family Center, and the public housing project are relevant to
professional practitioners, indigenous community workers, funders, and
academics concerned with program planning, management, and evaluation of
community-based programs. These include neighborhood characteristics,
organizational structure, and program goals.
Neighborhood Characteristics
PACT and its programs were located on-site at Kuhio Park Terrace
(KPT) public housing project in urban Honolulu, Hawai'i, a
development of about 2,500 very low-income people, largely immigrants.
The demographic profile of residents was similar to other public housing
projects in large cities: 94 percent were people of color; 68 percent of
families were headed by single parents; 80 percent received public
assistance; and the average annual family income was $11,412 in a city
where the median income for a family of four was $60,400. KPT, with two
high-rise towers and low-rise garden apartments, was located close to
the city's main highway in a densely populated neighborhood
surrounded by single-family homes, apartment complexes, an elementary
school, Honolulu Community College, and commercial firms. Other
high-density public and subsidized housing complexes were in close
proximity so that gang activity and turf wars were not uncommon. The
neighborhood had an unemployment rate of 11.5 percent, whereas the state
experienced a 5.7 percent unemployment rate as the national rate dipped
to 4.1 percent (Mulroy & Lauber, 1999). Recreational opportunities
and nonprofit social services agencies were resources in the
neighborhood that was well served with public bus transportation to a
nearby strip shopping mall just a few blocks away.
Organizational Structure
The Family Center was one of 15 programs of PACT, a large, private,
community-based, nonprofit family services agency that targeted most of
its programs in the KPT public housing development but had service sites
across the state. Its programs were clustered into five service areas:
(1) early childhood education, (2) mental health support, (3) community
economic development, (4) child abuse prevention and treatment, and (5)
domestic violence prevention and treatment. PACT's mission was to
promote and support healthy individuals, families, and communities by
creating opportunities for them to identify and address their own
strengths, needs, and concerns and to realize their potential. PACT was
an umbrella organization that pioneered the integration of
community-based services and the partnership concept of service
delivery. Founded in 1968, the agency's revenues were increasingly
generated from public grants and contracts.
Program Goals
In 1995 PACT received a three-year demonstration grant from the
U.S. Department of Health and Human Services [HHS] to help prevent
homelessness among at-risk, very low-income families living in a large
public housing development. The $218,000-a-year grant was administered
through the Family Center to implement the social policy goal of
"preventing homelessness and moving families to
self-sufficiency." The definition of moving to self-sufficiency in
federal terms meant moving to independence from government subsidies,
such as Temporary Assistance to Needy Families (TANF) and public
housing. The target population was current residents of KPT who were
previously homeless or at risk of homelessness, primarily through
eviction.
Family Center staff reframed the definition of self-sufficiency to
mean movement toward independence. Such movement was expected to be
different among their target population depending on life circumstance;
would not happen swiftly, and would include backslides and steps
forward. Staff's definition of preventing homelessness meant
helping those at risk of homelessness stabilize their tenancy by
remaining in the public housing complex in good standing or moving out
of KPT in good standing to another apartment if that was their choice.
Potential program participants were believed to have multiple
barriers to personal and material independence. They were less likely to
move out of public housing in good standing or off TANF roles and into
wage work than people with fewer barriers who were more easily served by
traditional job training programs offered by private industry councils
through community economic development programs (Strawn & Martinson,
2000).
At the inception of the grant, the program was designed to prevent
homelessness through the provision of intensive and comprehensive
support services to previously homeless families and families at risk of
becoming homeless. By the end of the funding period, the center expected
a majority of participants to have stable housing and move toward a job,
and for the entire public housing community to experience an increase in
civic pride and in resident participation. Staff included professionally
trained managers and local community workers familiar with the multiple
cultures and languages represented by resident groups. The budget
provided a modest amount for program evaluation conducted by an external
evaluator.
Method
Many government agencies that fund social programs require logic
modeling in grantee program planning and evaluation. A logic model can
be useful to practitioners and evaluators: It offers a clear
conceptualization of a chain of events that, when developed at the
beginning of a project, serves as a program planning tool with
guideposts for program implementation (Yin, 1998); the conceptualization
reflects the theory of program intervention (Mulroy & Shay, 1997;
Weiss, 1995; Weiss, 2000); it is sufficiently detailed to guide
formulation of the research design (Yin, 1998); and it can be used
developmentally as a project typically shifts and changes as it moves
forward in time (Alter & Murty, 1997).
HHS determined the purpose and boundaries of the evaluation of the
KPT project. First, this was a study of the center's second year of
operations only, setting the timeline to activities developed and
outcomes attained up to that point. Second, it asked for innovation in
programming and in evaluation design. It was made clear at the outset
that a major purpose of the evaluation was to provide policymakers in
Congress and in HHS, Office of Community Services, with lessons learned
to help them make sound decisions for future homeless prevention
programs. Knowing the purpose of the evaluation at the outset
facilitated decision making about the type of information needed, the
best way to measure the target variables, and the appropriate units of
analysis (Logan & Royce, 2001).
Before the evaluation began, a user-friendly approach to program
evaluation was put in place: HHS required all practitioners who applied
for homelessness prevention funds to complete a logic model as the
framework for their proposed program and to include it in their grant
application. HHS also required grant recipients to refine and expand
their logic models into real-world program plans. Among the materials
provided to grantees was a one-page outline of a logic model that
required the identification of clearly stated theories, principles, or
assumptions that guided their activities and interventions; activities
required to achieve the intended outcomes; anticipated outcomes defined
as immediate and long-term, and the ultimate goal. With a small budget
and a tight one-year time frame, the center contracted with a professor
of management, community planning, and social policy at a local school
of social work to conduct the external evaluation, with the promise of
involving graduate students.
An action research approach was used because it was consistent with
the evaluator's approach to knowledge building; afforded site-level
analysis for an in-depth examination of the social context; was
compatible with staffs interest in participation; and facilitated the
use of multiple methods that best fit the research questions and the
complexity of the context with the resources available (Greenwood &
Levin, 1998; Knapp, 1996; Quinn Patton, 1997; Witkin, 1994). The
evaluator had three front-end goals: (1) be a partner with staff by over
time conducting work face-to-face and on-site; (2) improve the quality
of data collection by creating a management information system, testing
it out with staff, redesigning it as needed, then leaving it behind; and
(3) improve the program's organizational capacity (Quinn Patton,
1997; Stevenson, Mitchell, & Florian, 1996; Yin, Kaftarian, &
Jacobs, 1996).
Logic modeling was selected as a framework for analysis in an
embedded single case study (Yin, 1994, 1998). Staff were familiar with
logic modeling as a program planning tool. Therefore, its continued use
had the potential to enhance staffs ability to think critically, that
is, to define assumptions and line of reasoning among assumptions,
activities, and outcomes that help link theory to practice (Alter &
Murty, 1997). Logic modeling also provided a framework for conducting a
process evaluation needed by legislators for their interest in
replication and helped focus the design of the outcome evaluation to
account for multilevel factors--individual, neighborhood, and societal
(Conrad et al., 1999; Knapp, 1996). Finally, findings from the data
could be compared with the original logic model in a pattern-matching
mode to determine the viability of the original conceptualization (Yin,
1998).
Data were gathered from multiple and diverse sources: review of the
center's case files, archival records, program documents, focus
groups, interviews, participant observation, a "physical
artifact" timeline (Yin, 1994), and housing authority tenant rent
role records. New instruments for collecting data on program activities
were codesigned with staff, pretested, and then revised for easier use
by staff. Three well-attended formal focus groups were held with 17
highly involved program participants to gauge their participation in the
center's activities. Group meetings, each facilitated by the
evaluator and staffed by two graduate research assistants and not
attended by staff, were taperecorded and transcribed (Krueger, 1994).
A physical artifact (Yin, 1994, 1998) Resident Participation Time
Line (RPTL) was created in the final focus group as one source of data
to determine residents' level and form of participation in the
center's activities. A 10-foot-long roll of paper was taped to a
wall in a familiar, comfortable activities room in the housing project
and marked with the months and years of the center's operations.
Residents were supplied with color-coded stickers, markers, and symbols
and asked to identify when and through which activity they first entered
the program, which services were used and when, and what critical
events, turning points, or milestones were most important to them in
their relationship with the center.
Multiple methods of data analysis were used (Bickman & Rog,
1998; Maxwell, 1998; Yin, 1998). Focus group data and the RPTL were
analyzed using traditional focus group methods (Krueger, 1994). Three
evaluators conducted an independent analysis of each focus group
transcript, after which the evaluating team convened to discover common
themes and patterns. Qualitative data were analyzed using an interactive
model consisting of data reduction, data display, and conclusion drawing
and verification (Huberman & Miles, 1994; Miles & Huberman,
1994). Conclusions were drawn only after all evaluators concurred on
emerging themes and patterns. Quantitative methods were used to analyze
tenant housing histories, rent payment schedules, and evictions.
Verification and a control for evaluator biases were achieved by
triangulating the multiple methods used to gather and analyze data and
by incorporating informant feedback (Bickman & Rog, 1998; Guba &
Lincoln, 1989; Heineman-Pieper, 1994; Maxwell; Quinn Patton, 1997;
Tyson, 1995; Yin, 1994, 1998). Center staff were involved in the
iterative process as data collectors and as a feedback loop for purposes
of confirmability. Regularly scheduled meetings were held between
evaluators and staff for this purpose, as well as to comanage the
evaluation.
Findings
Four phases unfolded sequentially that informed the way Family
Center staff understood and operated its demonstration program.
Phase 1: Logic Modeling as Program Planning
The first product of the evaluation was a revised logic model that
streamlined and systematized the originally conceived plan into a road
map through the complex community-based program (Figure 1). There is a
logical flow from a belief structure to related interventions, to
outcomes, and then to a goal. The model consisted of six assumptions or
principles targeted to multiple systems--families, communities,
neighborhoods, and institutions--that underlie five domains of program
activities and interventions. These activities were intended to help
participants achieve one or all of the four immediate outcomes as
indicated by their circumstances. We hoped that residents could then
attain the appropriate intermediate outcomes, and ultimately the
long-term goal of stable housing and movement toward self-sufficiency.
[FIGURE 1 OMITTED]
It took several months of intensive discussion, analysis, and
decision making between the practitioners and the evaluators (faculty
member and graduate research assistants) to revise the logic model. In
the end the number of principles proposed by staff was reduced from 28
to six (as shown in Figure 1). The program design was changed from 72
interventions to five domains of activities, and staff downsized the
number of people it intended to serve in this demonstration project from
all 2,500 public-housing residents to the 93 most at risk of
homelessness. The logic model revision had three effects. First, it
helped the staff scale back its own scope of work by better
understanding what was realistic and doable by a very small staff in a
time-limited demonstration project. Second, it helped the evaluators
understand the complexity of the interventions, the organizational
culture of the center, and the social environment of the public housing
project. Third, it offered an opportunity for practitioners and
evaluators to get to know each other and learn to work collaboratively
and respectfully.
Phase 2: Conceptualize the Intervention
The revision process highlighted the need to clarify to staff,
local leaders, and funders what the family support intervention
"looked like" and how it differed from traditional case
management interventions with the familiar one-on-one client--social
worker relationship. The intervention was found to be a holistic,
interactive, and preventive approach that took account of the multiple
factors affecting homelessness and movement toward independence for the
very low-income target population (Figure 2). The model had three
distinct characteristics: (1) It was based on a family support and
community building approach; (2) services were universal and accessible
through four domains; and (3) services were provided in parallel, that
is, simultaneously.
[FIGURE 2 OMITTED]
Family and Community Support Services. The activities focused on
family and community development simultaneously, a theoretical approach
drawn from the ecological perspective (Coulton, 1996; Weil, 2000) that
sees the family at the center of a ring of concentric circles of support
(Leventhal et al., 1997; Mulroy & Shay, 1997). Informal supportive
activities were offered to surround and nurture the family using local
community institutions and resources.
Accessibility through Four Domains. The intervention operated as
four interactive domains in support of an overarching community-building
goal (Bruner & Parachini, 1997; Chaskin, Joseph, &
Chipenda-Dansokho, 1997; Mulroy & Lauber, 2002): (1) family
strengthening (that is, provision of basic services and resources such
as emergency food, rent, furniture, parenting classes, crisis
counseling, information and referral); (2) education and life skills
activities (that is, literacy programs, money management, computer
classes); (3) worker development (that is, job readiness workshops,
pre-employment training, resume preparation, career counseling); and (4)
community improvement (that is, promotion of multicultural
understanding, family fun, community pride, and safety).
Parallel Provision of Activities. The ecological approach to
programming facilitated the parallel provision of activities (Cohen & Phillips, 1997; Mulroy, 1997) so that activities were offered as a
seamless array of opportunities. Action among the domains was fluid for
residents and staff (as depicted by the bidirectional arrows in Figure
2). There was no prescribed order for obtaining services and no specific
staff person who performed an intake "gatekeeper" function.
The multiple directions of the arrows in Figure 2 show how
activities in one domain were intended to influence outcomes in other
domains. This acknowledged staff's understanding of the complexity
and interdependence of elements considered to be building blocks of a
social support network.
Phase 3: Delineate Implementation Processes
Interorganizational relationships were developed with other on-site
PACT programs and with offsite agencies that resulted in an increased
number of Family Center offerings. For example, from October 1997
through September 1998, a total of 122 classes, workshops, or events
were coproduced with on-site and off-site partner programs and
organizations. Evidence shows that every activity in each domain was
coproduced in cooperation with at least one other organization or
program. Thus, each participating organization played a key role in the
provision of resources or services. The center leveraged its resources
by finding organizational partners who were willing to participate in
community building and then "packaged" diverse teams of
partners for different activities.
Public--Private--Nonprofit Partnerships. These interventions
represented multiparty and multisector partnerships with diverse
organizational types. On site these included the resident management
office of the private for-profit firm under contract with the Public
Housing Authority to manage this housing project, other PACT programs
with similar goals and objectives, and the Residents Association.
Off-site linkages included the neighborhood public schools, the Girl
Scouts, United Way, a private security guard firm, the State Department
of Employment, Hawai'i Literacy, Hawai'i Community Action
Program, Honolulu Community College, the Cooperative Extension Service at the University of Hawai'i, Manoa, and a myriad of local cultural
and civic associations.
Shared Resources to Increase Capacity. Compatible programs in the
PACT organization shared resources from their new federal funding
streams. Between March and June 47 meetings, classes, and workshops were
offered in all four domains. Of these, nearly 60 percent were worker
development activities (Figure 3). This surge in offerings was the
result of new partnerships that coalesced at that time from a newly
released source of federal funding combined with a private-sector job
training initiative. The time-limited job readiness classes ended in
June but resumed in the fall, eventually developing tighter linkages to
another newly funded PACT program in community economic development.
[FIGURE 3 OMITTED]
This finding confirmed partnership formation as a key element of
community building (Chaskin & Ogletree, 1995) that builds on
traditional intervention strategies to enable neighborhoods "to
acquire, develop, and use human, economic, and institutional resources
for the benefit of residents" (Naparstek & Dooley, 1997, p.
79). The effect of partner formation both among PACT programs and with
community agencies and firms enabled residents to participate in diverse
activities at the same time and over an extended period, implementing
one key principle of the logic model.
Phase 4: Client Outcomes with Intensive, Customized Help
Evidence suggests that residents took different paths through these
activities and services depending on their needs, life circumstances,
and assets, confirming another underlying principle of the logic model.
When new resources were accessible, residents used them. However, rather
than follow a prescribed order, highly involved residents participated
in activities in all four domains simultaneously.
lob Readiness Outcomes. Worker development activities were
available to all residents of KPT, not just clients of the center. A
total of 124 heads of household (16.5 percent of all resident heads of
household in KPT) attended structured job readiness workshops, including
security guard training offered in partnership with another PACT program
funded with a federal Drug Elimination Grant. A total of 108 heads of
household (87 percent of those enrolled) completed the trainings,
achieving one key program outcome. Resident participation in these
worker development programs peaked in May and June, when the most
courses were offered (Figure 3).
Although data were not available on all participants in the worker
development trainings, a subsample of 24 of 31 clients of the center who
completed job readiness workshops was available (seven no responses)
(Table 1).
Sixteen heads of household (67 percent) were engaged in some form
of employment: seven (29 percent) had full-time employment in one job;
five (21 percent) were employed in multiple jobs working both full-time
and part-time; three (13 percent) were engaged in part-time work; one (4
percent) worked part-time, while also attending community college and
volunteering. Of those not yet working, five (21 percent) were engaged
in volunteering; 2 (8 percent) were enrolled in community college and
volunteering, and one (4 percent) was in late-term pregnancy and out of
the labor market.
Although there were limitations to these data because of the number
of no responses, the range of outcomes experienced by these participants
suggests that many definitions of "successful" job-related
outcomes are required. The job-related outcomes found in this study were
not small achievements in this neighborhood. There was interest and
motivation among these public housing residents to prepare for
employment when opportunities were accessible on site or in the
neighborhood.
Intensive, Customized Help: Case Studies. Residents selected their
own paths through the center's activities and customized their
services use to meet their individual needs. Evaluators adapted the
logic model (Table 2) to trace the paths of three highly involved
residents engaged in the worker development activities:
"Tomas," an American Samoan immigrant father of four;
"Livia" and "Silah," Native American Pacific
Islander mothers, each with four children. Path tracings and the case
stories suggest that the center's programs in supportive counseling
and volunteerism played important roles in helping residents move toward
the program goal and that flexibility and access to activities on site
facilitated program completion in hard times.
The reason for program entry for Tomas, a recent immigrant to
Hawai'i, was to obtain furniture; for Livia, to participate in an
aerobics class; and for Silah, to attend a job readiness workshop. From
that point of entry, however, Tomas went on to participate in 17
activities; Livia and Silah participated in nine and seven,
respectively.
Supportive counseling was used "on demand" and provided a
safety net in times of crisis. Supportive counseling was generally
sought only after trusting relationships were built with staff through
other activities. All three residents used supportive counseling as
their second activity, and the common theme was a family crisis. For
example, Tomas was unsettled because he was separated from his wife and
one child who were still in Samoa, and he was, in effect, a single
parent of three small children. Livia's family situation changed
dramatically when her husband was sent to prison for a second time.
Silah, with sporadic attendance in job readiness training classes,
arrived late to the class graduation with a noticeable black eye--the
victim of domestic violence. Supportive counseling during these periods
of crisis helped to stabilize each of these residents. This facilitated
movement to a range of other activities that, over time, resulted in
attainment of immediate and intermediate outcomes (Table 2).
Structured volunteer training and volunteer activities were key
elements woven into the program that helped build self-confidence and
define a social support network. One focus group participant said,
"You know things are getting better when you are able to give back
to others." Tomas, Livia, and Silah participated in volunteer
training. This training gave them knowledge of neighborhood services and
legitimacy to serve as resource experts for other residents. Over time,
they volunteered in the resident civic association and eventually
assumed leadership roles.
The critical juncture for most residents was found to be moving
from immediate to intermediate outcomes--a hurdle made difficult because
of multiple and unanticipated crises and setbacks. Use of an enlarging
social support network and flexibility in Family Center procedures
buffered these crises by helping to build personal resilience. For
example, after completing two eight-week sessions of job readiness
training and one eight-week session of security guard training, Tomas
expected to get a job. However, when he applied for work he was
rejected. Instead of remaining despondent and discouraged, he went back
to the center to improve his job-seeking skills. Eventually he got a
full-time job on-site at $8.05 per hour with benefits.
Livia was moving toward greater economic independence by becoming
involved in center recreation activities, applying to Community College,
applying for financial aid, and attending volunteer training. Then her
husband, Keoni, got out of prison and came home to try and put the
pieces of family life back together. Keoni enrolled in the center's
job readiness classes. Then unexpectedly, he got seriously ill and was
homebound. In a dramatic role reversal, he assumed a homemaker role, and
the family tried to adjust to a new balance. The couple participated in
center activities in all four domains, and their four young sons, ages
five to 12, were involved in PACT's Community Teen Program. With
new priorities in his life, Keoni's family became important to him.
Livia continued to juggle numerous responsibilities necessary to comply
with Welfare-to-Work requirements. She secured part-time work at $8.00
per hour, 19 hours a week at the on-site Teen Video Store; attended
Community College on financial aid; and continued to volunteer on-site.
The couple developed a renewed will to keep the family together.
Silah eventually and quietly returned to worker-development
activities to receive individual career counseling, assistance with a
resume, and eventually to complete her job readiness training. She kept
her history of abuse very private. She then received concrete services
through the family-strengthening domain, because she needed food and
school supplies for the children. Her next step was to complete
volunteer training and then to participate in volunteer projects onsite.
Eventually, Silah got a part-time job that paid $15.92 per hour for 20
hours a week at the local elementary school.
Silah still lives in an abusive domestic relationship. Her
increased economic independence may again be threatened by battering.
Family Center staff observed a pattern of domestic violence as women
stepped out of traditional, homebound, subservient roles to become more
self-reliant.
Discussion
Several implications can be drawn from this study that inform
policymakers, community-oriented practitioners, and academics.
Social Policy
Policymakers concerned with improving program outcomes should note
the extent to which the federal agency's investment in technical
assistance facilitated local-level organizational learning. Findings
suggest that staff originally reached beyond their capacity, setting
themselves up for program failure. HHS's investment in both
national training and on-site technical assistance facilitated midcourse corrections and improvements, while also providing insightful comments
regarding the evaluation methodology. At the local level a fear of
failure was transformed into a spirit of pioneering. This view from the
field suggests that federal demonstration grants can have positive
effects on building the capacity of small programs when much-needed
technical assistance is built in.
Social Work Practice
The articulation of a few clear and compelling principles in the
logic model served as the guideposts for program design and determined
the form of practice (Connell et al., 1995; Weiss, 1995; Weiss, 2000).
Family Center staff brought the values and principles of the family
support movement to the demonstration project because it existed as a
Family Support Center first. The positive affirming statements reflected
the principles of family strengthening and asset building derived from
family support practice (Resource Coalition of America, 1996). The
concept of practice did not conform to the traditional view of social
work with individuals, groups, and communities--an approach staff
considered unnecessarily narrow and unrealistic. Rather, to help
residents reach the goal of "stable housing and movement toward
self-sufficiency," practitioners needed to use a wide lens of
working toward systems change, partnership formation, community
development, and family strengthening simultaneously, an approach that
is consistent with community building (Mulroy & Lauber, 2002;
Naparstek & Dooley, 1997; Weil, 1996, 1997).
Research
Information Management. Practitioners involved in complex
community-based initiatives need to systematically collect and manage
appropriate data. The Family Center's evaluation was hindered by a
lack of baseline data concerning client participation in their myriad
activities. Although the family support model may be ideologically
compatible with community building, practitioners must pay careful
attention to documentation to tease out the diversity of client pathways
followed in complex, comprehensive programs and to determine whether
intended outcomes were achieved.
Researcher--Practitioner Collaboration. Practitioners can develop
researcher--practitioner partnerships when certain criteria are met: the
program manager and staff help formulate the research questions; the
final report is coauthored by the researcher and practitioners;
practitioners require that findings flow back to them, not only to a
funder; a cooperative relationship is developed and sustained throughout
the study; the researchers stay around to provide technical assistance
with the implementation of findings (Schuman & Abramson, 2000).
Collaboration in this case was facilitated by four factors: (1) the
goals, role, attitude, and actions of technical support personnel at the
funding agency; (2) the resources of seven social work graduate students
who served as research assistants in every phase of this labor-intensive
study; (3) an egalitarian attitude and open process that conferred
respect on participants as knowledgeable in their own fields, and (4)
mutual benefits that accrued to practitioners and evaluators.
Language and the maintenance of professional boundaries were
critical to sustaining a collegial relationship and required sensitivity
and vigilance (Quinn Patton, 1997). The term "external"
evaluator was always used with two effects: It clarified roles, and it
set clear expectations for behavior. The term external evaluator
conveyed to all the responsibility and commitment to adhere to established ethical standards of conduct in research methods (NASW,
2000). In a mutually beneficial internal--external exchange,
practitioners permitted the evaluators to get a much-needed
"insider view," and in return, the faculty member was expected
to share cumulative knowledge and skills (Quinn Patton) gained from
previous experience evaluating community-based programs in other parts
of the country.
This resulted in a two-way transfer of knowledge. The evaluators
were catalysts who offered theoretical insights that helped
practitioners examine their work in a reflective, systematic way and who
left behind evaluation tools and conceptual frameworks used by staff in
subsequent grant applications. The staff offered evaluators deep access
to the public housing community and an organizational setting for a
prolonged period. This perspective from the field stimulated evaluators
to critically examine the relevance of existing theories intended to
inform practice, and equally important to consider how practice might
better inform theory.
Conclusion
Community-based programs can increase their organizational capacity
when they engage in the coproduction of evaluation that offers an
ongoing opportunity to critically examine and improve work processes and
products systematically. Logic modeling offers promise as an analytic
framework to help practitioners and evaluators develop baselines, move
toward better outcomes, and monitor program management in a funding
environment that demands increased accountability and in a community
environment that continues to pose challenges, risks, and rewards.
Table 1
Outcomes for Family Center Clients
Completing Job Readiness Workshops
Working Not Working
Outcome No. % No. %
Full-time job 7 29
Full- and part-time jobs 5 21
Part-time job 3 13
Part-time job, school,
and volunteering 1 4
School and volunteering 2 8
Volunteering 5 24
At home 1 4
All 16 67 8 33
NOTE: n = 24 is a subsample of 31 center clients who
completed workshops with 7 no responses.
Table 2
Adapted Logic Model: Customized Paths of Client Service Use and
Movement toward the Program Goals
Variable Tomas Livia
Recruitment Concrete services-needed Aerobics class
furniture (Family (Community
Strengthening) Improvement)
Initial point of Referred by Housing FSC Staff
access (domain) Management
Second service Crisis counseling Crisis counseling
used (domain) (Family Strengthening) (Family Strengthening)
Third service Job-related services Volunteer training
used (domain) (Worker Development) (Education/Life
Skills)
Total number of 17 9
services used
Immediate out- Used emergency services, Used worker develop-
comes support groups, to meet ment services; applied
family's basic needs; to community college;
particpated in educa- applied for financial
tional, worker develop- aid; spouse used
ment courses. Social worker development
support network services. Social
expanded. support network
expanded.
Intermediate Completed classes; Increased family
outcomes volunteered; found full- activities; increased
time job on-site: $8.05/ volunteer participa-
hr., 40 hrs./week, tion; attended
benefits. community college;
incompliance with
welfare reform regula-
tions; part-time job
on-site: $8/hr., 19
hrs./week.
Goals Stable housing; movement Stable housing;
toward family stability movement toward family
and independence. stability and inde-
pendence.
Variable Silah
Recruitment Job-readiness training
(Worker Development)
Initial point of Self
access (domain)
Second service Crisis counseling (Family
used (domain) Strengthening)
Third service Budgeting/money manage-
used (domain) ment (Education/Life Skills)
Total number of 7
services used
Immediate out- Attended worker development
comes classes; used family strength-
ening services. Social support
network expanded.
Intermediate Completed job-readiness class;
outcomes completed volunteer training;
volunteered; increased inde-
pendence and self-confidence;
part-time job in neighborhood
schools: $15.92/hr., 20 hrs./
week.
Goals Stable housing; movement
toward family stability and
independence.
References
Annie E. Casey Foundation. (1997). Evaluating comprehensive
community change. Baltimore: Author.
Alter, C., & Murty, S. (1997). Logic modeling: A tool for
teaching practice evaluation. Journal of Social Work Education, 33,
103-117.
Bickman, L., & Rog, D. (1998). Handbook of applied social
research. Thousand Oaks, CA: Sage Publications.
Bruner, C., & Parachini, L. (1997). Building community.
Washington, DC: Together We Can Initiative.
Chaskin, R. J., Joseph, M. L., & Chipenda-Dansokho, S. (1997).
Implementing comprehensive community development: Possibilities and
limitations. Social Work 42, 435-444.
Chaskin, R., & Ogletree, R. (1995). The Ford Foundation's
family and neighborhood initiative: Building collaboration. Chicago:
University of Chicago, Chapin Hall Center for Children.
Cohen, C. S., & Phillips, M. H. (1997). Building community:
Principles for social work practice in housing settings. Social Work,
42, 471-481.
Connell, J., Kubisch, A., Schorr, L., & Weiss, C. (1995). New
approaches W evaluating community initiatives. Washington, DC: Aspen
Institute.
Conrad, K., Randolph, F., Kirby, M., & Bebout, R. (1999).
Creating and using logic models: Four perspectives. Alcoholism Treatment
Quarterly, 17(1 /2), 17-31.
Coulton, C. J. (1996). Poverty, work, and community: A research
agenda for an era of diminishing federal responsibility. Social Work,
41, 509-519.
Fetterman, D., Kaftarian, S., & Wandersman, A. (1996).
Empowerment evaluation: Knowledge and tools for self-assessment and
accountability. Thousand Oaks, CA: Sage Publications.
Forbes, D. (1998). Measuring the unmeasurable: Empirical studies of
nonprofit organizational effectiveness from 1977-1997. Nonprofit and
Voluntary Sector Quarterly, 27, 183-202.
Greenwood, D. J., & Levin, M. (1998). An introduction to action
research Thousand Oaks, CA: Sage Publications.
Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation
evaluation. Newbury Park, CA: Sage Publications.
Heineman-Pieper, M. (1994). Science not scientism: The robustness
of naturalistic clinical research. In E. Sherman & W. J. Reid
(Eds.), Qualitative research in social work (pp. 71-88). New York:
Columbia University Press.
Huberman, M., & Miles, M. (1994). Data management and analysis
methods. In N. Denzin & Y. Lincoln (Eds.), Handbook of qualitative
research (pp. 428-444). Thousand Oaks, CA: Sage Publications.
Jansson, B. (1994). Social policy: From theory to practice (2nd
ed.). Pacific Grove, CA: Brooks/Cole.
Knapp, M. (1996). Methodological issues in evaluating integrated
services initiatives. In J. Marquart & E. Konrad (Eds.), Evaluating
initiatives to integrate human services (pp. 21-34). San Francisco:
Jossey-Bass.
Krueger, R. (1994). Focus groups: A practical guide for applied
research (3rd ed.). Thousand Oaks, CA: Sage Publication.
Leventhal, T., Brooks-Gunn, J., Kamerman, S. (1997). Communities as
place, face, and space: Provision of services to poor urban children and
their families. In J. Brooks-Gunn, G. Duncan, & L. Aber (Eds.),
Neighborhood poverty (Vol. 1, pp. 182-204). New York: Russell Sage Foundation.
Logan, T., & Royce, D. (2001). Program evaluation. In B. Thyer
(Ed.), Handbook of social work research methods (pp. 193-206). Thousand
Oaks, CA: Sage Publications.
Marquart, J., & Konrad, E. (Eds.). (1996). Evaluating
initiatives to integrate human services. San Francisco: Jossey-Bass.
Maxwell, J. (1998). Designing a qualitative study. In L. Bickman
& D. Rog (Eds.), Handbook of applied social research (pp. 69-100).
Thousand Oaks, CA: Sage Publications.
Miles, M., & Huberman, M. (1994). Qualitative data analysis.
Thousand Oaks, CA: Sage Publications.
Mulroy, E. A. (1997). Building a neighborhood network:
Interorganizational collaboration to prevent child abuse and neglect.
Social Work, 42, 255-264.
Mulroy, E. A., & Lauber, H. (1999). Evaluation of the second
year Federal Demonstration Grant Preventing Homelessness and Moving
Families toward Self Sufficiency: Final report. (grant no. 90EXO057).
Submitted to the U.S. Department. of Health and Human Services, Office
of Community Services. June 7, 1999).
Mulroy, E. A., & Lauber, H. (2002). Community building in hard
times: A post-welfare view from the streets. Journal of Community
Practice, 10(1) 1-16.
Mulroy, E. A., & Shay, S. (1997). Nonprofit organizations and
innovation: A model of neighborhood-based collaboration to prevent child
maltreatment. Social Work, 42, 515-524.
Naparstek, A. J., & Dooley, D. (1997). Countering urban
disinvestment through community-building activities. Social Work, 42,
506-514.
National Association of Social Workers. (2000). Code of ethics of
the National Association of Social Workers. Washington, DC: Author.
O'Connor, A. (1995). Evaluating comprehensive initiatives: A
view from history. In J. Connell, A. Kubisch, L. Schorr, & C. Weiss
(Eds.), New approaches to evaluating community initiatives (pp. 23-63).
Washington, DC: Aspen Institute.
Quinn Patton, M. (1997). Utilization-focused evaluation (3rd ed.).
Thousand Oaks, CA: Sage Publications.
Resource Coalition of America. (1996). Guidelines for family
support practice. Chicago: Author.
Schalock, R. (2001). Outcomes-based evaluation (2nd ed.). New York:
KluwerAcademic/Plenum Publications.
Schuman, C., & Abramson, A. (2000). Collaborating for usable
knowledge: A Work in progress by the Nonprofit Sector Research Fund.
Nonprofit and Voluntary Sector Quarterly, 29(1), 11-23.
Stevenson, J., Mitchell, R., & Florian, P. (1996). Evaluation
and self-direction in community prevention coalitions. In D. Fetterman,
S. Kaftarian, & A. Wandersman (Eds.), Empowerment evaluation:
Knowledge and tools for self-assessment and accountability (pp. 208-233)
Thousand Oaks, CA: Sage Publications.
Strawn, J., & Martinson, K. (2000). Steady work and better
jobs. Washington, DC: Manpower Demonstration Research Corporation.
Tyson, K. (1995). New foundations for scientific social and
behavioral research: The heuristic paradigm. Boston: Allyn & Bacon.
Weil, M. (1996). Model development in community practice: An
historical perspective. Journal of Community Practice, 3(3/4) 5-68.
Weil, M. (1997). Community building: Building community practice.
In P. L. Ewalt, E. M. Freeman, S. A. Kirk, & D. L. Poole (Eds.),
Social policy: Reform, research, and practice (pp. 35-61). Washington,
DC: NASW Press.
Weil, M. (2000). Services for families and children: The changing
context and new challenges. In R. Patti (Ed.), The handbook of social
welfare management (pp. 481-510). Thousand Oaks, CA: Sage Publications.
Weiss, C. (1995). Nothing as practical as good theory: Exploring
theory-based evaluation for comprehensive community initiatives for
children and families. In J. Connell, A. Kubisch, L. Schorr, & C.
Weiss (Eds.), New approaches to evaluating community initiatives
(65-92). Washington, DC: Aspen Institute.
Weiss, J. (2000). From research to social improvement:
Understanding theories of intervention. Nonprofit and Voluntary Sector
Quarterly, 29(1), 81-110.
Witkin, S. (1994). Commentary: Evaluation in context: Is there any
other way? In E. Sherman & W. Reid (Eds.), Qualitative research in
social work (pp. 328-332). New York: Columbia University Press.
Wolch, J. (1999). Decentering America's nonprofit sector:
Reflections on Salamon's Crises Analysis. Voluntas, 10(1), 25-35.
Yegidis, B., & Weinbach, R. (1996). Research methods for social
workers. Boston: Allyn & Bacon.
Yin, R. (1994). Case study research. Thousand Oaks, CA: Sage
Publications.
Yin, R. (1998). The abridged version of case study research: Design
and method. In L. Backman & D. Rog (Eds.), Handbook of applied
social research (pp. 229-260). Thousand Oaks, CA: Sage Publications.
Yin, R., Kaftarian, S., & Jacobs, N. (1996). Empowerment
evaluation at federal and local levels: Dealing with quality. In D.
Fetterman, S. Kaftarian, & A. Wandersman (Eds.), Empowerment
evaluation: Knowledge and tools for self-assessment and accountability
(pp. 188-207). Thousand Oaks, CA: Sage Publications.
Elizabeth A. Mulroy, PhD, is associate professor, University of
Maryland, Baltimore, 525 West Redwood Street, Baltimore, MD 21201;
e-mail: emulroy@ssw.umaryland.edu. Helenann Lauber, BS, is program
director, Family Support Center, Parents and Children Together,
Honolulu, Hawaii. An earlier version of this article was presented at
the Council on Social Work Education Annual Program Meeting, February
28, 2000, New York.
Original manuscript received August 24, 2001 Final revision
received September 1, 2003 Accepted November 18, 2003