The Six Year Test - National Consortium for Recreation and Youth Development evaluates program effectiveness

PROGRAMS FOR AT-RISK YOUTH GO UNDER INVESTIGATION

For the past six years a group of academics, graduate students and practitioners have been involved in a unique effort to evaluate the outcomes of recreation programs for at-risk youth. The National Consortium for Recreation and Youth Development (originally called the National At-Risk Youth Recreation Consortium), a cooperative effort of faculty and graduate students from eight universities, along with practitioners from approximately 15 cities across the United States, was created with funding from the National Recreation and Park Association (NRPA) and National Recreation Foundation (NRF). Funding was made available and the Consortium created out of a concern that park and recreation departments were not commanding deserved respect and attention from political leaders and funders who were trying to find ways to deal with high levels of juvenile crime, drug use, school dropouts and failure, and teen pregnancy.

In the early 1990s, while money was forthcoming to deal with increasing youth risk behaviors--and the life-long consequences of these behaviors--park and recreation professionals were increasingly being asked to provide evidence that the programs they were undertaking made a difference in the lives of youth. Institutional structures such as schools, law enforcement agencies, churches and various non-profit youth serving organizations were being called upon to provide programs and services that would help to ameliorate these problems. Park and recreation departments wanted to be major players in these efforts, but they did not have evidence of their effectiveness that could justify funding. In many instances, recreation was perceived by funders as fun and games--activities without impact, designed to keep children busy. Park and recreation professionals correctly saw that in order to be players in a youth services system, they would have to change public perceptions about the value and potential impact of park and recreation programs. In order to accomplish this goal, the park and recreation field would have to do a better job of telling its story and produce the kind of information necessary to make the story credible. This latter step would require greater focus on documenting outcomes and contributions being made by park and recreation programs. While the park and recreation community toted the value of late night, after-school, weekend and other non-school-hour programs, little evidence existed beyond anecdotes that park and recreation programs made a difference.

In response to the need for improved documentation, two NRPA-funded three-year grants were received from NPRA/NRF. For the first three years (1993-1996), four universities were involved: Texas A&M, Arizona State (West and Main campuses), Clemson, and Penn State. For the second three years (1996-1999), these institutions were joined by Illinois, Georgia, North Carolina, and San Francisco State.

An initial report on studies undertaken by the Consortium appeared in the Journal of Park and Recreation Administration (JPRA: Witt & Crompton, 1996) and was reprinted in Parks and Recreation (P&R: Witt & Crompton, 1996). Many of the outcome reports published from 1993 to date are available on the Consortium's web site: wwwrpts.tamu. edu/witt/consort.htm. More will be added as projects are completed. Additional articles are scheduled to appear in both P&R and JPRA.

Evaluation Challenges

The research teams have faced several challenges while undertaking the evaluation efforts. A few issues have been prominent in almost every evaluation that has been undertaken. First, evaluation is dependent on a clear set of expected or desired outcome. However, programs often have no clearly articulated goals or have goals that do not lend themselves; easily to measurement. Secondly, even when goals have been articulated, programs are not always designed to effectively achieve them. And lastly, appropriate means are not available to measure whether certain targeted outcomes have been achieved.

Need for Clearly Delineated Goals

Our work suggests that many park and recreation programs are loosely constructed efforts to occupy children's time without a concrete plan for increasing skills (e.g., interpersonal, activity) or helping to develop or change attitudes necessary to decrease risk behaviors. While children and leaders report that participants have fun, this diversionary approach to service provision does not fully meet the expectation of funders who have more lofty crime prevention, risk behavior reduction or improved school performance goals in mind.

Need for Programs that Meet Stated Goals

In many communities, there still is a belief that any program developed and implemented will achieve a wide variety of outcomes. While NRPA has promoted the benefits of recreation participation, these benefits will probably only occur in programs designed to achieve them. In many instances, the benefits catalog is quoted without an understanding that programs must have goals consistent with the stated benefits and be designed to achieve the desired outcomes.

To deal with some of these issues, Allen and his colleagues have promoted Benefits-Based Management (BBM: Allen, Stevens, & Harwell, 1996; Allen & McGovern, 1997) and implemented train-the-trainer workshops to create wide acceptance of the BBM approach and the skills necessary to implement BBM. (In more recent publications, the BBM process has been relabeled as Benefits-Based Programming, (BBP) c.f., Hurtes, et. al., 2000). Unfortunately, to date, practitioners have not widely embraced these concepts, perhaps because a support system does not exist to get practitioners to the training and/or practitioners do not fully understand the importance of undertaking the process.

Given these issues, in some cases project teams were not able to begin the evaluation process without working with practitioners to articulate program goals and then make sure programs were designed to achieve the stated goals. For example, the Texas A&M evaluation team worked with program personnel in Austin to develop a matrix of program goals, program methods and evaluation strategies. Figure One on pages 92-93 provides an example of the matrix developed for the Neighborhood Teen Program, sponsored by the Austin Parks and Recreation Department. The matrix was a direct outgrowth of cooperative efforts between the university evaluation team and program staff from Austin.

There is some evidence that practitioners' resistence to setting goals and planning programs to meet goals may be changing. Through dissemination of the BBP process and what has been labeled "Programming for Benefits, Evaluating for Results" by members of the Consortium, presentations at conferences, and the availability of published materials, there appears to be increased interest in a more formal approach to developing and achieving program goals. But much work remains to be done.

Identifying appropriate outcome measurement instruments has been another issue that has arisen during the process of developing evaluation designs. In a number of cases, appropriate instruments based on underlying theories of youth development do not exist. Witt & Crompton (1997), Hurtes, et. al (2000), and Baldwin (2000) all have called for the development of models to guide our understanding of the basic needs of youth, what it means to be at-risk, and the potential role of protective factors and resiliency in youth development.

There have also been several lessons learned about program design from the evaluation projects undertaken to date. Four seem particularly worthy of mention:

* the need to involve participants to have a sufficient amount of contact with the program to achieve desired outcomes

* the need for consistency in leadership

* the need to build capacity in local communities to further program planning and evaluation efforts

* the need to involve youth in decision making.

Insuring Enough Program Involvement to Make a Difference

Even when a program is well designed to achieve particular outcomes, participants must be involved enough in the program in order for an impact to take place. If one is sick with the flu, the doctor prescribes a particular dosage of medication and urges that the medication be taken a sufficient length of time to make a difference. Most of the goals that recreation programs seek to achieve are long term, yet programs are often offered for only a few weeks, with participants only participating a few hours per week. There neither is a sufficient strength of dosage nor a long enough period of taking the "medicine" to achieve a demonstrable change in behavior, skills or attitudes. Thus, although we plan programs, each needs to be part of an overall system of services designed to help youth achieve desired outcomes over the long term.

Consistent with the idea of sustained program involvement is the need for longer term evaluation efforts. Too often evaluations have been program-centered rather than youth-centered. We have tended to look at outcomes of a specific short term program, rather than focusing on the needs of specific youth and what involvements each needs over time to be successful. For most youth, risk behaviors took a long time to develop and may take a long time to change. It is also important to understand the knowledge, skills and attitudes that youth need to be more resilient to risk factors in the first place.

Finally, youth are not just involved in programs sponsored by park and recreation departments. They go to school, have contact with health agencies and live in a community that influences their everyday behavior and attitudes. As a consequence, what recreation and park departments do must be part of a larger system of services and evaluation strategies must be designed that can take account of the myriad services or experiences that any one youth might have.

Consistency in Leadership is Key

Leaders also need to be involved with youth over a long period of time. Since most programs rely on developing strong relationships between leaders and youth, constantly changing leaders does not provide the opportunity to build strong mentoring relationships (Witt & Crompton, 1999). Turnover among leaders can be the result of several actions taken by park and recreation departments. Low salaries (often without health and retirement benefits) can lead practitioners to seek employment elsewhere or be anxious to move up in a department to achieve better compensation rates and access to benefits. Failure to remunerate those we rely on to make a difference in the lives of youth saves money in the short run, but may ultimately undermines the effectiveness of the work we are trying to accomplish. As noted by Witt & Crompton (1999): "the national movement to raise teachers' salaries has not extended to other professionals working with youth, who effectively have been relegated to second class status. Since many youth workers do not have college degrees, ways have to be developed to compensate them for the talents, competencies and importance of the work they do, as opposed to the degrees they hold."

Too often the best face-to-face workers are promoted to managerial positions in order to secure salary increases. Too many youth workers quit the field when they are confronted with the need to choose between their idealism and the reality of earning a higher salary to support their own families.

In addition to low pay, unstable funding also besets many programs. For youth, cancellation or curtailment of a program can undermine trust. The withdrawal of funding when trust and a mentoring relationship have been established becomes another broken promise in the life of a child.

The Need to Build Capacity For Program Planning and Evaluation Efforts

In most park and recreation departments, resources are stretched so thin that adequate time for program planning and development of evaluation strategies is not available. Many practitioners become so busy with the "doing" that there is little time to reflect on the rationale, processes, and strategies necessary to achieve goals. Evaluation is often thought of only at budget time when indicators of success are expected by local funders (government, foundation, etc.). It would seem appropriate that larger cities create a position for an evaluation specialist who can work with programs on a year-round basis to establish evaluation procedures. The expenditure of salary and support funds for this type of position should pay for itself in sustained or additional funding available to those who do a good job of documenting outcomes. Hiring university personnel to take on evaluation efforts is also an option, but in the long run, the major urban programs need to develop their own on-going capacity to undertake this type of work.

Whether departmental, university or some combination of personnel are involved, a fundamental change in the culture of most park and recreation departments must occur if evaluation efforts are to be successful. The experience of many Consortium members, as they attempted to work with practitioners to conduct evaluation efforts, was that keeping good records (e.g., attendance, surveys, program descriptions) was not a priority among practitioners. In some cases, agreements to undertake evaluations were made with senior administrators without buy-in from the front-line personnel who actually needed to do the work. Many practitioners are so caught up in the necessity of developing and putting on programs, that they have little time for collecting information about the programs they are offering. Again, this suggests the need for personnel dedicated to these efforts and a fundamental change in culture, and possibly the reward system, that would make evaluation activities more important in the everyday responsibilities of practitioners.

Need to Involve Youth in Decision Making

Youth need to be empowered as positive actors in making decisions and planning services that impact their lives. This principle is one of the cornerstones of the youth development philosophy. Program process may be just as important program content. Our experience also suggests that youth can also be involved in the research process through helping to collect and interpret data.

Consortium Accomplishments

The Consortium has been a unique approach for dealing with one of the important issues in the park and recreation field. The consortium has had a number of successes and some notable problems. Among the successes has been the development of additional outcome information across a wide range of settings. Dissemination is taking place through presentations at conferences, publications and through the Consortium web site.

There is also evidence that Consortium efforts helped elevate concern for evaluation and documentation in the park and recreation field. Through articles published in journals and presentations made at conferences, there appears to be a growing concern for evaluation. Consortium members have also been successful in developing methodologies and instruments that may be useful to park and recreation departments and other academics seeking to undertake evaluation efforts. Another spin-off of the Consortium effort has been the involvement of a number of graduate students in the evaluation process. These individuals will help expand the number of university personnel and practitioners interested in and capable of undertaking evaluation projects.

Evaluation Methods

Consortium members, in completing the various evaluation projects, adopted a number of different strategies. Both quantitative (e.g., surveys, examination of attendance records) and qualitative (e.g., interviews, observations) data collection processes were used. In general evaluation, efforts included components that answered the following six groups of questions:

Workload: Who did the program serve (age, ethnicity, gender, income level, etc.)? How many youth attended the program? When and for how long did they participate? What percentage of program capacity was utilized (if program could handle 20 youth at each of 20 sites, how many youth were actually registered)?

Costs: How much did it cost to put on the program?

Cost per Unit of Service: How much did it cost for each hour (day, total length of program) of service per youth (total cost divided by appropriate work-load measure)?

Quality: How did youth (and/or parents or other stakeholders) evaluate the quality of program leaders, facilities, advertising, registration processes, etc.?

Satisfaction: How satisfied were youth (and/or parents or other stakeholders) with the program? Would youth sign up again? Would they recommend the program to a friend?

Outcomes: Did the program achieve stated goals?

In some of the evaluations all of the components were included. In others, concentration of effort was on only a few of the components. In all cases, however, efforts to determine workload and outcomes were included.

Overall Lessons Learned

The art and practice of evaluation is not easy. It will take a long time to fully inculcate evaluation as part of the professional culture within the park and recreation field. Nonetheless, the demand for accountability will continue to drive the need to increase and improve current evaluation practices. At a recent conference presentation, one practitioner asked for the bottom-line, what should he be doing to better document his programs? The answer was similar to the passenger who asked the New York taxi driver how to get to Carnegie Hall, to which the Taxi driver responded: "Practice, practice, practice."

For both university and park and recreation department personnel, there is a necessity to become more committed to the need for and value of evaluation efforts, learn and develop methods for undertaking evaluations, and finally, share the results of evaluation efforts. While the Consortium projects provided recognition within the field that evaluation is possible, much work remains to be done. There is need for more long term evaluation, park and recreation department commitments to funding evaluation efforts, and improved understanding of the relationship between goal setting, programming and evaluation. While external demands for accountability demand greater attention to evaluation, a professional demands that we know if what we are doing makes a difference.

FIGURE ONE. NEIGHBORHOOD TEEN PROGRAM, AUSTIN PARKS AND RECREATION DEPARTMENT

PROGRAM OBJECTIVES AND MEANS PROGRAMS ARE USING TO ACHIEVE OBJECTIVES

Objective                        Means for Achieving Objective

To motivate youth to stay        * providing tutoring
in school and maintain           * show interest and discuss
passing grades.                    advantages of youth staying in
                                   school
                                 * provide points for getting good
                                   grades and being in school

To provide a setting where       * create a positive, supportive
youth can feel a sense of          club environment
belonging, be off the streets    * provide mentoring and
and in a positive, supportive      interaction with other
environment.                     * provide opportunities for teens
                                   to be responsible
                                   and make positive choices
                                   through leadership the teen club

To increase participants'        * provide alternatives to feeling
ability to make positive           a sense of belonging without
choice about issues such as        having to join a gang
avoiding drug and alcohol
usage, avoiding gangs,
avoiding pregnancy, and to
utilize services of outside
organizations to provide
information about drug and
alcohol abuse and safe sex/
abstinence practices.

To contribute to participants'   * teach job-related skills
personal growth and job          * provide opportunities for teens
readiness.                         to hold jobs and learn
                                   appropriate job-related skills

To teach youth positive means    * provide opportunities to
for resolving conflicts.           discuss and find positive
                                   alternatives for resolving
                                   conflict situations

To increase trust and respect    * provide opportunities for youth
for other teens, adult             to interact with positive adult
mentors, and other authority       role models (e.g., center staff
figures.                           and other adult community
                                   leaders)
                                 * provide opportunities for youth
                                   to interact with fellow teens in
                                   an environment that encourages
                                   respect and trust

To provide opportunities for     * provide opportunities for youth
new experiences in order to        to experience new recreation
increase participants'             activities
recreation and job choice        * provide volunteers and other
repertoire.                        community service opportunities

To provide opportunities for     * provide adult mentors to talk
youth to gain help with            with teens about difficult
difficult personal and family      issues
issues.                          * When issues are beyond staff
                                   capacity to provide guidance,
                                   provide referral to appropriate
                                   organizations and agencies that
                                   can assist youth

To encourage cultural            * provide opportunities to
diversity awareness                interact with teens from other
                                   recreation centers/ethnic and
                                   cultural backgrounds.

Objective                        Means for Measuring Objectives

To motivate youth to stay        Protective Factor Subscales:
in school and maintain           * Value on Achievement
passing grades.                  * Positive Attitude Toward the
                                   Future

                                 Other Measures:
                                 * Improved school attendance
                                 * Improved grades
                                 * Attend tutoring sessions on a
                                   regular basis

To provide a setting where       Protective Factor Subscales:
youth can feel a sense of        * Neighborhood resources
belonging, be off the streets    * Sense of acceptance and
and in a positive, supportive      belonging
environment.
                                 Other Measures:
                                 * repeat attendance
                                 * efforts to recruit other teens
                                   to join the program

To increase participants'        Protective Factor Subscales:
ability to make positive         * Models for conventional behavior
choice about issues such as
avoiding drug and alcohol        Other Measures:
usage, avoiding gangs,           * Express negative attitudes
avoiding pregnancy, and to         toward use of alcohol/drugs
utilize services of outside      * Amount of time spent with youth
organizations to provide           undertaking conventional
information about drug and         behavior
alcohol abuse and safe sex/      * Increased participation in teen
abstinence practices.              programs

To contribute to participants'   Protective Factor Subscales:
personal growth and job          * Neighborhood resources
readiness.                       * Value on achievement
                                 * Positive attitude toward the
                                   future

                                 Other Measures:
                                 * Ability to fill out a job
                                   application
                                 * Knowledge of where to look for
                                   a job
                                 * Successful undertaking of job
                                   opportunity when available

To teach youth positive means    Protective Factor Subscales:
for resolving conflicts.         * Ability to Worth with Others
                                 * Ability to Work out Conflicts

To increase trust and respect    Protective Factor Subscales:
for other teens, adult           * Interested and Caring Adults
mentors, and other authority     * Ability to Work with Others
figures.                         * Ability to Work out Conflicts
                                 * High Controls Against Deviant
                                   Behavior

                                 Other Measures:
                                 * Teen rating of knowledge,
                                   teaching ability and preparation
                                   of teen leader

To provide opportunities for     Protective Factor Subscales:
new experiences in order to      * Neighborhood resources
increase participants'           * Value on achievement
recreation and job choice        * Positive attitude toward the
repertoire.                        future

                                 Other Measures:
                                 * Participation of teens in new
                                   recreation activities, volunteer
                                   and other community service
                                   activities

To provide opportunities for     Other Measures:
youth to gain help with          * Attendance at self/
difficult personal and family      self-awareness programs
issues.                          * Asks staff for help or where to
                                   go for assistance

To encourage cultural            Other Measures:
diversity awareness              * Willingness to interact with
                                   teens from various neighborhoods
                                 * Positive statements about the
                                   value and importance of
                                   diversity

References

Allen, L. R., Stevens, B., & Harwell, R. (1996). "Benefits-Based Management activity planning model for youth in at-risk environments." Journal of Park and Recreation Administration, 14(3), 10-19.

Allen, L. R., & McGovern, T. D. (1997). "BBM: It's working." Parks and Recreation, 32(8), 48-55.

Baldwin, C.K. (2000). "Theory, program and outcomes: Assessing the challenges of evaluating at-risk youth recreation programs." Journal of Park and Recreation Administration, 18(1).

Hurtes, K.P., Allen, L.R., Stevens, B.L., & Lee, C. (2000). "Benefits-based programming: Making an impact on youth." Journal of Park and Recreation Administration, 18(1).

Witt, P.A., & Crompton, J.L. (1996). "The at-risk youth recreation project." Journal of Park and Recreation Administration, 14(3), 19. (Reprinted in Parks and Recreation, 32(1), 54-61).

Witt, P.A. & Crompton, J.L (1997). "The protective factors framework: A key to programming for benefits and evaluating for results." Journal of Park and Recreation Administration, 15(3):1-18.

Witt, P.A. & Crompton, J.L. (1999). "Youth recreation services: embracing a new paradigm for the new millennium." Parks and Recreation.

Also in this issue, Witt leads us through evaluating several programs for at-risk youth. Witt explores the need for goal-oriented programs, strengthened participation and consistency in leadership. Witt found that at-risk youth perform at a higher rate when they trust their leaders and touched on why leaders often leave their positions: "Too many youth workers quit the field when they are confronted with the need to choose between their idealism and the reality of earning a higher salary to support their own families."

COPYRIGHT 2000 National Recreation and Park Association
COPYRIGHT 2001 Gale Group