Insuring After-School Programs Meet Their Intended Goals - education, United States

Documentation of outcomes associated with after-school program participation is increasing. However, to date, no comprehensive meta-analysis of outcome studies has been undertaken, but several catalogs of outcomes have emerged over the last several years (e.g., Fashola, 1998; Reno & Riley, 1998, 2000). While the results of these outcome studies can be criticized from a number of perspectives, awareness of what is and is not working can assist program providers greatly in designing quality after-school programs.

The studies reviewed in the July 2001 Research Update column provided evidence that after-school programs are offered for three major reasons: (a) need for a safe and secure environment, (b) efforts to increase school achievement, and (c) the necessity of school reform (Reno & Riley, 1998, 2000). However evidence of the impact of programs designed to achieve these objectives needs to be viewed with some caution.

Just what are the problems with these studies? Fashola (1998) undertook a review of outcome studies and concluded that, methodologically, research is at a rudimentary stage. A number of studies fail to meet minimal research design standards. For example, selection bias is a problem leading to failure to account for why some children attend programs and others do not. Random assignment of children who wish to be in a program is difficult from both a practical and political point of view, thus many studies do not meet the required rigor to establish cause and effect between program and outcomes. In addition, many studies rely on retrospective post-test-only designs (notably, however, this practice is widespread in large-scale national evaluations of many social programs).

Many organizations do not have the expertise or financial resources to undertake evaluation beyond post-test only surveys designed to measure program quality. At the same time, few studies have attempted to look at changes beyond a one-year perspective.

In addition, the surveys used have often not been subjected to procedures to determine their reliability or validity. Reports by children, parents or teachers may be biased because of fear that, if no impact is found, programs might be canceled. Given the necessity for after-school childcare in many participating families, parents may overstate impacts simply to preserve the program. Additionally, in a number of cases, outcome data is based on the opinions of experts instead of formal evaluations. Awareness of these flaws in existing outcome studies is an important step towards improving the quality of evaluation down the road.

One factor that makes evaluations more difficult should be considered. Many after-school programs are not designed to meet their stated objectives. For example, a program intended to achieve changes in academic performance might not contain opportunities for homework completion or tutoring or provide appropriate enrichment activities (Witt & Baker, 1997).

A Growing Body of Evidence Supporting After-School Programs

Despite the difficulties outlined above, there seems to be a growing body of evidence that after-school programs can make a difference (see Figure 1 on page 35 for a list of potential programmatic outcomes). For example, decreases in juvenile crime have been associated with participation in a number of different after-school programs (Fox & Newman, 1998; Mclennan Youth Collaboration, 1997; Schinke, Orlandi, & Cole, 1992). Children in after-school programs (versus non-participants) have reported feeling safer than during the after-school program hours (Brooks, et al., 1995), and the Baltimore Police Department reported a 44 percent drop in children becoming crime victims after starting an after-school program in a high crime area (Baltimore Police Department Division of Planning and Research, 1998).

[ILLUSTRATIONS OMITTED]

Better grades and high academic achievement also appear to be associated with after-school programs that contain academic or enrichment components. For example, in 30 of the 40 schools that had children participating in the Chicago Lighthouse Program, children showed achievement gains in reading scores, with 39 schools showing gains in mathematics scores (Chicago Public Schools Office of Schools and Regions, 1998); students participating in the LA's Best after-school program were reported to have made academic gains beyond students not participating in the program (Brooks & Mojica, 1995); and Baker and Witt (1996) reported that the more students were involved in an after-school program, the more academic performance improved. Thirty-month follow-up evaluation of children participating in the Boys & Girls Club's Project Learn indicated an increase in grade average, school attendance, and study skills compared to children not participating in the program (Schinke, Cole, & Poulin, 1998).

A number of studies have also reported increases in school attendance and reduced drop-out rates. McLennan Youth Collaborations (1997) found that 57 percent of students participating in the after-school program improved their school attendance, while Cardenas et al. (1991) reported that absences for students participating as tutors in a cross-age tutoring program was one percent compared to a comparison group's 12 percent rate.

Several studies have reported that students participating in after-school programs have shown fewer behavior problems (Gregory, 1996; Steinberg, Riley, & Todd, 1993), handle conflicts better (Carlisi, 1996; Gregory, 1996), and show improved social skills (Steinberg, Riley, & Todd, 1993; Terao, 1997).

Increases in parental involvement in the lives of their children who participate in an after-school program have also been documented. This outcome is mainly accomplished through either requiring, or strongly encouraging, parents to volunteer in their child's after-school program (Carnegie Council, 1992) or teaching parents how to help their children with homework (Chicago Public Schools, 1998).

After-school program attendance has also been associated with helping children increase their aspirations for the future, including their intention to go to and complete high school. For example, one survey reported that students in after-school activities are more likely to indicate that school is preparing them for college and that they plan to attend university (Peter D. Hart, Research Associates, 1999), while other studies have reported higher graduation rates as a result of after-school participation (e.g., Hahn, Leavitt, T., & Aaron, P., 1994). Finally, at least one study has documented the cost savings ($72,692 over three years) because students participating in an after-school program avoided being retained in grade or being placed in special education (Gregory, 1996).

Good After-School Programs

While there is widespread belief that promising models for after-school programs exist, it is not yet clear which models or elements of models are the most effective (Fashola, 1998). While there is a growing literature on the program elements necessary to produce desired outcomes, efforts are still in their early stages.

Several reports have attempted to identify the critical program elements necessary to bring about desired results. For example, reports by the California Department of Education (1996) and Roman (1998) have indicated that successful programs should offer age-appropriate learning activities, a low student-staff ratio, qualified staff, linkages with the regular school program and with community organizations, safety, a wide range of both structured and unstructured activities, program evaluation, and parent involvement.

Gambone and Arbreton (1997) have identified seven program practices and organizational features hypothesized to be associated with high levels of youth development experiences: providing informal and formal opportunities for leadership; enabling to receive social support from adults; creating a sense of belonging; providing challenging and interesting activities; providing opportunities for input and decision making; providing opportunities for community service; and creating a safe environment.

In addition, Olsen and Scharf (2000) have noted the importance of after-school programs being accessible and available to all by honoring ethnic and cultural diversity needs of participants. They contend that: "Where a young person lives determines what level and quality of services they receive" (p. 1) and that few programs serve an enrollment representative of the ethnic, cultural, and probably socio-economic make-up of the areas served.

Overall, two program elements have emerged as particularly significant: (1) program involvement that has sufficient intensity and duration to bring about desired results, and (2) the availability of caring, committed and qualified adults who can develop in depth, meaningful relationships with participants (Witt & Crompton, 1999).

Intensity and Duration (or "Dosage")

The duration, amount, and intensity of the program, or "dosage," participants receive appears to be directly related to outcomes. For example, Huang, et al. (2000) in a study of the LA's Best program, followed students in a program from 1993-94 school year to 1997-98, and looked at differences in outcomes as a result of high involvement with the program (more than 75% of days present); medium involvement (26 to 74%) of days; and low involvement (25% or less) of days. Their findings suggest that for students who participated over at least 4 years, higher participation was related to positive achievement on standardized math, reading and language arts tests (after controlling for gender, ethnicity, income, and language status). Their results suggest that higher levels of participation led to better subsequent school attendance, which in turn is related to higher academic achievement as measured by standardized test scores.

Feister, White, Reisner, and Castle (1999) report that the impact of an after-school program on cognitive and emotional development was greatest for those students with high rates of average attendance. Baker and Witt (1996) came to similar conclusions regarding academic progress variables.

Thus, besides providing in-depth programming, programs should not be offered for a short period of time and then dropped. This limits programmatic impact and usefulness. Failing to create on-going opportunities for involvement ignores that the need for a safe and secure environment is on-going, five days per week, throughout the school year, during after-school hours; and for many children, during school holidays, teacher days, and summer vacation periods as well.

The Necessity of Caring Adults

The importance of committed and caring adults is emphasized in almost every article written about after-school programs. Kahne, et al. (1999) found that students in a variety of different programs reported preferring the affective context of after-school program to that experienced during the school day. This may be due to school-day programs placing more emphasis on order, discipline, and control, while after-school programs are generally less formal and structured.

However, Kahne, et al. (1999) also suggest that after-school programs may offer more support for youth development by providing frequent contact with other participants and a developmental, rather than a recreational, focus. In addition, programs that provide meaningful support do more than provide safe and pleasant contexts: they are staffed by individuals who reach out to and build trusting relationships with youth, monitor their development, and intervene when necessary.

Good leaders will be able to encourage youth to be physically and mentally engaged, and should be able to actively motivate and appropriately reward acceptable behavior. To accomplish these ends, they should know and understand how to utilize basic principles of child development.

Necessity of Adequate Resources

Programs should have enough resources to do a good job. In many cases providers have not fully understood the costs involved in such an endeavor (Halpern, Deich, & Cohen, 2000). Agreeing to run a program on a shoestring may create the impression that no additional funding is needed. In addition, it may lead to a self-fulfilling prophecy: that the service is either not worth supporting or will be of low quality. Permanent sources of funding are preferable to an endless string of grants and other one-time sources. Political officials (city councils, park and recreation commissions, school boards, etc.) should be lobbied to make a commitment to fund programs as a basic service. Funding should also be at a level that insures an appropriate leader to child ratio (1:15 for children 9 years and older and 1:10 for younger children). In addition, funding should be sufficient to pay salaries at rates that attract and keep committed staff. Low pay and lack of benefits can lead to staff turnover and low quality of employees (Halpern, Spielberger, & Robb, 2000). This, in turn, can impact participants' trust and program outcomes.

After-school programs appear to have great potential to impact the lives of participants. While evaluation information is still sparse, data is increasing that properly designed and implemented programs can achieve a multitude of important goals beyond just keeping kids in a safe and secure environment until their parents come to pick them up.

References

Baker, D. A., & Witt, P.A. (1996). Evaluation of two after-school recreational programs. Journal of Park and Recreation Administration, 14(3), 60-81. http://wwwrpts.tamu.edu/ rpts/faculty/pubs/wittpub3.htm

Baltimore Police Department Division of Planning and Research. (1998). Juvenile victimizations comparison for Goodnow PAL Center area. Baltimore, MD: Baltimore Police Athletic League.

Brooks, P.E. et al. (1995). Longitudinal study of LA's Best after school education and enrichment program, 1992-1994. Los Angeles, California: UCLA Center for the Study of Evaluation, 1995.

Brooks, P.E. &, Mojica, C.M. (1995). Final evaluation report: Longitudinal study of LA's Best after-school education and enrichment program. Los Angeles: Center for the Study of Evaluation, 1995.

California Department of Education (1996). School-age care in California. Sacramento, CA: California Department of Education.

Cardenas, J.A. et al. (1991). The Coca-Cola Valued Youth Program: Dropout prevention strategies for at-risk students. Texas Researcher, 3: 111-130.

Carlisi, A. M. The 3:00 Project Program Evaluation. Decatur, GA: Georgia School Age Care Association, 1996.

Carnegie Council on Adolescent Development. (1992). A matter of time: Risk and opportunities in the nonschool hours. Carnegie Corporation of New York, New York.

Chicago Public Schools Office of Schools and Regions (1998). The McPrep Lighthouse Program. Chicago: Chicago Public Schools

Fashola, O. S. (1998). Review of extended-day and after-school program and their effectiveness. New Jersey: John Hopkins University, Center for Research on the Education of Students Placed at Risk, Report No. 24

Feister, L., White, R.N., Reisner, E.R., & Castle, A.M. (1999). Increasing and improving after-school opportunities: Evaluation results from the TASC after-school program's first year. New York: Policy Studies Associates.

Fox, J.A. & Newman, S. (1998). After-school crime or after-school programs: Tuning into the prime time for violent juvenile crime and implications for national policy. Washington, D.C.: Fight Crime: Invest in Kids.

Gambone, M. A., & Arbreton, A. J. A. (1997). Safe havens: The contributions of youth organizations to healthy adolescent development. Philadelphia: Public Private Ventures.

Gregory, P.J. (1996). Youth opportunities unlimited: Improving outcomes for youth through after school care. Manchester: University of New Hampshire.

Hahn, Leavitt, T., & Aaron, P. (1994). Evaluation of Quantum Opportunities Program (WOP): Did the program work. Waltham, MA: Center for Human Resources, Brandies University.

Halpern, R., Spielberger, J., & Robb, S. (2000). Evaluation of the MOST (Making the Most of Out-of-School Time) initiative. Final Report. Chicago, IL: Chapin Hall Center for Children, University of Chicago.

Halpern, R., Deich, S., & Cohen, C. (2000). Financing after-school programs. Washington, D.C.: The Finance Project. http://www. financeproject.org/financing_afterschool_ programs.htm

Huang, D., Gribbons, B., Kim, K.S., Lee, C., Baker, E.L. (2000). A decade of results: The impact of the LA's Best after school enrichment program on subsequent student achievement and performance. Los Angeles, CA: UCLA Center for the Study of Evaluation. http://www.lasbest.org/learn/uclaeval.pdf

Kahne, J., Nagaoka, J., Brown, A., O'Brien, J., Quinn, T., & Thandiede, K. (July 28, 1999). School and after-school programs as contexts for youth development. Unpublished paper.

McLennan Youth Collaboration, Inc. (1997). Communities in schools case management staff evaluation. Waco, TX: Lighted Schools.

Olsen, L. & Scharf, A. (Fall, 2000). Realizing the promise and opportunity of after school programs in a diverse state: A preliminary analysis of equity and access issues in California's after school programs. Unpublished manuscript. Oakland, CA: California Tomorrow.

Peter D. Hart, Research Associates. (August 25, 1999) The Shell Education Survey Poll. Funded by The Shell Oil Company in Houston.

Reno, J. & Riley, R.W. (1998). Safe and smart: Making the after-school hours work for kids. Washington, D.C.: U.S. Department of Education. Http://www.ed.gov/pubs/SafeandSmart/

Reno, J. & Riley, R.W. (2000). Working for children and families: Safe and smart: making the after-school hours work for kids. Washington, D.C.: U.S. Department of Education. Http://www.ed.gov/pubs/SafeandSmart/

Roman, J. (1998). The NSACA Standards for quality school-age care. Boston, MA: The National School-Age Alliance.

Schinke, S., Orlandi, P.M., & Cole, K (1992). Boys and Girls Clubs in public housing developments: Prevention services for youth at Risk, Journal of Community Psychology, 118-128.

Schinke, S.P., Cole, K.C., & Poulin, S.R. (1998). Thirty-month data and process findings: Evaluation of educational enhancement program of Boys & Girls Clubs of America. Unpublished paper. New York: Columbia University School of Social Work. http://www. financeproject.org/Brief1.pdf

Steinberg, J., Riley, D., & Todd, C. (1993). Preventing problem behaviors and raising academic performance in the nation's youth: The impacts of 71 school-age child care programs supported by the CES youth-at-risk initiative. Illinois: University of Illinois at Urbana-Champaign and the University of Wisconsin Center for Action on the Family.

Terao, K. (1997, Winter). Rural out-of-school times newsletter, 1(1).

Witt, P. A., & Baker, D. A. (1997). Developing after-school programs for youth in high risk environments. Journal of Physical Education, Recreation and Dance, 68(9), 18-20.

Witt, P.A., & Crompton, J.L. (1999, December). A paradigm of the times. Parks and Recreation, 34(12), 66-73, 75.

RESEARCH INTO ACTION: AN EMERGING BODY OF KNOWLEDGE

Introduction

Documentation of outcomes associated with after-school program participation is increasing. However, to date no comprehensive meta-analysis of outcome studies has been undertaken, but recentely several catalogs of outcomes have emerged. A body of knowledge is emerging indicating that after-school programs can bring about changes in a number of areas, including: crime reduction (by creating positive spaces for children to be during the after-school hours); an increase in positive behaviors (including improved social skills, ability to handle conflicts, decreasing use of drugs and alcohol); and increased self-confidence. Positive relationships have also been found between after-school program involvement and increased school performance, including interest in going to school and school achievement. Academic-related outcomes are dependent on creating a program that has strong homework completion and enrichment components. Finally, after-school programs can have impacts on strengthening schools, families and communities, and increasing parents involvement in their children's education.

Impact of this Research

Quality evaluation and documentation is critical both to help justify programs to funding sources and to facilitate program improvement. While existing evaluation studies have a variety of methodological problems, there is clear evidence that properly planned programs (e.g. those that include the appropriate amount of involvement of children with the program and quality leadership) have the potential to make critical differences in the lives of children, their family, and their community.

How to Use this Research

If programs are not able to generate their own evaluation data, there is body of knowledge that can be used to justify program development. However, communities should make evaluation an important part of their programmatic efforts to both add to existing knowledge about program outcomes and to create local information that will strongly resonate with local funding sources and stakeholders. Other suggestions can be made based on this research:

* Program designers should also be aware of the relationship between program design and program outcomes. Purposively planned programs have the best chance of achieving results. Program goals should be carefully selected in concert with the schools and parents. Subsequently, programs can be designed with specific components to meet the pre-determined goals. The figure included in this research update provides a comprehensive list of typical program outcomes.

* Evaluations should be designed to specifically address goals and program components. Consider partnering with a university program for assistance with evaluation during the program planning stage.

* Programs should be both in-depth and long-term to meet children's need for a safe, secure environment. This means planning for five days a week, plus vacations and holidays when some parents will have to work. Programs should strive to find permanent funding; inform stakeholders about your plans and form partnerships to build lasting financial support.

* Committed, caring adults are one key to program success. The importance of hiring, training and retaining quality staff is key. The children will benefit most when staff are committed and able to reach out and build trusting relationships with youth. Parent involvement can be encouraged through volunteer opportunities.

* Providers need to consider the ethic and cultural diversity of the areas they serve and proactively ensure that programs serve all members of the community.

FIGURE 1: OUTCOMES ASSOCIATED WITH AFTER-SCHOOL PROGRAMS(*)

I. Safe and Secure Environment

Crime reduction

* decreased juvenile crime

* decreased violent victimization

* decreased vandalism at schools

Prevention of negative behaviors and increase of positive behavior

* preventing negative influences that lead to risky behaviors such as
drug, alcohol and tobacco use

* decreased television watching

* fewer behavioral problems generally

* increased ability to handle conflicts

* improved social skills

* improved self confidence

II. School Performance

* increased grades and higher academic achievement

* increased interest in and ability to read

* improved school attendance and reduced dropout rate

* turning in more and better quality homework

* increased time on task

* reduced retention in grade and placement in special education

* improved school behavior

* monetary savings to school districts

* development of new skills and interests

* higher aspirations for the future, including intention to complete
high school and to go to college.

III. School Reform

* strengthening schools, families and communities

* greater family and community involvement in children's
learning and schools

* increased parental involvement in schools

* development of community schools

(*) Areas provided by Reno & Riley (1998, 2000) have been placed
under the three main categories by the author of this paper, with
the first category further divided into two subcategories.

Editor's Note: This research update is Part 2 of a 2 part series on after-school programs. Part 1, published in the July Research Update examined the role of park and recreation departments in after-school programs.

Peter A. Witt is the Elda K. Bradberry Recreation and Youth Development Chair at Texas A & M University. He has undertaken numerous evaluation studies of after-school programs.

Research Update is edited by Cheryl A. Estes, Ph.D., assistant professor in recreation and leisure studies at East Carolina University in Greenville, North Carolina.

COPYRIGHT 2001 National Recreation and Park Association
COPYRIGHT 2001 Gale Group