Evaluating wraparound services for seriously emotionally disturbed youth: pilot study outcomes in Georgia.

Copp, Hilary L. ; Bordnick, Patrick S. ; Traylor, Amy C. 等

In the United States, nearly 21% of children and adolescents aged 9-17 are diagnosed with a mental health or addictive disorder that causes at least minimal impairment in daily functioning (U.S. Department of Health and Human Services [USDHHS], 1999). The most prevalent diagnoses are anxiety disorders (13%), disruptive disorders (10.3%), and mood disorders (6.2%) (USDHHS, 1999). Approximately 9-13% of all children and adolescents (6-9 million) are diagnosed with a severe emotional disturbance [SED] (Friedman et al., 1996), a legal term defined in 1993 by the Substance Abuse and Mental Health Services Administration [SAMHSA] as:

 Persons from birth up to age 18 who currently or at any time during
 the past year had a diagnosable mental, behavioral, or emotional
 disorder of sufficient duration to meet diagnostic criteria
 specified within the DSM-III-R, and that resulted in functional
 impairment which substantially interferes with or limits the
 child's role or functioning in family, school, or community
 activities (p. 29425).

These children may have legal, school, family, and social problems and require services from a number of different providers, but families are often left without the proper resources to manage the disorders or provide care for their child(ren) across service agencies. Navigating the many agencies providing services can be a daunting or impossible task for parents and caregivers. This approach leads to gaps in care and results in a lack of cohesion of services, leaving families without necessary resources. Given these serious issues, families of children with SEDs are in need of a comprehensive system of care to provide coordination of care.

In response to this need for coordination, the Substance Abuse and Mental Health Services Administration (SAMHSA) in the U.S. Department of Health and Human Services initiated the Comprehensive Community Mental Health Services for Children and Their Families Program in 1992 so as to provide grants to establish local systems of care for children with SEDs and their families in select communities throughout the U.S. and its territories; to date, 92 programs have been funded (USDHHS, 2004). The system of care model is based upon four primary concepts: that services should be family-centered and strengths-based; that services should be collaborative and community-based; that services should be culturally appropriate; and that families should be partners in the system of care (USDHHS, 2004). In 2001, SAMSHA awarded one such grant to a site in Georgia, where services were intended to serve families in three rural counties.

The SAMSHA Wraparound System of Care model is the largest mental health initiative ever undertaken in the United States, in terms of financing and scope. It has been widely adopted to deal with children's emotional problems, educational deficiencies, and juvenile delinquency issues (e.g., Hansen, Litzelman, Marsh, & Milspaw, 2004; Huffine, 2002; Walker & Schutte, 2004) and is beginning to have an impact on graduate training programs in the human services (e.g., Herrick, Arbuckle, & Claes, 2002). Relatively few empirical evaluations of comprehensive wraparound services appear to have been published, which is a conspicuous omission given the considerable public resources that have been expended on promulgating this model. Those that have been conducted seem to find comparatively few improvements among those children and families receiving wraparound services in comparison to those exposed to treatment-as-usual care available via the public educational, social service, mental health, and juvenile justice systems (Bickman, Smith, Lambert, & Andrade, 2003; Carney & Buttell, 2003).

This paper focuses on three primary objectives of the initial evaluation efforts for the Georgia site. The first objective was to examine the feasibility of implementing a comprehensive computer-based assessment protocol used to evaluate participant functioning. The second objective was to conduct an evaluation of children and families enrolled in the system of care at intake and at 6 months. The third objective was to provide empirically based feedback to the system of care service providers in order to improve program services and evaluation efforts.

METHOD

Procedures

To complete the first objective, laptop computers were loaded with standardized evaluation/instrumentation software developed by Macro International Inc., a private program-evaluation company that is the primary contracted evaluator for the national wraparound demonstration program sites. All laptop computers were synched with a web-based data upload system (Quick SAT Version 3.1, Macro International Inc.), which allows sites to upload evaluation data, track completion status, correct errors, receive feedback on progress of data collection, and download individual and cumulative reports. Three graduate-level social workers were trained in the use of the laptop-based Macro International software, interview protocol, clinical assessment instruments, and data upload procedures. Upon completion of training, all interviewers were prepared to conduct field interviews for the evaluation project.

Once the technological and training aspects of the assessment system were in place, the second objective, conducting an evaluation of families and children, was initiated. Children were referred to the local system of care via schools, the Georgia Department of Juvenile Justice, mental health agencies and other community services. Children and families who met the system criteria for enrollment were enrolled into the system of care. Contact information and signed release forms for these families were then forwarded to the local university-based evaluation team. A member of the evaluation team then contacted the caregiver to acquire consent to enroll the family in a long-term outcome study, with interviews to be conducted at baseline and every 6 months thereafter with the primary caregiver and separately with the child (if at least 11 years old). Eligibility requirements for the long-term study included caregiver consent, youth assent (if 11+), enrollment in the system of care, and child's age between 5 and 17.5 years at baseline. Families were scheduled to be interviewed every 6 months out to 24 months after baseline. Families were dropped from the study at the caregiver's request or after two consecutive data collection points were missed due to inability to contact the family, caregiver refusal, family moving out of the area, or other experience that prohibited data collection. All study procedures were approved by the University of Georgia Institutional Review Board, and adhered to the national assessment and evaluation protocol developed by Macro International, Inc.

Interviews were conducted at the family's home and took between 2 and 4 hours to complete. Interviewers utilized laptop computers to administer 9 assessment instruments and descriptive questionnaires to caregivers at baseline (11 at follow-up), and when appropriate, 5 instruments to youth at baseline (6 at follow-up). Subjects were paid $10/hour for interview time. This paper reports data from 2 of the caregiver instruments administered as part of both the baseline and 6-month interviews.

Participants

A total of 45 children were enrolled in the Georgia wraparound system of care demonstration study between October 2001 and May 2004. Complete baseline and 6-month follow-up data reported by caregivers are available for only 15 children, whose data are reported in this paper. Of these 15 children, 8 (53.3%) were male and 7 (46.7%) are female. Six (40%) are African American, 8 (53.3%) are White, and 1 (6.7%) Mexican). Seven of the 15 (46.7%) were between ages 8 and 10 years at the time of their baseline interview and therefore ineligible for a youth interview. The remaining 8 (53.3%) were between ages 11 and 14 years at baseline. The mean age at intake was 10.5 years.

Participants reported between 2 and 23 total problems leading to referral to the system of care. One participant (6.7%) reported 23 problems; 5 participants (33.3%) reported 10-15 problems, and the remaining 9 participants (60%) reported 2-8 problems. Those most frequently reported were attention difficulties (73.3%), hyperactive-impulsive (66.7%), non-compliance (60%), poor self-esteem (60%), physical aggression (53.3%), and poor peer interaction (53.3%).

Outcome Measures

The Child Behavior Checklist for Ages 4-18 (CBCL; Achenbach, 1991) is a 113-item questionnaire administered to the child's caregiver. The caregiver indicates whether the child displays certain behaviors, such as "Cruel to animals," "Feels worthless or inferior," and "Not liked by other kids" utilizing a 3-point Likert scale (2-very true or often true; 1-somewhat or sometimes true; 0-not true) (Achenbach & Edelbrock, 1983). The scale scores 3 Internalizing syndromes (withdrawn, somatic complaints, anxious/depressed, 2 Externalizing syndromes (delinquent behavior, aggressive behavior), and 3 neither Internalizing nor Externalizing syndromes (social problems, thought problems, attention problems) (Achenbach, 1999). Raw scores range from 0-236 and T scores range from 23-100. Scores of 64 and higher on the Total Problem, Internalizing (total), or Externalizing (total) are considered to be in the clinical range; scores of 60-63 are in the borderline clinical range, and scores of 59 and less are considered to be in the normal range (Heflinger & Simpkins, 2002). In this paper only the Total Problem, total Externalizing and total Internalizing scores are presented. The CBCL has been demonstrated to have high test-retest and inter-rater reliability and construct and criterion-related validity (Achenbach & Edelbrock, 1983).

The Child and Adolescent Functional Assessment Scale (CAFAS; Hodges, 1994), used with children ages 7-17 years, allows clinicians or researchers to rate a child's level of impairment across various aspects of functioning (school/work, home, community, behavior toward others, moods/emotions, self-harmful behavior, substance use, and thinking) based upon the child's worst behavior during the evaluation period (e.g., previous 3 months, 6 months). These 8 subscales are scored from 0-30 (0-minimal or no impairment; 10-mild impairment; 20-moderate impairment; 30-severe impairment). A total score (0-240) is also returned, with 0-10 indicating minimal or no impairment, 20-40 indicating that outpatient treatment is probably sufficient, 50-90 indicating that additional services beyond outpatient may be necessary, 100-130 indicating that more intensive care is probably necessary, and 140 and higher indicating that intensive care is probably necessary in combination with risk and community resource assessment (Hodges, 1994).

The CAFAS has been demonstrated to have high test-retest and inter-rater reliability (Hodges & Wong, 1996; Hodges, 1999; Hodges, Doucette-Gates, & Liao, 1999) content and face validity (Hodges, 1999), and criterion-related and construct validity (Hodges & Kim, 2000; Hodges & Wong, 1996; Hodges et al., 1999; Hodges, 1999). The CAFAS may be used both to evaluate individual consumers' impairment and as a measure of treatment outcome (Hodges, 1999; Hodges & Kim, 2000; Hodges et al., 1999).

RESULTS

SPSS version 10.0 was used to run all statistical analyses (SPSS, 2001). Mean scores and standard deviations are presented in Table 1 for the CAFAS and CBCL subscales and total scores at intake and 6-month follow assessments. Paired t-tests examined potential differences on these variables at the two points in time, with a Bonferroni correction used to calculate a more conservative p-value, due to the large number of t-tests conducted. No significant differences were found on any variable, indicating neither improvement nor deterioration in participant functioning over this time period.

DISCUSSION

Our results failed to demonstrate any meaningful changes on outcome measures during the first six months of participation in comprehensive wraparound services among 15 participating children and their families in rural Georgia. Relatively high participant attrition caused by either our inability to recontact these families after six months, their refusal to be re-assessed, or their dropping out of the program, makes the representatives of our 15 participants (out of an initial cohort of 45) a potentially confounding variable. An additional factor is our service program's failure to systematically assess adherence to the guiding principles of wraparound services. This means that our failure to find predicted improvements in child functioning post-intervention cannot be unambiguously attributed to deficiencies in the wraparound model of care, in the absence of assessing fidelity to the treatment model itself. While our service providers were well-trained in wraparound services, and made every effort to follow such comprehensive system-of-care guidelines, formal assessments of treatment fidelity were not obtained, which introduces the possibility that the independent variable, wraparound services, may not have been provided in an effective manner. We recommend that future program evaluations of wraparound services make use of some system of formal fidelity checks, such as that developed by Bruns, Burchard, Suter, Leverentz-Brady, and Force (2004), as a potential means of controlling for this confounding variable. Thus our failure to find improvements in participant function may be due to actual deficiencies in the overall model of wraparound services in living up to its hypothesized effectiveness, or to the methodological limitations in our preliminary study of outcomes.

The goals of this evaluation focused on implementing and conducting a comprehensive computer-based evaluation program and on providing empirically based feedback to the agency providing wraparound services. The first goal was to examine the feasibility of implementing a comprehensive, computer-based assessment protocol with severely emotionally disturbed children and their families. We found that it was feasible, albeit with very high attrition rates. The second goal was to conduct an outcome study of child functioning after receiving six months of wraparound services, using standardized assessment instruments. This was accomplished with 15 participants. Our informal observations are that the undue length of the assessment protocol, involving over a dozen standardized measures (only two of which are reported here), acted as a barrier to obtaining participant cooperation, despite our provision of a modest honorarium. Interviews ranged between 2 and 4 hours and many questions were asked repeatedly because they were included in more than one assessment instrument. Reducing the length of the interviews by streamlining the questions and asking for only applicable and nonredundant data could lead to a decrease in attrition rates as more families may be willing to participate in the assessment process if it did not take as long to complete.

Finally, the third goal was to provide empirically based feedback to improve program services and evaluation efforts. Information collected during this evaluation was presented to the Georgia service providers, allowing them to utilize empirical data to better understand the strengths and needs of their program. Whether this results in improvements to the program and in its effectiveness remains to be seen.

While the "face validity" and intuitive appeal of the general principles of comprehensive system-of-care wraparound services for seriously emotionally disturbed youth and their families is considerable, our sense is that the expansion of this program nationally has exceeded the degree of empirical evidence of its effectiveness. This is not to assert that wraparound services are ineffective, only that smaller scale pilot studies should have been undertaken, with positive results, prior to its widespread (and expensive) adopting. This, of course, is a familiar refrain in the program evaluation field, where the selection of services are sometimes driven more by ideology than by scientific merit. While our initial outcomes in Georgia with wraparound services are not promising, it may be that longer-term assessments may reveal more positive results, or that with additional experience in providing wraparound services our providers will become more effective.

REFERENCES

Achebach, T. M. (1991). Child Behavior Checklist for Ages 4-18 (CBCL). Burlington, VT: ASEBA Research Center for Children, Youth, and Families.

Achenbach, T. M. (1999). The Child Behavior Checklist and related instruments. In Mark E. Maruish (Ed.), The use of psychological testing for treatment planning and outcomes assessment (2nd ed., pp. 429-466). Mahwah, NJ: Erlbaum.

Achenbach, T. M., & Edelbrock, C. (1983). Manual for the Child Behavior Checklist and revised child behavior profile. United States of America: Queen City Printers Inc.

Bickman, L., Smith, C. M., Lambert, W. E., & Andrade, A. R. (2003). Evaluation of a Congressionally mandated wraparound demonstration. Journal of Child and Family Studies, 12, 135-156.

Bruns, E. J., Burchard, J. D., Suter, J. C., Leverentz-Brady, K., & Force, M. M. (2004). Assessing fidelity to a community-based treatment model for youth: The Wraparound Fidelity Index. Journal of Emotional and Behavioral Disorders, 12(2), 79-89.

Carney, M. M., & Buttell, F. (2003). Reducing juvenile recidivism: Evaluating the wraparound services model. Research on Social Work Practice, 13, 551-568.

Freedman, R. M., Katz-Leavy, J., Manderscheid, R., & Sondheimer, D. (1996). Prevalence of serious emotional disturbance in children and adolescents. In R. W. Manderscheid & M. A. Sonnenschein (Eds.), Mental health, United States, 1996 (pp. 77-91). Washington, DC: U.S. Government Printing Office.

Hansen, M., Litzelman, A., Marsh, D. T. & Milspaw, A. (2004). Approaches to serious emotional disturbance: Involving multiple systems. Professional Psychology: Research and Practice, 35, 457-465.

Heflinger, C. A., & Simpkins, C. G. (2002). The clinical status of children in state custody. In N. S. Le Proh, K. M. Wetherbee, E. R. Lamont, T. M. Achenbach, & P. J. Pecora (Eds.), Assessing youth behavior using the Child Behavior Checklist in family and children's services (pp. 53-68). Washington, DC: CWLA Press.

Herrick, C. A., Arbuckle, M. B. & Claes, J. A. (2002). Teaching interprofessional practice: A course in a system of care for children with severe emotional disturbance and their families. Journal of Family Nursing, 8, 264-281.

Hodges, K. (1994). Child and Adolescent Functional Assessment Scale (CA-FAS). Ann Arbor, MI: Functional Assessment Systems.

Hodges, K. (1999). Child and Adolescent Functional Assessment Scale (CA-FAS). In Mark E. Maruish (Ed.), The use of psychological testing for treatment planning and outcomes assessment (2nd ed., pp. 631-664). Mahwah, NJ: Erlbaum.

Hodges, K., Doucette-Gates, A., & Liao, Q. (1999). The relationship between the Child and Adolescent Functional Assessment Scale (CAFAS) and indicators of functioning. Journal of Child and Family Studies, 8, 109-122.

Hodges, K., & Kim, C. (2000). Psychometric study of the Child and Adolescent Functional Assessment Scale: Prediction of contact with the law and poor school attendance. Journal of Abnormal Child Psychology, 28, 287-297.

Hodges, K., & Wong, M. M. (1996, December). Psychometric characteristics of a multidimensional measure to assess impairment: The Child and Adolescent Functional Assessment Scale. Journal of Child and Family Studies, 5, 445-464.

Huffine, C. (2002). Child and adolescent psychiatry: Current trends in the community treatment of seriously emotionally disturbed youths. Psychiatric Services, 53, 809-811. Macro International Inc. (2002, June 3). QuickSAT Version 3.1.

SPSS, I (2001). SPSS Version 10.0), Chicago, IL: SPSS, Incorporated.

Substance Abuse and Mental Health Services Administration (1993). Final notice establishing definitions for (1) children with a serious emotional disturbance, and (2) adults with a serious mental illness. Federal Register, 58, 29422-29425.

U.S. Department of Health and Human Services. (1999). Mental health: A report of the Surgeon General. Rockville, MD: Author.

U.S. Department of Health and Human Services. (2004, April). Comprehensive community mental health services program for children and their families. Rockville, MD: Author. Retrieved on October 4, 2004, from www.mentalhealth.samhsa.gov/publications/alipubs/CA-0013/ default.asp

Walker, J. S., & Schutter, K. M. (2004). Practice and process in wraparound teamwork. Journal of Emotional and Behavioral Disorders, 12, 182-192.

Hilary L. Copp, Patrick S. Bordnick, and Amy C. Traylor, School of Social Work, University of Georgia.

Bruce A. Thyer, College of Social Work, Florida State University.

Reprint requests should be sent to Patrick S. Bordnick, Ph.D., School of Social Work, University of Georgia, Athens, GA 30602 or via the Internet using bordnick@bellsouth.net

Table 1. Mean Scores for Intake and 6-Month Follow-up on
Outcome Measures (N = 15)

 Intake 6 Month

 M SD M SD t p
CBCL (raw scores)

 Internalizing 17.0 9.8 15.9 7.7 0.91 0.38
 Externalizing 29.1 10.0 26.9 8.9 1.38 0.19
 Total Problem
 Score 75.1 30.7 72.9 24.8 0.51 0.62

CBCL (T scores)

 Internalizing 65.0 10.1 64.3 9.8 0.47 0.65
 Externalizing 71.7 7.4 70.2 6.6 1.19 0.26
 Total Problem
 Score 72.1 8.6 71.5 7.4 0.44 0.66

CAFAS (raw scores)

 School/Work 22.0 8.6 23.3 8.2 -1.00 0.33
 Home Role 23.3 9.0 22.1 8.9 0.25 0.81
 Community/
 Role 4.7 7.4 4.3 7.6 -0.29 0.78
 Behavior
 Toward
 Others 22.7 7.0 20.7 7.0 1.15 0.27
 Moods/
 Emotion 16.7 8.2 16.7 10.5 0.00 1.00
 Self-Harm 7.3 11.6 5.3 9.2 0.90 0.38
 Substance
 Abuse 0.0 0.0 0.0 0.0 n/a n/a
 Thinking 8.7 11.3 2.7 5.9 2.20 0.05
 Total Score 105.3 37.0 93.3 28.2 1.51 0.15