A critique of measures that assess community efforts to reduce disproportionality and disparities in child welfare and juvenile justice agencies.

Barbee, Anita P.

I was asked to review an article written for this special issue by Brad Richardson and Dennette Derezotes about efforts in the state of Iowa to reduce disproportionality and disparities in the child welfare and juvenile justice systems. The article focused on the use of three measures to facilitate reduction of the proportion of Native Americans in these two systems and to communicate the effects of interventions on the problem: the Disproportionality Diagnositc Tool (DDT), the Race Equity Scorecard, and the Ecomap of community organizations and their impact on Native American families. While the article revealed some improvement in the way the juvenile justice agency handled minority youth and short term effects of disproportionality reduction efforts in child welfare, the article was not maximally informative for other states, regions, counties and tribes who wish to replicate their efforts. In this critique, I will point out some of the key shortcomings of the method used and of each measure and offer some additional ideas to improve methodological and measurement issues in this area of addressing disproportionality/disparities in large systems.

DISPROPROTIONALITY DIAGNOSTIC TOOL

The first measure discussed was the DDT developed and tested by several scholars and practitioners for the American Public Human Services Association (APHSA, 2008, Fabella, Slappey, Richardson, Light & Christie, 2007). The DDT is an assessment tool of the community, organizational and workforce conditions that could undermine fair treatment and effective work with families and children leading to deleterious outcomes. In the Richardson and Derezotes (this issue) article the DDT was administered three times to assess the efforts being conducted to reduce disproporitionality and disparities. The implication was that the scores on the DDT would improve over time. However, while there was a general boost a few months after the baseline assessment, two years later many results were actually worse than at baseline.

The DDT instrument contains 94 items with a response format of yes, no, sometimes and don't know (in the Richardson & Derezotes article sometimes was not included in the results). The tool includes pertinent questions about policies and practices that could affect disproportionality and disparity rates. One problem with the tool is that all 94 primary questions ask about whether or not a strategy is present. Sometimes primary questions are followed by more probing questions that begin to address the quality of the strategy or practice or perceptions about the item, but quality of approaches and perceptions are not rated every time. Thus, the tool could be improved by ensuring that appropriate questions utilize a dichotomous response format and/or a scale to assess the extent to which an area is addressed or the quality of the execution of a policy or practice. For example, the question, "Does your agency's strategic plan address issues of diversity in the values, mission and goals?" is verifiable, but perceptions of the quality of the plan or the items in the plan that address diversity could differ.

This leads to a related point in that 59 (63%) of the primary questions are verifiable facts. However, not everyone in the community or agency will actually know the answer to the question. So if multiple individuals in the community or agency complete the tool, the percentage of individuals that answer a verifiable question in the affirmative may not be 100% because sometimes people don't know what they don't know or because different parts of a community or organization may operate slightly differently. So for a careful analysis of a state, region, county, city or tribe to take place, I would recommend that questions related to facts that are verifiable be grouped together and answered by leaders who can also supply documentation for the presence of a particular strategy. If the jurisdiction is large like a state or large region, county or city, then these same 59 questions could be answered by leaders in the central office and in each satellite office to examine consistency in the articulation and execution of particular policies and practices across the jurisdiction. In either case, if these types of questions are answered and documented, then the group monitoring progress and deciding on interventions can at the very least know what is and is not in place at the baseline assessment.

However, these same questions should also be included in a survey of numerous stakeholders in the form of ratings of the extent to which the policy or practice is followed or about other relevant information, so as to gain a greater understanding of where weaknesses may be in the community, organization or among individuals. For example, the question "Does the agency have staff that represent the community being served?" could be rewritten as "To what extent does the agency staff represent the community being served?" 1- not at all to 5- very much or "What percentage of staff represent the community being served?" or "How much does the racial and ethnic make-up of the agency staff affect outcomes for minority children?" The rest of the questions are not as clear cut in terms of verifiability. These types of questions are not suited for a dichotomous response format. It would be better to use a rating scale response format instead. There was some attempt to utilize a rating scale by including the category of "sometimes," but the scale is not properly used in this case. The items should either be rated as yes or no (that a strategy or practice is or is not present) or rated from "all of the time" to "never" with "sometimes" in the middle position of the rating scale. In addition, different items really need different or multiple rating scales. For example, the question "Does our staff demonstrate that they have internalized the values, mission, and goals related to diversity and disproportionality?" is really difficult to know without extensive observation of each employee as they interact with clients, and case reviews of their documentation of work with clients. Most states do not have such data to verify the question. Thus, it could be transformed to read "To what extent do staff demonstrate that they have internalized ..." or the items could be part of a staff evaluation tool that supervisors complete for each of the workers on their teams in order to generate a mean score for employee demonstration of internalization of values, mission and goals related to diversity.

Furthermore, since actual attitudes, emotions, cognitions, or practices are not included in the instrument, evaluators and administrators may want to utilize existing tools or develop new tools to assess more micro processes that are likely to directly impact disproportionality and disparity numbers. Thus, the types of items that were missing or not completely operationalized in the DDT that should be measured as both part of the assessment of the workforce, but also as part of the mediators of change in the system include: 1) The level of prejudice and/or racist, classist, sexist and other negative attitudes held by every employee, but particularly every front line worker and supervisor; 2) The amount of stereotyping staff engage in when thinking about and working with clients; 3) The types of attributions staff make about why families on their caseloads are in the predicament that they are in and the feelings staff have about their clients and their behaviors; 4) A step by step analysis of how individual employees make case decisions and an account of specific remedies available for correcting errors through training, coaching, supervision or tool refinement; 5) The tools that staff have available to them to correct errors in thinking or to enhance good decision making processes (i.e. whether or not the agency has adopted Structured Decision Making Assessment Tools, the quality and understandability of risk assessment and safety assessment tools utilized by the agency, whether or not the agency conducts blind case reviews to assess the level of discrimination that is actually occurring in the agency or to spot incidences of discrimination), (1) and 6) The actual behaviors staff engage in when interacting with clients including adherence to best practices with regards to engagement, assessment, case planning, intervention, documenting change and celebrating success (for example- Solution Based Casework, Christensen, Todahl, & Barrett, 1999). Thus, other jurisdictions who decide to use the DDT should measure workplace behavior (through observations), decision making (through interviews and case reviews), and statistics (of minority entry into the systems and treatment in the system).

And, a final note, if a longitudinal design was utilized that randomly chose a sample of participants from the community and agency to complete the tool across the three time periods, then the results either mean that the intervention failed or that a mix of experts and non-experts were included that clouded the findings. If a cross sectional design was utilized and the DDT was sent to the entire population where a different set of sub-samples of participants voluntarily completed the DDT at each time period, then the results can be attributed to a flaw in the design. If different people completed the instrument each time, then one does not know if the answers were due to actual changes in the community, agency or individual behaviors or due to the perceptions or knowledge of those completing the DDT. Variability across time could be confounded with the sample. A third explanation for the wild variability in answers could be attributed to the sample size, which was also not revealed in the article. If the sample size was small, then the answers of a few respondents could skew the results and create the perception that circumstances were better or worse than they actually were.

RACE EQUITY SCORECARD

The second tool that was utilized in Iowa was the Racial Equity Scorecard. This measure focused on four measures of disproportionality and disparity. Instead of focusing on the decisions made in substantiating abuse and neglect, removing children from the home, placement of children into various settings and movement of children back to the home or to an adoptive home, current measures like the Racial Equity Scorecard only focus on numbers of children in the system at these various points in time compared to their proportion in the geographic area. This, of course, was how the problem was first detected (e.g., Hill, 2006). The use of these measures to determine the extent of disproportionality and disparities in a jurisdiction is based on the assumption that there is no difference in the incidence rate of child maltreatment between ethnic and racial groups. At first glance, this was a plausible approach because there were three national incidence studies conducted in 1979 and 1980 (NIS-1), 1986 (NIS-2) and 1993 (NIS-3) which all found no differences in child maltreatment by race (Hill, 2006, 2007). The strength of the methodology of the national incidence studies is that the estimation of the number of maltreated children in the US is based on a sample of both children who have come to the formal attention of the child welfare agencies in each jurisdiction and children who are informally known by professionals who work with children who have not yet encountered the child welfare agency. The only weakness in this methodology is that there is no rigorous household survey included in the study that asks caregivers and other adults living in the home questions about the specific behaviors they do or do not engage in with regards to the children in the home. Thus, children who suffer from child maltreatment who are not detected are not included in the incidence studies. It is likely that many of the children that go undetected are from middle and upper income families who may have the social skills to hide marks, neglectful behaviors and certainly emotionally abusive behaviors from professionals and neighbors.

So, while discrimination towards the poor, which has an overrepresentation of minorities, is a huge problem, jurisdictions and researchers need to be careful about how they measure success. Since the racism and classism that created oppression of the poor and minorities is also pervasive in the institutions that seek to monitor problems of crime and child maltreatment, then another layer of intervention needs to be within the operations of the agencies themselves i.e., simply counting the numbers of minority children who enter or move through these systems is not adequate to truly understand how much and in what ways these systems are contributing to the problem.

The measurement of disproportionality and disparities in juvenile justice and child welfare systems, thus, needs to be more rigorous. Blind reviews of cases (removing any reference to race or social class of the family members) need to be conducted to determine if and when decisions are being made appropriately and if families are being treated fairly. Thus, child welfare agencies and juvenile justice staff need to conduct these types of reviews to determine when minority and/or poor children are being treated unfairly in their systems and determine the root cause of such discrimination so that specific interventions can be devised to change the thinking and behavior of decision makers in the systems. These case reviews need to be conducted regularly to show if discrimination persists or declines as a result of interventions.

Instituting strategies to prevent child maltreatment in the community, differentially responding to families who come to the attention of CPS, utilizing family team meetings, investing in family preservation efforts, and being committed to kinship care and finding that those interventions among others actually reduce CPS referrals, substantiations of maltreatment and removal from the home for minority clients is a sign of success. In addition, more razor sharp determinations of biases of staff and discrimination by staff need to be conducted in order to discern exactly where, when and how minority and/or poor children are being treated unfairly. This more precise measurement will both document the unfair treatment of minority and/or poor families and pinpoint the exact nature of the problem so that other innovative solutions can be implemented.

ECOMAP MEASURE

The third measure that Iowa utilized was an adaptation of an ecomap. All of the organizations that could possibly affect Native American well being were depicted in the ecomap as either influencing disproportionality or disparities positively or negatively. This type of measure raised awareness of the role of many systems, not just child welfare and juvenile justice in determining disproportionate rates or disparate outcomes for minority families. While many of the organizations were seen positively in the ecomap developed by Richardson and Derezotes, in most communities, certain systems are contributing a great deal to this problem nationally and routing many minority and poor children into the child welfare and juvenile justice systems.

Specifically, what the ecomap does is highlight the need for all institutions that work with children and families such as the health care system, the mental health system, the school system, the child welfare system and the justice system to assess the level of fairness in their treatment of minority and poor families. Perhaps one outgrowth of the ecomap would be for communities to create scorecards for all of their institutions to ensure that minorities are given the same opportunities as those from the dominant population and are not mistreated in these systems.

The efforts in Iowa seemed to be extensive and there was improvement in collaboration and outcomes in the county where Native Americans were having the most problems. The teams there did the most with the knowledge that had been generated to date, and the members of the Iowa team helped to create and highlight the utility of several measures used in this arena. Iowa is to be commended for their progressive stance on this issue. My hope is that Iowa and other states and jurisdictions will build on knowledge generated in this special issue and expand the work to be even more incisive and effective so that all children, but particularly those that are the victims of additional prejudice and discrimination by systems that were created to help or rehabilitate them, will thrive.

Dr. Barbee would like to thank Dr. Riaan van Zyl for helpful comments on a draft of this paper.

REFERENCES

Alvey, K. (1992). Black parenting: Strategies for training. New York: Irvington Publications.

APHSA (2008). Disproportionality Tool Description. As found in www.ocfs.state.ny.us/main/recc/.../Disproportionalit yToolInstructions.pdf

Barbee, A. P., Henry, K., & Johnson, L. (2010). A Report on the Effectiveness of Undoing Racism and the Overall Change Effort to Reduce Disproportionality and Disparities in Child Welfare in Louisville, Kentucky. Submitted to the Race, Community and Child Welfare (RCCW) Advisory Board.

Bartholet, E. (2009). The racial disproportionality movement in child welfare: False facts and dangerous directions, 51 Arizona Law Review 871.

Christensen, D.N., Todahl, J., & Barrett, W.G. (1999). Solution-based casework: An introduction to clinical and case management skills in casework practice. New York: Aldine DeGruyter.

Hill, R. B. (2006). Synthesis of research on disproportionality in child welfare: An update. Seattle, WA: Casey Family Programs.

Hill, R.B. (2007). An Analysis of Racial/Ethnic Disproportionality and Disparity at the National, State and County Levels. Washington, DC: Casey Center for the Study of Social Policy.

Johnson, L.M., Antle, B.F. & Barbee, A.P. (2009). Addressing disproportionality and disparity in child welfare: Evaluation of an anti-racism training for community services providers. Children and Youth Services Review, 31, 688-696.

Moore, K. A., Redd, Z., Burkhauser, M. Mbwana, K., & Collins, A. (2009). Children in poverty: Trends, consequences and policy options. Child Trends, 11, 1-12.

Piketty, T. & Saez, E. (2007). Income inequality in the United States, 1913-2006 in A. B. Atkinson & T. Piketty, T. (Eds.). Top Incomes Over the 20th Century. Oxford: Oxford University Press.

SAMHSA (2001). Mental health: Culture, race, ethnicity. Supplement to Mental Health: Report of the Surgeon General.

Sedlak, A.J., Mettenburg, J., Winglee, M., Ciarico, J., and Basena, M. (2010). Fourth National Incidence Study of Child Abuse and Neglect (NIS-4): Analysis Report. (Prepared by Westat under contract to the Administration for Children and Families.) Washington, DC: Administration for Children and Families.

ANITA P. BARBEE

University of Louisville

(1) The DDT does include the use of FGDM and diligent recruitment of foster families, but not other strategies.