The evidence game: home win or score draw?

Bullock, Roger

At a recent seminar on child development, one of the world's leading experts came out with the comment, 'evidence usually wins in the end'. Further elaborating, he argued that 'even if it does not provide categorical answers, it sets a framework, indicates probabilities, highlights important questions and closes lines of enquiry'. This confident announcement set me thinking: 'Is this true with regard to children in need?' The speaker's background was in medicine where the supply of and demand for reliable information are probably greater than in social care and, in some circumstances, the number of potentially significant factors is easier to handle.

Having spent 46 years in research, I have seen the respect given to evidence fluctuate. When, as at present, resources are diminishing, it becomes much sought after by agencies and project managers seeking to prove their services' effectiveness, demonstrate value for money and fend off cuts. In this competitive climate, all sorts of claims are made about children and it is hard to know whether they are genuine wishes for the wider good or appeals to strengthen the negotiating position of those speaking out.

Two recently reported propositions about looked after children illustrate this. One says that a high proportion of young men and women entering prison custody have been in care, and the other wants more children to be removed from home. From a research perspective, it is difficult to know what to make of these statements, especially as it is the bullet points that tend to attract publicity. With regard to prison custody, the statistic that over one-third of entries are ex-care is undoubtedly accurate but gives a misleading picture when considered in isolation. Each year in England some 150 out of 25,000 care leavers (54% of them boys) go directly to jail and so form part of the 2,000 (92% of them male) 15-to 17-year-old annual custody admissions. In addition, follow-up information shows that only about three per cent of looked after 16-year-olds are in custody three years later. So, how can these apparent discrepancies be reconciled? The solution is one familiar in drawing conclusions from sequential processes: when a small receptacle is fed by a much larger reservoir, inferences about the relationship of one to the other need handling with caution.

Similarly with regard to the proposal that more children should be taken into care, it is difficult to comment without knowing the epidemiological basis and the assessments of types of care and their ability to meet particular needs. It may be right but so too might be the critics who claim with equal force that some children in care should not be there, or that the 'in' or 'out' question is irrelevant as it is the system itself that needs reforming. In the absence of sound evidence, the debate feels more like watching ping pong than listening to discourse.

Social scientists have long asked why certain public concerns lead to political action while others do not. In child care, researchers posed this question with regard to child protection and concluded that policy reflected the threshold drawn between 'abuse' and 'not abuse'. They went on to argue that this was influenced by four factors--legal/moral questions, users' views, pragmatic considerations and evidence--and that the balance between these factors changes. (1) So, for the reason explained, evidence currently is riding high in the policy equation.

This conclusion raises the key question of what is evidence with regard to looked after children? Evaluating outcomes is an academic minefield fraught with searching questions, such as whose outcome, when it should be assessed and how we decide whether it is good or bad. More difficult still is proving that effects, however well measured, are the result of what has been done and not something else. There are also many who claim that a task as broad as parenting cannot be assessed on simple criteria as it is as much an art as a science; evidential evaluation is only really suitable for uncomplicated preventive or therapeutic interventions. Moreover, others, such as psychotherapists, who study the unconscious as well as the conscious, will defend their work from any reductionist thinking and phenomenologists will stress the importance of the meaning and emotions we attribute to situations --after all, this is what makes us human even if it messes up cause-and-effect models.

But research is not a monolith and different types of study carry different sorts of messages with different time dimensions. There are, for example, general theories of child development that offer a conceptual framework. These cannot be tested in themselves although hypotheses derived from them can. They often take a long time to seep into thinking and the most fashionable of these at the moment in fostering and adoption is Bowlby's theory of attachment. As this work is 50 years old, Bowlby must surely be awarded a prize for 'winning in the end'. Then there are the large-scale empirical studies for which the US and UK are renowned. These link children's experiences and identify the risk and protective factors associated with particular outcomes. Finally, there are the small-scale studies of the sort that abound in this journal, which some critics might say are scientifically weak but which can add a smidgeon of support and refinement to existing knowledge.

It is at this point that technical questions begin to bite because evaluations of different kinds tend to produce different results. There is a hierarchy of methodologies to assess impact that is based on quality, ranging from descriptive studies at the bottom, via those using a control group, through quasi-experimental designs, to randomised-controlled trials at the top. Unfortunately, the likelihood of finding effects diminishes as the scale is ascended; so, as most studies of looked after children are in the lower echelons, the knowledge that practitioners rely on might be over overconfident.

If evidence-based social work is inescapable, how should it be handled? The recent report on child protection by Eileen Munro (2) is helpful since it discusses the value and challenges associated with evidence-based interventions in children's services. She defines these as work with regard to prevention, early intervention or treatment that is proven by a high standard of evidence to improve children's health or development.

But what qualifies as an evidence-based programme? Criteria generally accepted in other disciplines are that the intervention is supported by at least two robust evaluations in which the effects on those receiving the intervention were compared with a control group. One of these should be a randomised control-led trial.

Given this criterion, what types of evidence-based programme exist in the child development field? They fall into at least five categories: community/ public health strategies; health visiting activities; early years support; parenting initiatives; and therapeutic interventions. Details of specific examples can be found in the Allen (3) (pp 145-7) and Munro Reviews (paras 6.31-39), which offer important points of reference when deciding how best to allocate scarce resources in the areas of prevention/early intervention and child protection.

It is, of course, important to use the same high-quality evaluations to find out what does not work, as some popular services are likely to be exposed as ineffective or even to have serious unintended consequences.

But it is here that the other three factors affecting the policy threshold come into play. Evidence does not produce a prescription for what must be encouraged or axed but suggests a direction of travel and encourages further testing and exploration. While every decision has a downside, the dilemma for professionals is whether the strengths of the upside are sufficient to go ahead. The dearth of rigorous evaluations in UK children's services means that it is not known whether many popular services are effective but that legal/ moral and pragmatic considerations that something 'needs to be done' demand their continuation.

The problem with evidence-based programmes

This brief discussion of evidence-based programmes runs the danger of giving the impression that they have the potential to eradicate the problems in hand. Sadly, they do not and some qualification is necessary.

One is that an exciting result can have contrasting implications for individuals as opposed to larger groups. For example, a two per cent reduction in the rate of conduct disorders among the looked after population might seem small but would benefit over a thousand children. Another is that exceptions to expected outcomes are not necessarily irrelevant; those cases that do fit the evidence can stimulate important research studies. A further issue is that many proven interventions originate from outside the UK. This does not invalidate them but it does urge retesting to ensure that the ideas travel well.

There are also inevitable difficulties in getting evidence-based programmes accepted and implemented on a wide scale and ensuring that once in place, their fidelity is maintained--that is, as they were intended, with the correct levels of training, coaching and adherence to the manuals. Failure to do this can reduce their intended benefits and can even be damaging. At the moment mainstream services in the UK have limited experience of providing such services as most of them depend on short-term, marginal funding. Getting systems ready for evidence-based programmes and evidence-based programmes ready for systems is, therefore, fundamental to any progress in this area.

Some of these suggestions will undoubtedly be anathema to many professionals and carers, but the current recession means that the evidence debate cannot be ignored. It is highly functional in cash-strapped times as it is ethical in that it identifies the most effective response, justifies public spending in terms of value for money and protects agencies from litigation at a period when many lawyers and pressure groups are also struggling to maintain their employment.

Finally, one has to ask sceptics and critics: What is the alternative? That we allow people to do what they like to families and children or remain free to dictate what others need? Good results might emerge from such an approach, but they are most likely to be due to serendipity rather than systematic calculation. Child care history is littered with good intentions that now appear, at best crackers, or, at worst, wicked. While some seeking to assess child welfare might regret replacing the ability to recite the catechism with the Strengths and Difficulties Questionnaire, the fact is that the latter is more likely to produce the ethical, least harmful and more effective practice that, in my view at least, should be among the hallmarks of a modern society.

So to return to the seminar; I am not sure if 'evidence is winning in the end' but in the present climate it is certainly a candidate for a good score draw.

(1) Department of Health, Child Protection: Messages from research, London, HMSO, 1995, p 16

(2) The Munro Review of Child Protection, Final Report, Department of Education, Cmd 8062, May 2011

(3) Early Intervention: The next step, An independent report to Her Majesty's Government by Graham Allen MP, HM Government, January 2011

Roger Bullock is Commissioning Editor of Adoption & Fostering and a Fellow, Centre for Social Policy, Warren House Group at Dartington