What works?

• Author: Roberts, Helen

Abstract

"What works?" is a fundamental question for policy makers, practitioners and service users. As well as asking what works, and how to implement it, we also need to think about what does not work, and how to stop it. If, as many of us believe (and as there is good evidence to demonstrate), the most effective time to make a difference to outcomes is in childhood, then it is likely that this is also a time when we need to be careful of doing damage. Intervening in children's lives is not just a research, policy and practice issue for those of us at the supply end. It is also a rights issue for children and young people. Those of us in the evidence-based arena who have been pushing policy makers and practitioners to adopt evidence-based practice need to be careful that we do not sell the "What works?" agenda as a simple way to solve problems. Social and educational interventions are complex and are capable of doing as much or even more harm than medical ones. Drawing on examples that relate to children and young people, this paper will suggest that the public and NGO sectors need to invest more heavily in the "D" part of R&D, and that we need to stimulate demand for research-based interventions. In the world of childbearing and HIV/AIDS, services in the United Kingdom have been transformed by powerful alliances between the evidence and the "user" lobbies. What would the world look like for young people if those having difficulties with their education, in trouble with the law, or in the care of the state, were to ask what the evidence is behind the services they are offered?

INTRODUCTION

It is important not only to know what we know, but that we know what we do not know. (Lao-Tze, Chinese philosopher) As we know, there are known knowns. There are things we know we know. We also know there are known unknowns. That is to say we know there are some things we do not know. But there are also unknown unknowns, the ones we don't know we don't know. (Donald Rumsfeld, United States Defence Secretary (2)) As this conference, the one that preceded it and the considerable work done by public and social policy, practice and academic colleagues in New Zealand make clear, "What works?" is a fundamental question for policy makers, practitioners and service users. Indeed, it is a fundamental question for all of us. "What works?" and "Does this work better than that?" or "Is it better to do something, or do nothing at all?" are questions we can relate to in making small decisions as well as big ones, both in our everyday lives and in our professional activities.

New Zealand has been a leader in this field, and has addressed some important questions. The best evidence syntheses on the Ministry of Education website (3) provide good pragmatic examples of evidence use, matching research design to the particular questions asked. SPEAR itself produces helpful practice guidelines on the policy cycle. (4) In 2002, the New Zealand Treasury asked what we know about the effectiveness of early years services (Annesley et al. 2002). New Zealand academics have asked about the effect of incomes on outcomes (Connor et al. 1999). Your country questioned the evidence for going to war in Iraq, the evidence for which has been the subject of so much discussion in the United Kingdom since (Butler 2004).

At present, practitioners, parents and children and young people looking for good research evidence on common problems will often find the evidence cupboard disappointingly bare. Where it is not bare, the outcomes measured will not always be those that seem most important to these players. It is refreshing that in New Zealand, the learning outcomes for schooling derived from best evidence syntheses include not just achievement and skills, but wellbeing and whanau spirit, as well as respect for others (rangimarie), tolerance, non-racist behaviour, caring or compassion (aroha), diligence and hospitality or generosity (manaakitanga) (Ministry of Education 1993:17).

The role of formal research evidence in the formation of social and public policy, at least in the United Kingdom, has a relatively short history. There has probably been a role for "intelligence" for as long as countries have had foreign policies, and this is likely to have been a mixture of sound research, know how, and something a bloke said in a bar (or possibly at a cocktail party if we are talking about the diplomatic corps). Much of this will have involved matters such as trade, military competition and conflict. The relative distance of the majority of the expatriate British from local populations, so well described by P.M. Forster, George Orwell and Doris Lessing suggests that the learning that could have been had from different countries' or indigenous peoples' ways of doing things was not a core part of the task of those who spent time overseas.

In the 19th century there were certainly a number of "investigations" which bear the hallmarks of good qualitative research. One example of "listening to children" can be seen in the report of Andrew Doyle to the local government board at Whitehall on the emigration of pauper children to Canada. He spoke to many children, some of whom had had happy experiences, some less so. Of course if the United Kingdom's policy of sending orphan children to New Zealand, Canada, Australia and Rhodesia (as it then was) had been subjected to a traditional evaluation, the report may well have said that the children looked well cared for, the food, transport and child-to-adult ratios were adequate, and the youngsters, asked about their experiences might well have said: "It's OK here" (or whatever the 19th century equivalent was). Doyle managed to collect the other side of the story, often untold even in current policy evaluations. Of the sea voyage, he was told: "we sicked all over each other," and a child patiently explained the meaning of adoption: "'Doption sir, is when folks get a girl to work without wages" (Doyle 1875).

There were other investigations which we would certainly view as research evidence now. Charles Booth's work in the United Kingdom in the late 19th century charted inequalities in health and welfare. Between 1886 and 1903 Charles Booth (and Mary Booth, about whom we hear rather less) worked on the life and labour of the people of London--17 volumes in all--working with, among others, Beatrice Webb. He lodged with working-class families, and observed both the inequalities and the resilience and coping strategies of families living with disadvantage that we continue to learn from today. Booth wrote:

The children in class E ["... above the line of poverty"], and still more in class D ["... the poor"], have when young less chance of surviving than those of the rich, but I certainly think their lives are happier, ... always provided they have decent parents. It is the constant occupation, which makes the children's lives so happy. They have their regular school hours. They have for playground the back yard, [or] the even greater delights of the street. Let it not be supposed, however, that on this I propose to base any argument against the desire of this class to better its position. Very far from it. [...] the uncertainty of their lot, whether or not felt as an anxiety, is ever present as a danger. (Booth 1902: 159-60) Booth was not only a keen social observer and essayist. He produced maps descriptive of London poverty which would be the pride of a 21st century social geographer, with investigators collecting data while accompanying policemen on their rounds. The Fabian Society, of which Booth was a member, was founded in 1884, and committed to gradual social reform. Also active in the Fabians were Beatrice and Sidney Webb, who saw research very specifically as informing social policy in a linear way which we might now consider unrealistic. The London School of Economics, now part of the University of London was founded by the Webbs in 1895 to inform social and public policy. Its motto, "rerum cognoscere causas"--to seek the cause of things, remains one key question at the heart of evidence-informed policy today. Another is what to do when we do know the cause of things, and a further one is to learn the consequences of things.

While there were huge scientific advances in all fields in the 20th century, some benign or better, some less so, the project of uniting research policy and practice became less central as a key intellectual interest. Indeed, even today there are those who recoil at the implied positivism of research-informed policy and practice. Although the social sciences and agriculture had been in the forefront of the development and use of research evidence, and in particular experimental methods, it was the establishment of the worldwide Cochrane Collaboration in health care in 1993 that gave fresh impetus to the use of sound research evidence to inform practice. In the late 1990s, and to some extent prompted by both social scientists and clinicians involved in Cochrane, the Economic and Social Research Council (ESRC) in the United Kingdom made available funding for a centre, and a number of "nodes" to work on research evidence. One of these ESRC-funded "nodes" is the Research Unit in Research Utilisation, whose co-director, Sandra Nutley, spoke at the 2003 Social Research and Evaluation Conference (Nutley 2003). Another is What Works for Children? which is a collaborative project between City University, the University of York and Barnardo's.

In this paper, I am drawing on work we have done as part of the What Works for Children enterprise and on joint work with Mark Petticrew from the University of Glasgow on ways of answering different kinds of research question, and of carrying out systematic reviews (Petticrew and Roberts 2003, Petticrew and Roberts 2005). In order to illustrate some of the problems (and steps towards solutions) in the world of evidence-based policy and practice, I am going to:

*...