Methodological lessons learned from the process evaluation of the comprehensive strategy for serious, violent, and chronic juvenile offenders.

• Author: Sridharan, Sanjeev

Abstract

Five important methodological lessons learned from the national process evaluation of the Comprehensive Strategy for Serious, Violent, and Chronic Juvenile Offenders, conducted in the United States, are discussed. The Comprehensive Strategy is an example of a comprehensive community initiative (CCI), which is an initiative targeted at an entire community and intended to provide holistic, multifaceted responses to complex social problems. The five lessons learned for future evaluations of CCIs are: (1) the need for explicating theories of change; (2) a process of clarifying stakeholder roles; (3) the importance of mechanisms for engagement of evaluator with sites; (4) the need to develop a stakeholder-driven anticipated timeline of change within communities; and (5) the value of empirically testing pathways of change in CCIs.

INTRODUCTION

In recent years, comprehensive community initiatives (CCIs) have increasingly been used as a means of addressing social problems in a number of sectors (Hebert and Anderson 1998, Kagan 1998, Kubisch et al. 1995, Milligan et al. 1998). In a recent issue of the Social Policy Journal of New Zealand, the relevance and promise of CCIs were explored in detail by Davies et al. (2002). As the popularity of CCIs increases, an exploration of methodological issues that are relevant for evaluations of CCIs is necessary. This paper will employ a large CCI implemented in the United States as a field example for methodological lessons learned.

CCIs target entire communities and are intended to provide holistic, multifaceted responses to complex social problems. CCIs have emerged in response to several trends, including:

* a recognition of the fragmentation of social services and the inappropriateness of responding to complex problems in a categorical manner

* funding requirements that encourage cost-effective prevention, reduction of expensive intervention measures and elimination of duplication in services

* a growing pragmatic and ideological support for the necessity of public-private partnerships and local action to solve social problems (Annie E. Casey Foundation 1995, Kubisch et al. 1997).

In the United States, CCIs have been implemented to address a wide variety of social problems. The Department of Justice has funded a number of CCIs to tackle a range of criminological problems. Examples of such initiatives include the Comprehensive Strategy for Serious, Violent, and Chronic Juvenile Offenders (henceforth Comprehensive Strategy); Safe Futures; Safe Kids/Safe Streets; Safe Start; and the Greenbook Initiative. The Comprehensive Strategy, for example, targeted reduction in juvenile delinquency, Safe Start focused on minimising the negative impacts of children's exposure to violence in communities, and the Greenbook Initiative sought to build linkages between child-protective service organisations, domestic-violence agencies and the judicial organisations within a community.

There has been some discussion in the CCI literature on the types of evaluation methods that are useful for evaluating CCIs (Annie E. Casey Foundation 1997, Kubisch et al. 1995). These discussions have focused on issues of methodological design, as well as on the more challenging issues of the appropriate role of CCI evaluators (Brown 1995, 1998). In this paper we reflect on some of the methodological lessons learned from working on the national process evaluation of the Comprehensive Strategy. This discussion should be relevant to funders, programme staff and evaluators in New Zealand, who are considering implementing and evaluating CCIs.

The key conceptual problem that cuts across all of the lessons discussed here is on how the evaluation can positively influence the initiative. The lessons learned encompass a wide variety of issues including the relevance of a theory of evaluation influence (Henry and Mark 2003, Mark and Henry 2004), the importance of clarifying the roles of the various stakeholders involved in CCIs, and a discussion of "specialised" methods that may be more appropriate in evaluating CCIs. Examples of such methods include network analysis (Scott 2000) and concept mapping (Trochim 1999).

In this paper we first discuss some of the challenges involved in evaluating CCIs. We then briefly describe the background of the Comprehensive Strategy and the goals of the national process evaluation. Both the discussion on challenges of evaluating CCIs and the background of the Comprehensive Strategy should help the reader better understand the nature of CCIs. Finally, the methodological lessons learned in evaluating CCIs are discussed.

CHALLENGES FOR EVALUATING COMPREHENSIVE COMMUNITY INITIATIVES

It is important to recognise that CCIs differ from conventional social programmes. An examination of the challenges involved in evaluating CCIs will help highlight the differences between CCIs and conventional programmes (Kubisch et al. 1995). Some of the challenges involved in evaluating CCIs include:

Operationally defining the initiatives

CCIs tend to be flexible (community-specific) and dynamic (evolving in response to lessons learned, as well as local needs and constraints). Given the potential variation in the scale, nature and sequence of the interventions, developing an evaluation framework that can be used to isolate programme effects is difficult.

Lack of appropriate control groups

While control groups can be created for individual programmes, control groups for comprehensive initiatives as a whole are very difficult to identify (Hollister and Hill 1995). The absence of appropriate control groups, coupled with the changing nature and variation in interventions across sites, makes causal attribution difficult. As an alternative, several researchers have proposed a focus on theory of change approaches (Connell and Kubisch 1998, Weiss 1995). A theory of change approach to CCI evaluation may be defined as "a systematic and cumulative study of the links between activities, outcomes and contexts of the initiative" (Connell and Kubisch 1998:16).

Horizontal and vertical complexity

CCIs are designed to promote horizontal collaboration between agencies and to affect outcomes at multiple (vertical) levels, including the individual, family, community and service systems. The evaluation challenge is to measure the synergistic benefits of collaboration between agencies, as well as to examine the relationship between and within each of the outcome levels. Measuring such collaborations and their respective benefits across multiple levels, and incorporating these findings into an operationalisable framework, is a complicated task.

Multiple outcomes and pathways of change

Since CCIs often represent diverse organisations with differing programme theories, there are multiple pathways by which interventions and processes can influence outcomes. Understanding and delineating these pathways and how they affect short-term, intermediate and long-term outcomes pose significant hurdles to evaluation. This is particularly important when there is diversity in the community contexts. The process of linking these varied contexts to individual outcomes in a measurable fashion is very complicated.

Different sites can also have different definitions of community

Within the same initiative, the community in question can range from small neighbourhood blocks to large communities. As an example, in the Comprehensive Strategy, communities included both a small zip code within Houston, Texas, as well as the entire county of San Diego, California (a county about the size of some states).

We do not attempt to "solve" the above methodological challenges in this paper. Instead, our approach through the thicket of methodological challenges is to focus on processes that will enhance the influence of the evaluation on stakeholders involved in the CCI. Recent work by evaluation theorists has stressed the importance of focusing on such a theory of influence (Kirkhart 2000, Henry and Mark 2003, Mark and Henry 2004) of evaluations. It is useful to think of the evaluation as contributing to a strategic conversation (Duignan 2002) between stakeholders. In our view, such a strategic conversation can be enhanced with a focus on the influence processes by which the evaluation can improve a CCI.

THE COMPREHENSIVE STRATEGY

The Comprehensive Strategy provided a research-based framework for combating juvenile crime by targeting prevention efforts at youth who are at risk of delinquent behaviour, intervening in early delinquent behaviour, and responding effectively to youth who become involved in serious, violent and chronic offences. The Comprehensive Strategy's principal components of risk-focused delinquency prevention and graduated sanctions are intended to provide a "continuum of care" that both prevents and interrupts the progression of delinquent and criminal careers. The Comprehensive Strategy achieves this goal through a systematic community-level and research-based planning approach to crime and delinquency reduction.

The Comprehensive Strategy planning process involved several interrelated and ongoing steps:

* comprehensive training that mobilises the community, key leaders, and other stakeholders and familiarises them with the goals, principles and elements of the Comprehensive Strategy

* completion of community assessments (of risk factors, existing systems and resources, etc.)

* development of strategic plans (usually referred to as the five-year plan) that identify appropriate services, programmes and approaches to address the community's identified needs

* specifying...