Challenges to implementing good practice guidelines for evaluation with Maori: a Pakeha perspective.

• Author: Roorda, Mathea
• Position: Report

Abstract

Evaluation contractors working in the Aotearoa / New Zealand government sector, whether Maori or non-Maori, are expected to use culturally appropriate processes when evaluating mainstream programmes where Maori are a significant subgroup. For independent evaluators, this expectation is generally made explicit in requests for proposals. A range of formal guidelines has been developed over the last decade to support both commissioning agencies and independent evaluators to conduct culturally appropriate evaluations. However, few of the processes suggested in these guidelines are fully incorporated into Pakeha evaluation practice. A case study of an anonymised "Programme X" identifies and reviews some of the challenges to good practice and process.

INTRODUCTION

Over the past 10 years evaluators in New Zealand have had access to different sets of guidelines that focus specifically on research involving Maori (Health Research Council 1998, Te Puni Kokiri 1999, Ministry of Social Development 2004). These guidelines have led to greater awareness of a range of considerations entailed in research and evaluation involving Maori. The launch of the Social Policy Evaluation and Research Committee (SPEAR) Good Practice Guidelines (2) (Guidelines) is a renewed prompt for evaluators and social policy agency officials to reflect further on the processes they engage in when evaluating government-sponsored programmes where Maori are participants. The Guidelines also have relevance to independent evaluators and researchers who work for government sector agencies (SPEAR 2008).

In a New Zealand social policy context there is an expectation--generally made explicit in the selection criteria of requests for proposals (RFPs) for research and evaluation work--that culturally appropriate processes will be adopted wherever Maori are a significant group or subgroup in the programme under review. Although an evaluator or evaluation team may start with strong intentions of observing the advice and direction of such guidelines, they often fall short of what is outlined as good practice. The focus of this paper is on some of the complex issues that attend the implementation of the Guidelines in real-world research and evaluation contexts. We show that, for independent Pakeha evaluators, the "intention of observance" and the apparent "failure of practice" pose ongoing challenges.

Identifying and understanding some of the factors that might contribute to the gap between intention and practice is one of the purposes of this paper. A second purpose is to stimulate dialogue among evaluators and commissioning agency staff about current practice and how "the standard of research and evaluation practice across the social sector as a whole" (SPEAR 2008:5) could be enhanced. Insights drawn from a review of the process of a government-commissioned evaluation, and experience in independent evaluation practice, inform the discussion in this paper.

We begin with a short contextual discussion of guideline development in New Zealand, and then move to the analysis of "Programme X". In this case study all identifying material has been removed so that the evaluators, commissioning agency and participants have their anonymity preserved. Notwithstanding the constraints and frustrations that were identified in this evaluation, all those involved worked hard, within complex constraints, to deliver a responsive, robust evaluation.

WHAT ARE GUIDELINES FOR RESEARCH AND EVALUATION WITH MAORI?

Guidelines, as the term implies, are not sets of rules or laws, but rather provide principled advice and direction. In the case of research and evaluation, such guidelines may steer the conduct of a specific project in the direction of ideal practice. Professional evaluators who belong to associations or societies such as the Australasian Evaluation Society (AES), or the recently formed Aotearoa / New Zealand Evaluation Association (ANZEA), will also work within ethical guidelines promulgated by the organisation. AES, for example, promotes the ethical practice of evaluation and aims to "foster continuing improvement in the theory, practice and use of evaluation" (Australasian Evaluation Society 2002:2). The ANZEA constitution identifies its purpose as being to:

Promote excellence in evaluation in Aotearoa New Zealand with a focus on the maintenance of appropriate ethical standards for the evaluation profession, development of effective practice and craft, and the promotion of reflective learning as a strategy for evaluation for the public and community well being. (ANZEA 2007)

Guidelines for research and evaluation involving Maori are intended to improve the quality of evaluation practice within Aotearoa / New Zealand government agencies and to "enhance our ability to carry out effective and appropriate research (and evaluation) with Maori" (Ministry of Social Development 2004:2).

Tauri (2004) outlines the chronological development of guidelines for research and evaluation involving Maori and notes that a number of initiatives implemented in the 1980s and 1990s by government agencies acted as a signal of increased recognition and commitment to the Treaty of Waitangi. The first to be developed were Guidelines .for Researchers on Health Research Involving Maori (Health Research Council 1998). These were aimed at researchers in the health sector and focus on processes for consultation. They were followed a year later by Te Puni Kokiri's Evaluation for Maori: Guidelines for Government Agencies (Te Puni Kokiri 1999). In 2004 the Ministry of Social Development's (MSD) Centre for Social Research and Evaluation (CSRE) published Nga Ara Tohutohu Rangahau Maori: Guidelines .for Research and Evaluation with Maori, aimed at both staff and contractors. These two most recent guidelines include suggestions for processes to consider during all stages of an evaluation, from planning through to reporting.

In 2002 SPEAR began developing a set of guidelines relevant to all stakeholders in social policy research and evaluation in Aotearoa / New Zealand. SPEaR's mandate is to "oversee the government's social policy research and evaluation purchase" (SPEAR, n.d. Purpose, para. 1). One of its functions is to "promote the utilisation of 'best practice' approaches, tools and techniques through development (where necessary) and/or dissemination" (SPEaR, n.d. Roles and functions, para. vii). The SPEAR Good Practice Guidelines are designed to:

... provide practice-based advice aimed at improving social sector research and evaluation systems and processes, enhancing the professional conduct of government officials and external research contractors, improve the generation of information that informs policy development and service delivery, support the saving and sharing of research data and encourage the development of respectful and ethical working relationships between all participants in social sector research and evaluation. (SPEAR 2008:5)

As stated in the Guidelines, good practice is seen to rest on five principles: integrity, respect, responsiveness, competency and reciprocity. The practice expectations that are included are consistent with those outlined in the previous guidelines (SPEAR 2008, Te Puni Kokiri 1999, MSD 2004). The SPEAR Guidelines inform the evaluation of Programme X that follows.

Given that considerable intellectual and organisational work underpins the development of the range of guidelines in the last decade, it is reasonable to expect some success stories in relation to their implementation. No such stories currently circulate, however, and the analysis that follows indicates how stories that do emerge are likely to highlight more challenges than successes.

EVALUATION OF PROGRAMME X

Programme X was a year-long evaluation of a government-funded programme that included Maori as a significant subgroup of participants. The programme that was evaluated was delivered by Maori staff employed by community-based service providers, most of which are iwi-based organisations. Community stakeholders were not involved in key decisions about the evaluation design prior to the request for proposals being developed because there were significant time and budget constraints (Agency contract manager, personal comment, 2008). The evaluation was contracted to an independent evaluation company and led by two experienced Paikeha evaluators. In terms of strategies designed to manage the limited budget, a senior Maori evaluation advisor from the commissioning agency provided 15 hours' support to the project team, and Maori staff, also from the agency, were recruited to assist with fieldwork and analysis.

Method of Review

The review of the activities in Programme X was undertaken prior to the finalisation of the Guidelines, consequently the draft Guidelines were the point of reference. However, the substantive items that comprised the review framework remained consistent between the Draft and final versions. For the analysis of the evaluation programme's uptake and implementation of advice and direction, five systematic (although not exhaustive) steps were followed:

1. An inventory of the practice expectations listed under each of five guideline principles was prepared.

2. The field notes and records made during the evaluation were scanned and additional notes, from the reviewer's own recollections of the process, were compiled.

3. The evaluative activities were broken down and matched with relevant expectations.

4. The matched findings were tabulated.

5. The review process was reviewed by another independent evaluator and the agency contract manager, both of whom confirmed the findings.

A summary of these review findings is given in Table 1. (3)

How this five-step review process worked can be followed in the present context. First, the reviewer considered the principle of respect in the Guidelines, which states that there is an expectation to "Involve Maori participants (whanau, hapu...