News

China-Australia Governance Program Shines in Monitoring and Evaluation Review

A recent review of the Monitoring, Evaluation and Quality Assurance Framework (MEQAF) for the China-Australia Governance Program (CAGP) has found it to be ahead of many of the Facilities examined, particularly in terms of the monitoring of quality assurance processes for activity development and implementation.

The recent focus on monitoring and evaluation (M&E) in AusAID can be attributed to domestic as well as international trends. As Australia’s aid budget increases in line with the White Paper there is a growing focus on demonstrating results (being accountable) and learning from these. At the same time, the Paris Declaration on Aid Effectiveness of 2005 and the ongoing work of the OECD Development Assistance Committee have placed development effectiveness high on the agenda of donor agencies. There is thus increasing attention on M&E at all levels of the Australian aid program, from AusAID’s own reporting processes to the M&E arrangements of each AusAID-funded initiative.

Reflecting this wider agenda, the Asia Program Quality Section at AusAID has been conducting a series of M&E Panel exercises with agency staff, contractors, and counterparts across the Asia region since late 2005. Following M&E Panels in Dili, Bangkok, and Manila (twice), Beijing’s turn arrived in November 2007. Facilitated by Graham Rady (Program Quality Advisor) and Mike Crooke (independent M&E consultant), the Beijing M&E Panel sought to: (a) raise awareness amongst AusAID, contractors, and counterpart staff of important M&E standards; (b) develop capacity to assess the quality of M&E and to apply a tool that can be used in planning and implementation; and (c), following a participatory review process, to reach consensus about future actions to improve. As well as the two facilitators, the CAGP M&E Panel involved participants from AusAID (both Australia-based and Overseas-based staff), the National Development and Reform Commission (Jiao Xueli and Candy Yue), the Program Management Office (Michael Young, Fiona Hanrahan, and Peter Leahy), and HAI’s Beijing office (Stewart Norup).

Participants identified and discussed the relative strengths and weaknesses of the MEQAF against six attributes of quality M&E developed by AusAID, namely: (i) an appropriate preparation process; (ii) a design that establishes an appropriate basis for M&E (particularly objectives and deliverables); (iii) clear specification of the information to be collected; (iv) clarity regarding the use and analysis of the information; (v) clear delineation of M&E roles and responsibilities; and (vi) measures to support long-term sustainability of M&E arrangements. Graham and Mike emphasized that these six attributes are intended as a guide rather than a rigid framework since not all criteria are relevant for all initiatives.

The format of the panel allowed for refreshingly open discussion in which the perspectives of different stakeholders could be shared. Highlights of the discussion include:

• The CAGP MEQAF is “cutting edge” in many respects, and is rich in lessons for the new China-Australia Health and HIV/AIDS Facility, the Australia-China Environment Development Program, as well as for AusAID generally. It is important that AusAID actively encourages cross-fertilization of knowledge between these initiatives.
• The complexity of the MEQAF is a barrier to wider ownership. Nevertheless this degree of sophistication is necessitated by unique features of the performance-based contract and the partnership structure.
• The process of scoring all aspects of the program to determine performance-based payments can sometimes distract attention from substantive issues of what is working (or not) and why. A similar debate on the value of performance-based payments systems that rely heavily on scoring is currently taking place in other AusAID country programs such as the Philippines. The complexity of the scoring system remains a subject for ongoing discussion between CAGP stakeholders.
• Systematic ‘triangulation’ from different sources such as the Advisory and Quality Assurance Group (AQAG) are a strength of the MEQAF and allow monitoring information collected by implementing organisations and the Program Management Office to be reviewed and confirmed or challenged.

While the M&E panel process highlighted many strengths of the MEQAF, it is crucial that the framework continues to evolve to reflect new challenges and insights. The annual review process carried out by the AQAG, and the ongoing M&E support that HAI has provided are important to enable this ‘virtuous cycle’ of continuous revision. In this sense then, the MEQAF remains perpetually unfinished.

Stewart Norup

Consultant / M&E Specialist

HAI Beijing