Rapidly Expanding Access to Care for hiv in Tanah Papua (reach) 2012 2016

3.4 Monitoring and evaluation

A guiding principle of REACH is that the program will be guided by the best available data. The majority of these data will be collected during the course of service delivery through routine monitoring systems. The monitoring system will ensure that data are appropriately analysed, regularly fed back to decision-makers and service providers, and used actively to manage the program and keep it on course towards its goals.

The focus of REACH monitoring systems is very clearly on ongoing program management and improvement, and no full-scale internal evaluations are planned during the life of the program. However CHAI strongly believes that transparency contributes to credibility and encourages high quality service delivery; monitoring data will be made available to anyone with a legitimate interest in the provision of HIV-related services in Papua. This will allow external evaluators to judge the contribution of the program, should AusAID wish to fund a comprehensive evaluation either during or after the program.

Eight per cent of the program budget has been allocated for operational research and monitoring activities.

3.4.1 The monitoring framework

Monitoring systems are most useful when they measure things that program implementers want to know; the people best placed to decide on information needs are not proposal writing teams, but program managers and implementing staff. After discussion with AusAID's Health Unit, it was therefore decided that a full monitoring framework would be developed at a very early stage of REACH implementation, rather than as part of this proposal. The monitoring system will be developed in accordance with the standards of design stipulated by AusAID Indonesia.

The process of developing a framework and set of indicators will be led by an Operational Research and Monitoring Advisor who will be contracted to oversee these functions over the life of the program. The Advisor will work with the Operations Research and Monitoring Manager, in tandem with the process of defining operational research questions described in section 2.8 as a priority task in the first quarter of implementation. Reporting frequencies, data sources and indicators will be decided for each program component in consultation with program staff, CoE, MoH, PHOs, DHOs and AusAID. (Indicative indicators are suggested in the ‘How will we know?’ sections of the program description: sections 2.5 – 2.8. REACH will also monitor how HIV-related services are raising standards and uptake of other non-HIV services.) The monitoring framework will, where appropriate, choose indicators that are already required by the MoH and its other international partners. We will, however, review existing indicators for utility, and will guard against letting the indicator tail wag the implementation dog.

One of the most common weaknesses of monitoring systems is that they are focused on mandated reporting requirements rather than on program management. They typically overinvest in collecting data that fit into predetermined boxes, underinvest in data management, and completely neglect rapid analysis of data and use of those analyses to guide program implementation in the field. REACH will avoid these pitfalls by structuring monitoring systems to prioritise flexible analysis and feedback.

Key service utilisation data will disaggregated by sex and analysed for gender differences. As needed, strategies to deal with gender inequality will be developed.

The monitoring structure for REACH and description of staffing support is in Annex 13.

3.4.2 Data feedback and use mechanisms

In order to ensure that data are actively used in program management, regular ‘live analysis’ sessions will be scheduled. These meetings will be structured as follows:

• 
  Report by program managers of changes made to program implementation as a result of the findings of the previous program analysis session
  
• 
  Presentation by analysts of any changes in service provision indicators subsequent to those changes in program implementation
  
• 
  Presentation by analysts of:
  
  • 
    thematic analysis (for example, one session might concentrate on STI services, the next on TB-HIV)
    
  • 
    core progress indicators
    
  • 
    any operational research results
    
• 
  Discussion of implications of analysis for programming (this is the ‘live analysis’ part: analysts may interrogate the data on the spot to support or refute points raised in the discussion and to ensure that decisions are informed by the facts).
  
• 
  Necessary changes to program strategies and implementation are decided. Program managers will report back on the implementation of these changes at the start of the next analysis session.
  

The provincial program analysts will be responsible for recording the outcomes of these meetings, and for systematically tracking changes in program implementation that take place as a result.

These meetings will be scheduled as follows:

• 
  At the CoE level, every month. These meetings will include REACH program managers and monitoring staff, CoE analysts and monitoring staff, CoE managers and service providers and DHO staff, and puskesmas program managers where possible.
  
• 
  At the provincial level, every three months. These meetings will include REACH provincial program managers and monitoring staff, CoE analysts and monitoring staff, and CoE managers and service providers where possible, PHO staff, DHO staff where possible, and members of the Provincial AIDS Commission CST working group. Other members of the Provincial Steering Committees will be invited to attend.
  
• 
  At the cross province level, every six months. Every second provincial meeting will be held jointly between the two provinces, alternating in location between Papua and West Papua.
  
• 
  Nationally, once a year. The Jakarta-based M&E specialist and staff managing the supply chain, policy and operational research components of the REACH program will join the Tanah Papua meetings once a year to review their progress and program synergies. The members of both PSCs will also be invited to attend one joint meeting annually.
  

3.4.3 Baseline data

Most of the data used in the REACH monitoring framework will come from routine reporting of service delivery. We recognise the importance of having solid baseline data against which we can measure progress. Good baseline data also help to refine program implementation. Baseline data will be collected shortly after completion of the monitoring framework. There are three important sources of baseline data in existence or planned and budgeted:

• 
  Existing service provision recording systems. Record keeping in most sites - hospitals, puskesmas and laboratories is relatively thorough. The ledgers kept by service providers are the most important source of baseline data for REACH. However none of it is computerised. To build the foundation of the REACH monitoring system, the program will provide short-term contract staff that will enter all the existing HIV-related ledger data into standardised databases that can be maintained at the puskesmas level. Where necessary, REACH will provide puskesmas with laptops and software to enable them to perform these tasks. Routine site visits will verify that equipment is being used.
  
• 
  The 2011 Health Facilities Survey, conducted by the MoH. Though this focuses mainly on physical infrastructure and official staffing levels, CHAI Phase II was able to add questions of relevance to the REACH program. These data should be made available shortly. The operational research component of REACH will carry out validation research on these data.
  
• 
  The 2012 HIV and STI IBBS survey should provide important data on HIV and syphilis prevalence levels, as well as on HIV testing history and other important HIV-related issues. AusAID is planning to provide extra funding for this survey, to allow for robust estimates to be made in the highland areas where REACH will work. A guarantee of access to raw data as well as strong quality control will be essential.
  

An important output of the REACH program will be strong networks and relationships between policy makers and service providers at different levels. We will measure progress towards building these networks through relationship mapping. This monitoring method is new. REACH will provide technical assistance to build skills in this area, and to develop baseline maps.

3.4.4 Longer term evaluation: a recommendation to AusAID

"Sustainability" is on everyone's lips, but in no-one's evaluation plans. This is in part because a program's monitoring systems and evaluation budget die with the program. The REACH program aims to ensure that the various governments of Tanah Papua take responsibility for services funded under REACH before the end of the program. However sustainability can only realistically be measured after a reasonable delay. The team that developed this proposal strongly recommends that AusAID schedules a sustainability evaluation two years after the REACH program ends. REACH will endeavour to set up data management systems that will allow changes to be measured beyond the life of the program.