March 4, 2021

Shikha Rana

  • Example of Health System Evaluation of Uttar Pradesh Technical Support Unit
  • Limitations of traditional health systems evaluation:
    • Change/progress seen in terms of alleviating input constraints
    • Linear frameworks
    • Limited interconnectedness and interdependency across system blocks/pillars
    • Doesn’t capture multi-dimensional and multi-level changes and impact
    • Doesn’t fully incorporate constructs like: capacities, absorptive capacities, and behavioural changes in the eco-system of systems evaluations
  • Key Questions to address:
    • How do we define success for a complex health system?
    • How do we ensure sustainability of a successful health system intervention?
    • What is the trade-off between outcomes and capacities?
    • How does evaluative thinking incorporate plans for exit strategies?
  • Contextualizing the evaluation
  • Absorptive Capacity as applied to the evaluation of the UP Technical Support Unit
  • Evaluation Objectives
    • Stakeholders’ perspectives on success and Ownership of the intervention
    • Understand stakeholders’ perspectives on sustaining the intervention without the technical support of the UPTSU, in the context and culture of the UP health system
    • Provide direction for the investments towards the exit strategies; identify gaps and strengths for moving forward

Jindra Cekan

  • Assumptions must be tested during project and ex-post:
    • Global Development Aid Industry widely assumes its work is sustainable.  Yet, Valuing Voices research finds less than 1% of all USAID and EU projects have been evaluated ex-post for actual sustainability.
    • Currently, there is little adaptation of timeframes or phasing over or down. Donors mostly exit out after funding has been used.  
  • We must see ourselves as them – as “the other”.  Their happiness needs to be as important as ours.  Their sustained health needs to be as important as ours.
  • Exit strategies process:  throughout the project, we need to phase down and phase over before we phase out.
  • Valuing Voices Sustained Exit Commitments and Conditions Checklist:
    • The most sustained aid projects have an implicit theory of sustainability and a mindset that projects can be locally sustained, that not all activities can/need to be, and a commitment to learning from what can/cannot last, and support what emerges from local priorities and efforts.
    • Key Conditions: Ownership/Motivation, Resources, Capacity Strengthening, Partnerships (including examining Assumptions about Accountability), Timeframe and adaptation of implementation based on M&E of transparent exit benchmarks, Accountable communication
    • Key Commitments: Exit/Handover sustainability phases: 1) phase down over time; 2) phase over to others; 3) phase out (exit at end of funding); Address Risks to Sustainability, e.g., Resilience to Shocks (economic, political)
  • We need to do post-project evaluation. We need to design for sustainability and exit with partners.  Create sustainability indicators and checking assumptions.  Sustainability monitoring and adaptation.  We need an informed exit, with stakeholder sustainability consultation at least one year out.
  • We must maximize collaborative evaluation focused on what locals can sustain. We need to value their voices, strengthen their ownership, existing capacities, partnerships, and resources.

Kylie Hutchinson 

  • What is the difference between programs that manage to stay open after the funding has ended and those that had to close?
  • Defines sustainability as:
    • The continuation of successful (pilot) programs after major or seed funding as ended
    • The extent to which (pilot) programs become embedded as core programs (institutionalized)
    • The maintenance of program benefits in community over the long-term
  • Key Factors Associated with Program Sustainability (from North American literature):
    • External Funding Environment
    • Length of Funding (longer is better)
    • Diverse Sources of Funding (4-legged table idea)
    • Significant in-kind resources
    • A sustainability plan (organizations that have this)
    • Documented worth and value of a program (evaluation)
    • Community-driven programs
    • Strong volunteer base
    • Volunteer delivery (train the trainer model)
    • Collaborative partners
    • Strong and continuous leadership
    • Engaged funders
    • Fit with host agency
    • Program champions
    • Community ownership
    • High visibility
  • See: “Survive and Thrive: Three Steps to Securing your Program’s Sustainability”, by Kylie Hutchinson
  • Sustainability Planning Tools – Examples: Program Sustainability Assessment Tool (www.sustaintool.org), Funding Matrix Worksheet, Worst Case Scenario activity, Sustainability Strategies worksheet, Action Plan, Champions Identification Worksheet

April Nakaima 

  • “Till Time (and Poor Planning) Do Us Part: Programs as Dynamic Systems – Incorporating Planning of Sustainability into Theories of Change” (Sridharan and Nakaima 2019)
  • In program evaluation, we often focus on the program (e.g., activities, outputs, outcomes, etc.). In last 5 years, we’ve been thinking more in terms of the clients’ trajectory
  • If you think about the client’s trajectory, it becomes easier to answer the question: what are the key ingredients that we want/need to sustain?
  • Example of Evaluation of Local Poverty Reduction Project, Ontario Provincial Government (women’s entrepreneurial program – home-based businesses)
  • COM-B Theory of Change Model (John Mayne: 2015, 2017)
    • COM-B = Capabilities, Opportunities, Motivation, Behaviour
    • Model “postulates that behaviour (B) occurs as the result of interaction between three necessary conditions, capabilities (C), opportunities (O), and motivation (M)” (Mayne 2017)
  • In the example, the program did a fantastic job in capabilities and motivation. But, it did not address the opportunities piece.
    • Women were in small, poor community and were selling to each other. Needed to make contact with outside markets and richer communities. They did not have the capability within the organization to facilitate that connection to outside markets.
  • Program participants were taken half-way across the river, but not all the way across the river.
  • Planning for sustainability should be an integral part of what we consider to be a useful theory of change
  • Developing the Theory of Change:
    • Make explicit the uncertainties in the theory of change
    • Map the connections between the intervention and rest of the organizational system
    • Explore heterogeneities of client-level needs
    • Explore key barriers that individuals will face in the intended impact journey
    • Explore the potential timelines of impact
    • Explore the dynamic context of supports needed both during and post-funding stages to assist with capabilities, opportunities, and motivational incentives
    • Explore the role of boundary partners

Ray Pawson

  • What works for whom – in what circumstances – in what respects – over what duration – and why?
  • We are all in tune on the broadening of the evaluation question above to include: over what duration
  • But what does sustainability mean in different types of interventions and in different policy domains?
    • Development – aim is to embed infrastructure and associated outcomes on the termination of the programme
    • COVID – desire is to remove programme restriction whilst maintaining its effects

-not to go back to status quo, but to a “new normal”

-time references are opposite and different

  • Crime – expectation is to constantly update programme in face of criminal innovation

-situational crime prevention – expectation is that programs never sustain themselves

-come up with better locks, better surveillance systems, etc., but know that within months or years the criminals will figure those out and will need to come up with an improved program

  • Law – ambition is for the law to permanently shift moral and disciplinary codes

–example of ban on smoking

–the best laws work because they are moral laws; they don’t depend on policing and surveillance

  • Individual – attempts to change individual attributes without being to change their environment

-e.g., mentoring programs aimed at kids in their early years – trying to change the individual, but have little control over the environment in which they live

-sustainability problem is when they go back to their communities, the individual gains will dissipate

  • Community – programs seek to improve locality without displacing gains

-the sustainability problem here is about displacement

-if a program is successful, sometimes areas gentrify and displace the locals. Sometimes improving the locality enables locals to leave the area

  • Maybe we need to think about all of these different meanings of sustainability because 2 or 3 of them may be at work in any program that you think of.