Author: Rachael Gibson

Fred Carden

Coherence: A Principles-Focused Approach

• What is Coherence?
  • OECD 1991 – initial focus on policy coherence
  • European Commission 1992 – put into law
  • Focus has been on global issues
  • Enshrined in SDGs
  • No accepted definition
  • OECD defines coherence as: an approach and policy tool to systematically integrate the economic, social and environmental dimensions of development at all stages of domestic and international policy analysis.
• What is Coherence in a Country Program?
  • Development, Trade, Diplomacy – how are they coherent?
  • There are different levels of coherence: 1) Internal Coherence; 2) Intra-Organizational Coherence; 3) Inter-Organizational Coherence; 4) International Coherence
  • Focus of Global Affairs Canada in Columbia case was on first two levels of coherence
  • We first looked at the notion that coherence is not only about policy – what does coherence mean in the context of programming of global affairs in Columbia?
•  Levels of Coherence
  • Policy Coherence
  • Organizational Coherence
  • Administrative Coherence
  • Knowledge Coherence
  • Informal Coherence
• Measuring Coherence
  • Developed four-level rubric – no coherence to coherence
  • Breaking it up into different elements very helpful in discerning what was/was not coherent – Example: coherence across international assistance; coherence across international engagement
  • Found that there was pretty good policy coherence, but administratively weak

Lori Bell

• In the context of UNHCR commissioned evaluations, coherence is always looked at through the lens of relevance – are we doing the right thing?
• Presentation provides concrete examples of evaluations where coherence was looked at from an internal and external point of view

System-wide evaluation of cyclone Idai Mozambique 2019

External Coherence

• Complexity, crowdedness of the humanitarian space
• The divide and conquer of different agencies
• How did we live up to our responsibilities and roles compared to what other actors did?
• If you want to look at External Coherence, you can’t do it on your own – we need to understand the comparative advantage of UNICEF, etc.
• We need to know a lot about other agencies operations and what the government is doing
• Inter-agency joint evaluation, rather than a single agency evaluation

The Importance of Personalities, Perceptions, and Culture

• Coherence boils down to individual behaviours and decision-making on the ground
• We need to look at culture and perceptions

Elliot Stern

Aspects of Policy Coherence

• ‘Policy Coherence’ in EU and OECD
  • Coherence now a “new” (2020) OECD criteria, but entered European policy vocabulary in 1992 Maastricht Treaty
  • Distinction between internal and external coherence – i.e., consistency of elements making up a policy action vs consistency between different policy actions (this distinction may be less clear-cut than 10 or 20 years ago)
• 4 Aspects of Contemporary Policy Coherence
  • Unit of analysis
    • Growing understanding of complex and interdependent policy priorities (e.g., innovation, climate change mitigation, equity and justice)
    • Extending the Unit of Analysis: i) decontextualized policy interventions (coherent with policy inputs); ii) contextualized policy interventions (coherent with policy/non-policy inputs/contexts); iii) interdependent policies interventions (coherent between multiple policies in multiple contexts); iv) policy systems (many coherence possibilities)
    • Systems Mapping for complex policy evaluation
  • Coordination and governance – Coherence doesn’t happen on its own!
    • Multi-part and separate but interdependent policy interventions must be aligned – importance of coordination and governance arrangements
    • Success often depends on pre-existing community, sectoral/territorial networks; credibility of system leaders; capacities to use information and adapt; participatory engagement of stakeholders
    • Coordination and governance itself must be evaluated – part of ToCs, etc.
    • Encouraging those in governance and coordination roles to reflexively learn and generate and use M&E data a way of strengthening coherence
  • Timing: Re-setting the evaluation clock
    • Move away from short-run, ex-post evaluation at end of program/policy cycle to integrate evaluation into program planning and “up-valuing” mid-term/ongoing evaluations
    • Ex-post evaluations are moving long-term – look back over longer time scales on rolling basis
    • Mid-term evaluations increasingly inform reprogramming and “steering” and adaptive management
  • Importance of substantive theory
    • When we move from narrow focus on programs to recognize importance of complexity, context and interdependencies, it changes where we look for theory
    • Program theory is not enough
    • Becoming standard in UK and EU structural funds to look to academic research to identify building blocks of Theories of Change (e.g., Innovations Systems Theory)
    • Centrality of literature reviews and partnerships with domain experts

Vijayalakshmi Vadivelu

UNDP’s programming:

• In past 3 decades, programs have looked at coherence in different program areas – support to livelihoods, peace building, etc. require synergies – intersecting elements to enhance coherence at the country level
• UNDP’s consistent emphasis on coherence and partnerships – implemented with a range of partnerships – integrated role within UN system
• Programming at national and subnational levels enables synergies/linkages

Coherence a key dimension of program performance

• Assessed at multiple levels: Internal, External, Intra-organizational
• Mapping of areas where maximum synergies are possible 
• Also look at intersecting areas in response to crises

External Coherence      

• UNDP collaborates with other actors to contribute to humanitarian response
• Determine pathways that link, map key actors working in the area, identify intersecting process indicators and strength of partnerships
• In bringing together different actors, coherence assessed using sector data
  • Increasing household income, increased investment = indicators for policy coherence

Different types of Coherence – horizontal and vertical pathways

• We try to determine the level of coherence
• Questions regarding linkages to sustainability
• Use causal pathways through the Theory of Change – coherence is one we try to examine
• Multiple methods – determining coherence is more challenging, especially subjectivity in identifying some of the indicators, even where synergies are happening
• Can be helpful to disaggregate and then develop a composite score
• For each level, we use a different rubric for mapping
• Determining national sector policy coherence is often much simpler, but determining the contribution of an agency/program can be extremely challenging

Michael Woolcock

Understanding Trajectories

• Change is rarely linear and can take a long time
• Example – Pace of Change for 5 different social events/processes over last 250 years:
  • In case of interracial marriage, we see that change took forever (nearly 200 years). But even in examples like same-sex marriage that seem to have had a rapid uptake in approval ratings and legality, if we look at the history, we see that people were campaigning for decades before any change really happened.
• Centuries worth of change still unfolding for certain issues (e.g., gender equality); others occur at a much faster rate.
• We need to incorporate these realities much more routinely into how we think about development interventions

Why Trajectories Matter

• We must Take Time Seriously – it is a crucial component to how we understand change
• Example of punctuated equilibrium (e.g., same-sex marriage).  Yet other interventions where the situation gets worse before it gets better (e.g., work on democracy reform, human rights)
• Also “Shooting Star” projects – example where pilot studies yield wonderful results, but when scaled up achieved much more modest gains.
• Baseline and follow-up not enough – need a much more measured sense of when and how things are being achieved
• We can’t expect the full plate of development tasks as rendered in the SDGs to be achieved within 10 years or even 20 years. The presumption that all of this can be achieved by 2030 is entirely sensible for some goals, but wildly unrealistic for others.

Rethinking Theories of Change

• Assessing impact claims against Counterfactuals and ‘Counter-temporals’
  • Focusing on Mechanisms & Processes – how impacts are achieved for different people in different places over different time periods
  • When and how things are happening for whom

Sandra Albert

• Context: North East Region (NER) of India, examples coming from Meghalaya
  • > 3.5 million people; 86% Indigenous/Scheduled Tribes
  • Matrilineal society – children given maternal clan names, women inherit ancestral property, birth of daughters is celebrated (unlike much of rest of India)
• Examples:
  • Pilot workshop on sexuality and reproductive health (SRH) for undergrads in new university in Meghalaya, assuming that SRH training would likely not be needed in such an evolved place. But responses to survey questions revealed otherwise – blaming the victim a prominent theme in responses.
  • SALT (Stimulate, Appreciate, Learn & Transfer) approach to improving immunization uptake in Assam.  Analysis found no effect on Full Immunization Coverage in 12- to 23-month-old children, but NGOs implementing insist a tangible difference is being observed. Is “tyranny of time” contributing to poor outcomes and measurement?
  • Behaviour change can also happen quite quickly – Example of practices related to hand hygiene, respiratory etiquette, and quarantine in Meghalaya pre- and post-COVID-19.
  • High unmet need for contraception – example of vasectomy figures in Meghalaya (0.1%) and reasons for men’s aversion to vasectomy (e.g., fear it would make them “fat and lazy” – associated with local practice of castrating pigs)

• The Meghalaya Paradox:
  • Matrilineal Society (like Kerala)
  • But poor maternal and child health indicators persist
• Context in the NER of India needs to be considered

Boru Douthwaite

“I Wouldn’t Start from Here: Making the Case for Outcome Trajectory Evaluation (OTE)”

• The program Theory of Change is the wrong place to start!
  • Focuses too much on the program, assumes program has more causal power than it actually does
  • Doesn’t adequately take into account broader context in which the program operates, which also has causal power
• Better place to start is the broader context – specifically the outcome trajectories that have produced program outcomes
  • Program outcomes of interest = arrangements of interlinking and self-reinforcing changes that emerge and evolve over time
  • Program outcomes emerge from outcome trajectories – interacting and co-evolving systems of actors, knowledge, technology, and institutions
• Example: Cassava seed certification arrangement to control two viral diseases in Tanzania
• How do we start from a better place?   
• Outcome Trajectory Evaluation – Steps:
  • Step 1: Agree on program outcomes to evaluate
  • Step 2: Select published middle-range theory that best fits the types of change under scrutiny
  • Step 3: Use middle-range theory to help identify and describe the outcome trajectories
  • Step 4: Use middle-range theory as basis of case-specific theories of change, one for each outcome trajectory
  • Step 5: Validate outcome trajectory timelines & case-specific ToC with key stakeholders
  • Step 6: Use case-specific ToC and validated timelines to answer the evaluation questions
  • Step 7: Review of final evaluation report before publication

• Comparison of OTE with Standard Program Evaluation (SPE):
  • Starting Point
    • OTE: program outcomes and the outcome trajectories from which they emerge
    • SPE: theory of change/results framework in program document
  • Use of Theory
    • OTE: use published theory as basis for ToC for each outcome trajectory
    • SPE: test whether ToC specified in Program Document was valid
  • Establishing Causation
    • OTE: by investigating whether: (i) the outcomes exist; (ii) they were generated by the respective trajectories; (iii) the program contributed to the trajectory
    • SPE: by investigating whether program activities contributed to outcomes without really knowing about the synergistic contribution of others. 
  • Learning from one evaluation to the next
    • OTE: possible because middle-range theory provides a common framework to do so
    • SPE: harder without a common framework

Shubham Gupta

“Context(s) Matter – Learning from Collaborative Evaluation in India”

• Why do we need to take context seriously?
  • Human Component – Development programs are not mechanical
  • Micro vs Macro lens to program development and evaluation
  • Realistic expectation of program success
• Outcome Trajectory – Taking the Contact Lens
  • Fascination with averages – leads us to think in a very linear way and miss out on significant small wins
  • Timelines of impact – a dominant context
    • Example: mapping outcome of a maternal health intervention
  • Making course correction based on contextual outcomes
• What can be counted as context?
  • Often, we think of basic social demographics, some kind of political/policy priority, community practices, belief & attitudes
  • Non-traditional ways of understanding context need to be incorporated into the program thinking and evaluation thinking
    • Examples: organizational culture is a very important contextual factor that can drive change, leadership is another major contextual factor, crisis/ecological challenges
• Curious case of infant mortality reduction: UP vs Kerala
  • 2018 snapshot – UP appears to be doing much worse than Kerala and worse than national average
  • When we look at the complete trajectory (1971-2018), the interpretation changes dramatically
  • Adding deeper understanding of context (not just time), and the interpretation changes even further (e.g., infant mortality rate across divisions, multiple kinds of disparities within districts)
• Incorporating contexts in the program’s Theory of Change
• Looking from below – collaborative learning with implementation teams
• Exit strategy and sustainability – based on contextual success

March 4, 2021

Shikha Rana

• Example of Health System Evaluation of Uttar Pradesh Technical Support Unit
• Limitations of traditional health systems evaluation:
  • Change/progress seen in terms of alleviating input constraints
  • Linear frameworks
  • Limited interconnectedness and interdependency across system blocks/pillars
  • Doesn’t capture multi-dimensional and multi-level changes and impact
  • Doesn’t fully incorporate constructs like: capacities, absorptive capacities, and behavioural changes in the eco-system of systems evaluations
• Key Questions to address:
  • How do we define success for a complex health system?
  • How do we ensure sustainability of a successful health system intervention?
  • What is the trade-off between outcomes and capacities?
  • How does evaluative thinking incorporate plans for exit strategies?
• Contextualizing the evaluation
• Absorptive Capacity as applied to the evaluation of the UP Technical Support Unit
• Evaluation Objectives
  • Stakeholders’ perspectives on success and Ownership of the intervention
  • Understand stakeholders’ perspectives on sustaining the intervention without the technical support of the UPTSU, in the context and culture of the UP health system
  • Provide direction for the investments towards the exit strategies; identify gaps and strengths for moving forward

Jindra Cekan

• Assumptions must be tested during project and ex-post:
  • Global Development Aid Industry widely assumes its work is sustainable.  Yet, Valuing Voices research finds less than 1% of all USAID and EU projects have been evaluated ex-post for actual sustainability.
  • Currently, there is little adaptation of timeframes or phasing over or down. Donors mostly exit out after funding has been used.  
• We must see ourselves as them – as “the other”.  Their happiness needs to be as important as ours.  Their sustained health needs to be as important as ours.
• Exit strategies process:  throughout the project, we need to phase down and phase over before we phase out.
• Valuing Voices Sustained Exit Commitments and Conditions Checklist:
  • The most sustained aid projects have an implicit theory of sustainability and a mindset that projects can be locally sustained, that not all activities can/need to be, and a commitment to learning from what can/cannot last, and support what emerges from local priorities and efforts.
  • Key Conditions: Ownership/Motivation, Resources, Capacity Strengthening, Partnerships (including examining Assumptions about Accountability), Timeframe and adaptation of implementation based on M&E of transparent exit benchmarks, Accountable communication
  • Key Commitments: Exit/Handover sustainability phases: 1) phase down over time; 2) phase over to others; 3) phase out (exit at end of funding); Address Risks to Sustainability, e.g., Resilience to Shocks (economic, political)
• We need to do post-project evaluation. We need to design for sustainability and exit with partners.  Create sustainability indicators and checking assumptions.  Sustainability monitoring and adaptation.  We need an informed exit, with stakeholder sustainability consultation at least one year out.
• We must maximize collaborative evaluation focused on what locals can sustain. We need to value their voices, strengthen their ownership, existing capacities, partnerships, and resources.

Kylie Hutchinson 

• What is the difference between programs that manage to stay open after the funding has ended and those that had to close?
• Defines sustainability as:
  • The continuation of successful (pilot) programs after major or seed funding as ended
  • The extent to which (pilot) programs become embedded as core programs (institutionalized)
  • The maintenance of program benefits in community over the long-term
• Key Factors Associated with Program Sustainability (from North American literature):
  • External Funding Environment
  • Length of Funding (longer is better)
  • Diverse Sources of Funding (4-legged table idea)
  • Significant in-kind resources
  • A sustainability plan (organizations that have this)
  • Documented worth and value of a program (evaluation)
  • Community-driven programs
  • Strong volunteer base
  • Volunteer delivery (train the trainer model)
  • Collaborative partners
  • Strong and continuous leadership
  • Engaged funders
  • Fit with host agency
  • Program champions
  • Community ownership
  • High visibility
• See: “Survive and Thrive: Three Steps to Securing your Program’s Sustainability”, by Kylie Hutchinson
• Sustainability Planning Tools – Examples: Program Sustainability Assessment Tool (www.sustaintool.org), Funding Matrix Worksheet, Worst Case Scenario activity, Sustainability Strategies worksheet, Action Plan, Champions Identification Worksheet

April Nakaima 

• “Till Time (and Poor Planning) Do Us Part: Programs as Dynamic Systems – Incorporating Planning of Sustainability into Theories of Change” (Sridharan and Nakaima 2019)
• In program evaluation, we often focus on the program (e.g., activities, outputs, outcomes, etc.). In last 5 years, we’ve been thinking more in terms of the clients’ trajectory
• If you think about the client’s trajectory, it becomes easier to answer the question: what are the key ingredients that we want/need to sustain?
• Example of Evaluation of Local Poverty Reduction Project, Ontario Provincial Government (women’s entrepreneurial program – home-based businesses)
• COM-B Theory of Change Model (John Mayne: 2015, 2017)
  • COM-B = Capabilities, Opportunities, Motivation, Behaviour
  • Model “postulates that behaviour (B) occurs as the result of interaction between three necessary conditions, capabilities (C), opportunities (O), and motivation (M)” (Mayne 2017)
• In the example, the program did a fantastic job in capabilities and motivation. But, it did not address the opportunities piece.
  • Women were in small, poor community and were selling to each other. Needed to make contact with outside markets and richer communities. They did not have the capability within the organization to facilitate that connection to outside markets.
• Program participants were taken half-way across the river, but not all the way across the river.
• Planning for sustainability should be an integral part of what we consider to be a useful theory of change
• Developing the Theory of Change:
  • Make explicit the uncertainties in the theory of change
  • Map the connections between the intervention and rest of the organizational system
  • Explore heterogeneities of client-level needs
  • Explore key barriers that individuals will face in the intended impact journey
  • Explore the potential timelines of impact
  • Explore the dynamic context of supports needed both during and post-funding stages to assist with capabilities, opportunities, and motivational incentives
  • Explore the role of boundary partners

Ray Pawson

• What works for whom – in what circumstances – in what respects – over what duration – and why?
• We are all in tune on the broadening of the evaluation question above to include: over what duration
• But what does sustainability mean in different types of interventions and in different policy domains?
  • Development – aim is to embed infrastructure and associated outcomes on the termination of the programme
  • COVID – desire is to remove programme restriction whilst maintaining its effects

-not to go back to status quo, but to a “new normal”

-time references are opposite and different

• Crime – expectation is to constantly update programme in face of criminal innovation

-situational crime prevention – expectation is that programs never sustain themselves

-come up with better locks, better surveillance systems, etc., but know that within months or years the criminals will figure those out and will need to come up with an improved program

• Law – ambition is for the law to permanently shift moral and disciplinary codes

–example of ban on smoking

–the best laws work because they are moral laws; they don’t depend on policing and surveillance

• Individual – attempts to change individual attributes without being to change their environment

-e.g., mentoring programs aimed at kids in their early years – trying to change the individual, but have little control over the environment in which they live

-sustainability problem is when they go back to their communities, the individual gains will dissipate

• Community – programs seek to improve locality without displacing gains

-the sustainability problem here is about displacement

-if a program is successful, sometimes areas gentrify and displace the locals. Sometimes improving the locality enables locals to leave the area

• Maybe we need to think about all of these different meanings of sustainability because 2 or 3 of them may be at work in any program that you think of.