Michael Woolcock

Understanding Trajectories

  • Change is rarely linear and can take a long time
  • Example – Pace of Change for 5 different social events/processes over last 250 years:
    • In case of interracial marriage, we see that change took forever (nearly 200 years). But even in examples like same-sex marriage that seem to have had a rapid uptake in approval ratings and legality, if we look at the history, we see that people were campaigning for decades before any change really happened.
  • Centuries worth of change still unfolding for certain issues (e.g., gender equality); others occur at a much faster rate.
  • We need to incorporate these realities much more routinely into how we think about development interventions

Why Trajectories Matter

  • We must Take Time Seriously – it is a crucial component to how we understand change
  • Example of punctuated equilibrium (e.g., same-sex marriage).  Yet other interventions where the situation gets worse before it gets better (e.g., work on democracy reform, human rights)
  • Also “Shooting Star” projects – example where pilot studies yield wonderful results, but when scaled up achieved much more modest gains.
  • Baseline and follow-up not enough – need a much more measured sense of when and how things are being achieved
  • We can’t expect the full plate of development tasks as rendered in the SDGs to be achieved within 10 years or even 20 years. The presumption that all of this can be achieved by 2030 is entirely sensible for some goals, but wildly unrealistic for others.

Rethinking Theories of Change

  • Assessing impact claims against Counterfactuals and ‘Counter-temporals’
    • Focusing on Mechanisms & Processes – how impacts are achieved for different people in different places over different time periods
    • When and how things are happening for whom

Sandra Albert

  • Context: North East Region (NER) of India, examples coming from Meghalaya
    • > 3.5 million people; 86% Indigenous/Scheduled Tribes
    • Matrilineal society – children given maternal clan names, women inherit ancestral property, birth of daughters is celebrated (unlike much of rest of India)
  • Examples:
    • Pilot workshop on sexuality and reproductive health (SRH) for undergrads in new university in Meghalaya, assuming that SRH training would likely not be needed in such an evolved place. But responses to survey questions revealed otherwise – blaming the victim a prominent theme in responses.
    • SALT (Stimulate, Appreciate, Learn & Transfer) approach to improving immunization uptake in Assam.  Analysis found no effect on Full Immunization Coverage in 12- to 23-month-old children, but NGOs implementing insist a tangible difference is being observed. Is “tyranny of time” contributing to poor outcomes and measurement?
    • Behaviour change can also happen quite quickly – Example of practices related to hand hygiene, respiratory etiquette, and quarantine in Meghalaya pre- and post-COVID-19.
    • High unmet need for contraception – example of vasectomy figures in Meghalaya (0.1%) and reasons for men’s aversion to vasectomy (e.g., fear it would make them “fat and lazy” – associated with local practice of castrating pigs)
  • The Meghalaya Paradox:
    • Matrilineal Society (like Kerala)
    • But poor maternal and child health indicators persist
  • Context in the NER of India needs to be considered

Boru Douthwaite

“I Wouldn’t Start from Here: Making the Case for Outcome Trajectory Evaluation (OTE)”

  • The program Theory of Change is the wrong place to start!
    • Focuses too much on the program, assumes program has more causal power than it actually does
    • Doesn’t adequately take into account broader context in which the program operates, which also has causal power
  • Better place to start is the broader context – specifically the outcome trajectories that have produced program outcomes
    • Program outcomes of interest = arrangements of interlinking and self-reinforcing changes that emerge and evolve over time
    • Program outcomes emerge from outcome trajectories – interacting and co-evolving systems of actors, knowledge, technology, and institutions
  • Example: Cassava seed certification arrangement to control two viral diseases in Tanzania
  • How do we start from a better place?   
  • Outcome Trajectory Evaluation – Steps:
    • Step 1: Agree on program outcomes to evaluate
    • Step 2: Select published middle-range theory that best fits the types of change under scrutiny
    • Step 3: Use middle-range theory to help identify and describe the outcome trajectories
    • Step 4: Use middle-range theory as basis of case-specific theories of change, one for each outcome trajectory
    • Step 5: Validate outcome trajectory timelines & case-specific ToC with key stakeholders
    • Step 6: Use case-specific ToC and validated timelines to answer the evaluation questions
    • Step 7: Review of final evaluation report before publication
  • Comparison of OTE with Standard Program Evaluation (SPE):
    • Starting Point
      • OTE: program outcomes and the outcome trajectories from which they emerge
      • SPE: theory of change/results framework in program document
    • Use of Theory
      • OTE: use published theory as basis for ToC for each outcome trajectory
      • SPE: test whether ToC specified in Program Document was valid
    • Establishing Causation
      • OTE: by investigating whether: (i) the outcomes exist; (ii) they were generated by the respective trajectories; (iii) the program contributed to the trajectory
      • SPE: by investigating whether program activities contributed to outcomes without really knowing about the synergistic contribution of others. 
    • Learning from one evaluation to the next
      • OTE: possible because middle-range theory provides a common framework to do so
      • SPE: harder without a common framework

Shubham Gupta

“Context(s) Matter – Learning from Collaborative Evaluation in India”

  • Why do we need to take context seriously?
    • Human Component – Development programs are not mechanical
    • Micro vs Macro lens to program development and evaluation
    • Realistic expectation of program success
  • Outcome Trajectory – Taking the Contact Lens
    • Fascination with averages – leads us to think in a very linear way and miss out on significant small wins
    • Timelines of impact – a dominant context
      • Example: mapping outcome of a maternal health intervention
    • Making course correction based on contextual outcomes
  • What can be counted as context?
    • Often, we think of basic social demographics, some kind of political/policy priority, community practices, belief & attitudes
    • Non-traditional ways of understanding context need to be incorporated into the program thinking and evaluation thinking
      • Examples: organizational culture is a very important contextual factor that can drive change, leadership is another major contextual factor, crisis/ecological challenges
  • Curious case of infant mortality reduction: UP vs Kerala
    • 2018 snapshot – UP appears to be doing much worse than Kerala and worse than national average
    • When we look at the complete trajectory (1971-2018), the interpretation changes dramatically
    • Adding deeper understanding of context (not just time), and the interpretation changes even further (e.g., infant mortality rate across divisions, multiple kinds of disparities within districts)
  • Incorporating contexts in the program’s Theory of Change
  • Looking from below – collaborative learning with implementation teams
  • Exit strategy and sustainability – based on contextual success