Embracing Complexity : Making Evaluations Matter

Category: Webinar 2

Insights from webinar two “Contexts, Trajectories, and Assessing Progress Towards Goals”

TECCHI’s webinar, “Contexts, Trajectories, and Assessing Progress Towards Goals”, gave rise to a lot of discussion on the topic of timelines and trajectories for understanding programs and their impacts. Below is a table of some insightful comments from different webinar participants as they engaged with the ideas being presented.

“ex-post we find things that ‘fail’ at final, sometimes succeed later, tho’ most of the time those that succeed (and we ASSUME sustained impacts) ‘fail’ a couple of years later.”
“People seem to think that funders and program designers are foolish to look for simple short term outcomes. I think a case can be made that such behavior is rational and the most effective tactic for addressing important issues.
“the short-term is, in my view, irrelevant to our participants and partners who need truly sustainable development that’s locally driven/ sustainable (listen to Time to listen by CDA/Mary Anderson’s crew)
“I agree the short-term view is problematic. I’m only saying that there are good reasons for it. If we don’t appreciate those reasons, progress will never happen.
“Surely the problem Jon Doe, is the preoccupation by traditional evaluators on projects/programmes because that’s who pays us. If you focus on policy and social change then the whole logic if short term impacts becomes absurd!
“Frankly I think we need Theories of Sustainability designed by those tasked with sustaining the outcomes, not just donors ToC during fixed lifetimes.
“I have been thinking about this. It is possible to know some relevant aspects of context in advance, but impossible to know how context affects outcomes except retrospectively.
“measuring multi-dimensional facets of contexts and trajectories should not force to add and burden the questionnaire/respondents…
“context, people and system matter! great said! How do we ensure the linkages to craft our evaluation more practical in day to day application? The human centered evaluation could lead us further towards reality of contexts?
“I always ask the evaluators in training to apply the concepts that we are learning to a program that the learners are working on or passionate about. Situational and Interpersonal Competencies Domains are so important to evaluators. There is more than the technical, reflective and management competencies domains that evaluators need.

Some of these comments shed light on the tensions that are present when thinking about timelines for impact. What are outcomes that we can expect to see in the short term? What is the nature of these outcomes, and how are they different from the long-term impacts that we hope to see in the future? How do short-term outcomes and long-term sustained impacts cohabitate in the same space of program planning?

When establishing timelines for impact, it is important to think about the assumptions that we are making for an impact to be achieved within a given timeline. It’s also perhaps important to reflect on how we go about creating these timelines in which we hope to observe positive change. Sandra Albert, one of the presenters in the webinar, hinted at how interventions are often planned around naive assumptions about the cultural practices in a given context. Given that contextual realities are often like murky water and there are multiple intersecting factors that play a role in the success or failure of interventions, how do we establish “reasonable” timelines for programs? Michael Woolcock, one of the presenters, spoke about the non-linear nature of trajectories and how evaluating an intervention at different points in time could lead to very different understandings on whether a program is successful or not. He also spoke about doing activities in obscurity before change can be observed. These actions often go unnoticed. Some of the webinar participants also mentioned how funding cycles are oftentimes built around achieving short-term wins. What is the value of these short-term gains? Perhaps, achieving these short-term wins that are less complex in nature could help create momentum and generate excitement about a project, and also bring a sense of hope that change is possible. However, it is important to balance this view with a lens of sustainability and helping communities develop the capacity to sustain impacts even once the official funding runs out. Given that observing change takes a lot of time, how do we go about creating theories of sustainability from the onset of a project?

Moving from sustainable impacts and outcome trajectories, to learning systems, I thought that the comments from the webinar participants on “human centered evaluation” and not “burdening” participants with completing questionnaires, quite intriguing. How do we make the process of giving feedback something that participants want to engage with? In order to feel motivated and empowered to provide feedback, and engage in the process of learning, program participants should feel a sense of ownership of the program, and should feel like the input that they are providing will get fed back into the project, ultimately making their experience interacting with the program more pleasurable. As we engage with these questions, it might be beneficial for program implementers and evaluators to collaborate with individuals across other fields such as design, urban planning, psychology etc. to understand how they engage with processes of continuous learning around the interactions of humans with their environment.

Summary of Presentations | Contexts, Trajectories, and Assessing Progress Towards Goals

Michael Woolcock

Understanding Trajectories

  • Change is rarely linear and can take a long time
  • Example – Pace of Change for 5 different social events/processes over last 250 years:
    • In case of interracial marriage, we see that change took forever (nearly 200 years). But even in examples like same-sex marriage that seem to have had a rapid uptake in approval ratings and legality, if we look at the history, we see that people were campaigning for decades before any change really happened.
  • Centuries worth of change still unfolding for certain issues (e.g., gender equality); others occur at a much faster rate.
  • We need to incorporate these realities much more routinely into how we think about development interventions

Why Trajectories Matter

  • We must Take Time Seriously – it is a crucial component to how we understand change
  • Example of punctuated equilibrium (e.g., same-sex marriage).  Yet other interventions where the situation gets worse before it gets better (e.g., work on democracy reform, human rights)
  • Also “Shooting Star” projects – example where pilot studies yield wonderful results, but when scaled up achieved much more modest gains.
  • Baseline and follow-up not enough – need a much more measured sense of when and how things are being achieved
  • We can’t expect the full plate of development tasks as rendered in the SDGs to be achieved within 10 years or even 20 years. The presumption that all of this can be achieved by 2030 is entirely sensible for some goals, but wildly unrealistic for others.

Rethinking Theories of Change

  • Assessing impact claims against Counterfactuals and ‘Counter-temporals’
    • Focusing on Mechanisms & Processes – how impacts are achieved for different people in different places over different time periods
    • When and how things are happening for whom

Sandra Albert

  • Context: North East Region (NER) of India, examples coming from Meghalaya
    • > 3.5 million people; 86% Indigenous/Scheduled Tribes
    • Matrilineal society – children given maternal clan names, women inherit ancestral property, birth of daughters is celebrated (unlike much of rest of India)
  • Examples:
    • Pilot workshop on sexuality and reproductive health (SRH) for undergrads in new university in Meghalaya, assuming that SRH training would likely not be needed in such an evolved place. But responses to survey questions revealed otherwise – blaming the victim a prominent theme in responses.
    • SALT (Stimulate, Appreciate, Learn & Transfer) approach to improving immunization uptake in Assam.  Analysis found no effect on Full Immunization Coverage in 12- to 23-month-old children, but NGOs implementing insist a tangible difference is being observed. Is “tyranny of time” contributing to poor outcomes and measurement?
    • Behaviour change can also happen quite quickly – Example of practices related to hand hygiene, respiratory etiquette, and quarantine in Meghalaya pre- and post-COVID-19.
    • High unmet need for contraception – example of vasectomy figures in Meghalaya (0.1%) and reasons for men’s aversion to vasectomy (e.g., fear it would make them “fat and lazy” – associated with local practice of castrating pigs)
  • The Meghalaya Paradox:
    • Matrilineal Society (like Kerala)
    • But poor maternal and child health indicators persist
  • Context in the NER of India needs to be considered

Boru Douthwaite

“I Wouldn’t Start from Here: Making the Case for Outcome Trajectory Evaluation (OTE)”

  • The program Theory of Change is the wrong place to start!
    • Focuses too much on the program, assumes program has more causal power than it actually does
    • Doesn’t adequately take into account broader context in which the program operates, which also has causal power
  • Better place to start is the broader context – specifically the outcome trajectories that have produced program outcomes
    • Program outcomes of interest = arrangements of interlinking and self-reinforcing changes that emerge and evolve over time
    • Program outcomes emerge from outcome trajectories – interacting and co-evolving systems of actors, knowledge, technology, and institutions
  • Example: Cassava seed certification arrangement to control two viral diseases in Tanzania
  • How do we start from a better place?   
  • Outcome Trajectory Evaluation – Steps:
    • Step 1: Agree on program outcomes to evaluate
    • Step 2: Select published middle-range theory that best fits the types of change under scrutiny
    • Step 3: Use middle-range theory to help identify and describe the outcome trajectories
    • Step 4: Use middle-range theory as basis of case-specific theories of change, one for each outcome trajectory
    • Step 5: Validate outcome trajectory timelines & case-specific ToC with key stakeholders
    • Step 6: Use case-specific ToC and validated timelines to answer the evaluation questions
    • Step 7: Review of final evaluation report before publication
  • Comparison of OTE with Standard Program Evaluation (SPE):
    • Starting Point
      • OTE: program outcomes and the outcome trajectories from which they emerge
      • SPE: theory of change/results framework in program document
    • Use of Theory
      • OTE: use published theory as basis for ToC for each outcome trajectory
      • SPE: test whether ToC specified in Program Document was valid
    • Establishing Causation
      • OTE: by investigating whether: (i) the outcomes exist; (ii) they were generated by the respective trajectories; (iii) the program contributed to the trajectory
      • SPE: by investigating whether program activities contributed to outcomes without really knowing about the synergistic contribution of others. 
    • Learning from one evaluation to the next
      • OTE: possible because middle-range theory provides a common framework to do so
      • SPE: harder without a common framework

Shubham Gupta

“Context(s) Matter – Learning from Collaborative Evaluation in India”

  • Why do we need to take context seriously?
    • Human Component – Development programs are not mechanical
    • Micro vs Macro lens to program development and evaluation
    • Realistic expectation of program success
  • Outcome Trajectory – Taking the Contact Lens
    • Fascination with averages – leads us to think in a very linear way and miss out on significant small wins
    • Timelines of impact – a dominant context
      • Example: mapping outcome of a maternal health intervention
    • Making course correction based on contextual outcomes
  • What can be counted as context?
    • Often, we think of basic social demographics, some kind of political/policy priority, community practices, belief & attitudes
    • Non-traditional ways of understanding context need to be incorporated into the program thinking and evaluation thinking
      • Examples: organizational culture is a very important contextual factor that can drive change, leadership is another major contextual factor, crisis/ecological challenges
  • Curious case of infant mortality reduction: UP vs Kerala
    • 2018 snapshot – UP appears to be doing much worse than Kerala and worse than national average
    • When we look at the complete trajectory (1971-2018), the interpretation changes dramatically
    • Adding deeper understanding of context (not just time), and the interpretation changes even further (e.g., infant mortality rate across divisions, multiple kinds of disparities within districts)
  • Incorporating contexts in the program’s Theory of Change
  • Looking from below – collaborative learning with implementation teams
  • Exit strategy and sustainability – based on contextual success