TECCHI’s webinar, “Contexts, Trajectories, and Assessing Progress Towards Goals”, gave rise to a lot of discussion on the topic of timelines and trajectories for understanding programs and their impacts. Below is a table of some insightful comments from different webinar participants as they engaged with the ideas being presented.
“ex-post we find things that ‘fail’ at final, sometimes succeed later, tho’ most of the time those that succeed (and we ASSUME sustained impacts) ‘fail’ a couple of years later.”
“People seem to think that funders and program designers are foolish to look for simple short term outcomes. I think a case can be made that such behavior is rational and the most effective tactic for addressing important issues.
“the short-term is, in my view, irrelevant to our participants and partners who need truly sustainable development that’s locally driven/ sustainable (listen to Time to listen by CDA/Mary Anderson’s crew)
“I agree the short-term view is problematic. I’m only saying that there are good reasons for it. If we don’t appreciate those reasons, progress will never happen.
“Surely the problem Jon Doe, is the preoccupation by traditional evaluators on projects/programmes because that’s who pays us. If you focus on policy and social change then the whole logic if short term impacts becomes absurd!
“Frankly I think we need Theories of Sustainability designed by those tasked with sustaining the outcomes, not just donors ToC during fixed lifetimes.
“I have been thinking about this. It is possible to know some relevant aspects of context in advance, but impossible to know how context affects outcomes except retrospectively.
“measuring multi-dimensional facets of contexts and trajectories should not force to add and burden the questionnaire/respondents…
“context, people and system matter! great said! How do we ensure the linkages to craft our evaluation more practical in day to day application? The human centered evaluation could lead us further towards reality of contexts?
“I always ask the evaluators in training to apply the concepts that we are learning to a program that the learners are working on or passionate about. Situational and Interpersonal Competencies Domains are so important to evaluators. There is more than the technical, reflective and management competencies domains that evaluators need.
Some of these comments shed light on the tensions that are present when thinking about timelines for impact. What are outcomes that we can expect to see in the short term? What is the nature of these outcomes, and how are they different from the long-term impacts that we hope to see in the future? How do short-term outcomes and long-term sustained impacts cohabitate in the same space of program planning?
When establishing timelines for impact, it is important to think about the assumptions that we are making for an impact to be achieved within a given timeline. It’s also perhaps important to reflect on how we go about creating these timelines in which we hope to observe positive change. Sandra Albert, one of the presenters in the webinar, hinted at how interventions are often planned around naive assumptions about the cultural practices in a given context. Given that contextual realities are often like murky water and there are multiple intersecting factors that play a role in the success or failure of interventions, how do we establish “reasonable” timelines for programs? Michael Woolcock, one of the presenters, spoke about the non-linear nature of trajectories and how evaluating an intervention at different points in time could lead to very different understandings on whether a program is successful or not. He also spoke about doing activities in obscurity before change can be observed. These actions often go unnoticed. Some of the webinar participants also mentioned how funding cycles are oftentimes built around achieving short-term wins. What is the value of these short-term gains? Perhaps, achieving these short-term wins that are less complex in nature could help create momentum and generate excitement about a project, and also bring a sense of hope that change is possible. However, it is important to balance this view with a lens of sustainability and helping communities develop the capacity to sustain impacts even once the official funding runs out. Given that observing change takes a lot of time, how do we go about creating theories of sustainability from the onset of a project?
Moving from sustainable impacts and outcome trajectories, to learning systems, I thought that the comments from the webinar participants on “human centered evaluation” and not “burdening” participants with completing questionnaires, quite intriguing. How do we make the process of giving feedback something that participants want to engage with? In order to feel motivated and empowered to provide feedback, and engage in the process of learning, program participants should feel a sense of ownership of the program, and should feel like the input that they are providing will get fed back into the project, ultimately making their experience interacting with the program more pleasurable. As we engage with these questions, it might be beneficial for program implementers and evaluators to collaborate with individuals across other fields such as design, urban planning, psychology etc. to understand how they engage with processes of continuous learning around the interactions of humans with their environment.
Example – Pace of Change for 5 different social events/processes over last 250 years:
In case of interracial marriage, we see that change took forever (nearly 200 years). But even in examples like same-sex marriage that seem to have had a rapid uptake in approval ratings and legality, if we look at the history, we see that people were campaigning for decades before any change really happened.
Centuries worth of change still unfolding for certain issues (e.g., gender equality); others occur at a much faster rate.
We need to incorporate these realities much more routinely into how we think about development interventions
Why Trajectories Matter
We must Take Time Seriously – it is a crucial component to how we understand change
Example of punctuated equilibrium (e.g., same-sex marriage). Yet other interventions where the situation gets worse before it gets better (e.g., work on democracy reform, human rights)
Also “Shooting Star” projects – example where pilot studies yield wonderful results, but when scaled up achieved much more modest gains.
Baseline and follow-up not enough – need a much more measured sense of when and how things are being achieved
We can’t expect the full plate of development tasks as rendered in the SDGs to be achieved within 10 years or even 20 years. The presumption that all of this can be achieved by 2030 is entirely sensible for some goals, but wildly unrealistic for others.
Rethinking Theories of Change
Assessing impact claims against Counterfactuals and ‘Counter-temporals’
Focusing on Mechanisms & Processes – how impacts are achieved for different people in different places over different time periods
When and how things are happening for whom
Context: North East Region (NER) of India, examples coming from Meghalaya
> 3.5 million people; 86% Indigenous/Scheduled Tribes
Matrilineal society – children given maternal clan names, women inherit ancestral property, birth of daughters is celebrated (unlike much of rest of India)
Pilot workshop on sexuality and reproductive health (SRH) for undergrads in new university in Meghalaya, assuming that SRH training would likely not be needed in such an evolved place. But responses to survey questions revealed otherwise – blaming the victim a prominent theme in responses.
SALT (Stimulate, Appreciate, Learn & Transfer) approach to improving immunization uptake in Assam. Analysis found no effect on Full Immunization Coverage in 12- to 23-month-old children, but NGOs implementing insist a tangible difference is being observed. Is “tyranny of time” contributing to poor outcomes and measurement?
Behaviour change can also happen quite quickly – Example of practices related to hand hygiene, respiratory etiquette, and quarantine in Meghalaya pre- and post-COVID-19.
High unmet need for contraception – example of vasectomy figures in Meghalaya (0.1%) and reasons for men’s aversion to vasectomy (e.g., fear it would make them “fat and lazy” – associated with local practice of castrating pigs)
The Meghalaya Paradox:
Matrilineal Society (like Kerala)
But poor maternal and child health indicators persist
Context in the NER of India needs to be considered
“I Wouldn’t Start from Here: Making the Case for Outcome Trajectory Evaluation (OTE)”
The program Theory of Change is the wrong place to start!
Focuses too much on the program, assumes program has more causal power than it actually does
Doesn’t adequately take into account broader context in which the program operates, which also has causal power
Better place to start is the broader context – specifically the outcome trajectories that have produced program outcomes
Program outcomes of interest = arrangements of interlinking and self-reinforcing changes that emerge and evolve over time
Program outcomes emerge from outcome trajectories – interacting and co-evolving systems of actors, knowledge, technology, and institutions
Example: Cassava seed certification arrangement to control two viral diseases in Tanzania
How do we start from a better place?
Outcome Trajectory Evaluation – Steps:
Step 1: Agree on program outcomes to evaluate
Step 2: Select published middle-range theory that best fits the types of change under scrutiny
Step 3: Use middle-range theory to help identify and describe the outcome trajectories
Step 4: Use middle-range theory as basis of case-specific theories of change, one for each outcome trajectory