Evaluating Sustainable Development Goals

Embracing Complexity : Making Evaluations Matter

Should we hold low-income countries accountable for leaving no-one behind?

May 31, 2021 / / 3 Comments

“To accomplish great things, we must not only act, but also dream; not only plan, but also believe.”

“It is well for the heart to be naïve and for the mind not to be.“

Anatole France

THE GOOD INTENTION

Leaving no-one behind is the “transformative promise” of the SDGs. It has become a mantra in development planning. It is everywhere, invading discussions, plans and interventions with a raft of good intentions. It has even developed into an acronym, a sure sign of having arrived in a world awash with short forms of speech. It is a persuasive argument that unless those worst off see their lives dramatically improved, we cannot claim that the SDGs have been met:

“As we embark on this great collective journey, we pledge that no-one will be left behind. Recognising that the dignity of the human person is fundamental, we wish to see the Goals and targets met for all nations and peoples and for all segments of society. And we will endeavour to reach the furthest behind first.” (my emphasis)

And:

“By 2030, empower and promote the social, economic and political inclusion of all, irrespective of age, sex, disability, race, ethnicity, origin, religion, or economic or other status.”

Governments and those engaged in development everywhere feel compelled to attend to it, and over the past five years evaluators, evaluation thinkers and evaluation commissioners have had this notion firmly in their line of sight. Speakers in conferences and webinars continue to ponder on how best to help ensure NOLB through evaluative practice. Discussion documents, guidelines and evaluations with a NOLB focus have proliferated, giving hope and showing strong commitment to the intent in spite of the challenge of moving from rhetoric to action.

THE PROBLEM

In their thoughtful 2019 book, Leave no-one behind: Time for specifics on the SDGs, Homi Kharas and his co-authors bring together many perspectives and experiences applied to different NOLB foci – gender, poverty, farmers, refugees and migrants, education, health care and more. Like so many articles and reports, they offer insights and recommendations on what can be done. They also eloquently point out the scale of the challenge, and the countries most deeply affected.

The key point? The countries where NOLB is of greatest concern, where most of the interventions are concentrated, are all in the Global South, that part of the world in shades of red or some other vibrant “this is bad” colour in just about every map that visualises data on the state of the world, where everything to do with development is more challenging, more resource-intensive, and a greater burden on governments and others who want to make a difference. 

This means that the burden of supporting and sustaining a majority of ‘leaving no-one behind’ efforts fall inevitably on many of the poorest (low-income) countries in the Global South.

The problem is that they cannot afford it, nor can they sustain it.

It will therefore be unfair to hold such countries accountable for ‘leaving no-one behind’ strategies. 

THE ETHICS OF ‘LEAVING NO-ONE BEHIND’

There are two sides to the ethical imperative for ‘leaving no-one behind’.

It is unethical to keep on neglecting the most vulnerable and those on the margins of society. It is equally unethical to intervene with time-bound interventions to ‘leave no-one behind’ when they have very little chance of being sustained once the source of financing leaves. This causes too much destruction of societal systems, too much hope that is created and then shattered, too much behaviour that is changed temporarily without thinking of the long-term, often negative consequences.

There is the need to dream, to have spectacular ambitions. But there is also the need for pragmatism among those engaged with development as well as those who evaluate for it.

As Oumoul Ba Tal from Mauritania, a former President of the African Evaluation Association, once said in a heated exchange with a senior evaluation specialist from an international agency:

“It is not about your project; it is about my country”. 

THE NEED FOR PRAGMATISM  

The pressure to achieve the SDGs means that aid agencies inevitably have a focus on no-one left behind. Authorities in the Global South also feel compelled to attend it as a priority. It is a fact that this type of focus resonates well with all of us. Development and evaluation specialists are at heart most often idealists who want to help to make the world a better place. Aiming to ensure that no-one gets left behind appears to be a noble part of what we can do at this time in the world. It has become soft-on-the-ear-and-mind rhetoric, something to be done because it sounds and feels right.

But from a pragmatic perspective, if we really want to see development that is effective and that sustains, and that eventually succeeds in leaving no-one behind, we have to consider the following:

One, the level of cost and effort matter. In some countries ‘no-one left behind’ relates to the urban-rural divide; in others to the invisibility everywhere of the poorest of the poor, the most marginalised, the most neglected; in others to those displaced or victims of war. Those left behind typically present a ‘last mile’ problem that is always much more costly in terms of effort, resources and time – and even much more so in low-income or ‘least developed’ countries where whole systems need to co-evolve to ensure a positive development trajectory over a long period. It becomes even more problematic if the intent is to address first those who would qualify as most marginalised or vulnerable, as the 2030 Agenda for Sustainable Development and even the recently released guidance for the revised DAC criteria both suggest. This is a highly undesirable approach, also for the reasons that follow.

Two, sustainability matters. Development projects and programmes tend to be terminated after three or five or perhaps ten years, often without having achieved much. In low-income countries the chance that even well-crafted and implemented interventions aimed at ensuring no-one is left behind will sustain is highly unlikely without a massive, intensive effort over a significant period. Once aid agencies leave, local actors have to pick up the pieces – something that is incredibly difficult to do in the often fragile, resource-constrained contexts in much of the Global South.

Three, long-term development trajectories matter. Countries that are already well advanced on a positive development trajectory – where many vectors or indicators of national progress move in desired directions, where many institutional and other systems have already advanced and reached some form of reasonable stability – will find it much easier to make the ‘last mile’ no-one left behind effort. This is why China, despite its extraordinarily rapid economic development, waited until recently before focusing on eradicating extreme poverty in rural areas.

Four, dealing with choices and trade-offs matter. If a low income country starts to move along a positive development trajectory, trade-offs will be required. One of them is that some will have to be left behind, until infrastructure and institutional systems, societal relationships and capacities have evolved to a stage where there is enough to support and sustain the development of those who are most marginalised and vulnerable. What is important is how carefully and systematically inequalities are uncovered through evaluative practices, and managed and slowly woven into strategies as they evolve throughout the different stages of development. In other words, mindful that there will be trade-offs, the notion that no-one should be left behind can be implemented systematically and pragmatically rather than too quickly based on the idealism reflected in so many unsustainable development interventions.

Five, complexity concepts matter, in particular ‘co-evolution’. All of the above reflect the need to view development through a complex adaptive systems lens – considering the path-dependence of societies’ development, a long-term development trajectory instead of snapshot thinking, dealing pragmatically and systematically with choices and trade-offs, and ensuring the type of sequencing of action that will help ensure that positive results sustain.

Most importantly for successfully dealing with no-one left behind strategies is recognising that system co-evolve, especially in countries where much starts from a low base. Development normally requires societal values and beliefs, capacities, institutions, markets and more to evolve together. This takes time and ongoing effort. In low-income countries interventions can try to nudge systems in desired directions, but if they try to shift part of a system or a nested system too quickly, it may not work; it will strive to return to its former state.

THE RESPONSIBILITY OF EVALUATION SPECIALISTS

Aid funding is increasingly concentrated in low(est) income countries in the Global South, and even within these contexts there is often a focus on the most vulnerable and fragile. It is crucial to think during evaluations about the systems on which they will depend when donor funding ends. What will be expected from local and national authorities when the financiers withdraw?

If the potential for sustainability of any positive achievements is lacking when financing comes to an end, any assessment against criteria such as ‘relevance’, ‘effectiveness’, ‘efficiency’ and ‘impact will mislead unless others take on or complement the support. This places a major responsibility on evaluation professionals to be very careful in the assessments that we make, and to consider the implications for evaluative practice of issues raised in this post.

As Oumoul Ba Tal inferred so long ago, we need a country and systems lens, not a project lens, if we truly want to ensure development that sustains and bears fruit for all once the financiers have left.

The Role of Evaluation in Navigating Spectacular Ambitions

May 24, 2021 / / 0 Comments

The Aspiration and the Responsibility

The Sustainable Development Goals (SDGs) provide a spectacular vision for

“eradicating poverty in all its forms and dimensions, combatting inequality within and among countries, preserving the planet, creating sustained, inclusive and sustainable economic growth and fostering social inclusion are linked to each other and are interdependent.” 

Further, the SDGs have a sharp focus on “leaving no one behind”:

“As we embark on this great collective journey, we pledge that no one will be left behind. Recognizing that the dignity of the human person is fundamental, we wish to see the Goals and targets met for all nations and peoples and for all segments of society. And we will endeavour to reach the furthest behind first.“

Such grand visioning and rhetoric need to be tempered by harsh realities how COVID, Climate Change, debts and discriminatory practices and structures have had cumulative and intersectional impacts on poor countries and poor people.  

Change often happens through ambitions and aspirations to achieve the near impossible. While the cold gaze of measurement and design should not dampen the enthusiasm inherent in long-term ambitions, the evaluation community could ethically and responsibly provoke questions around the types of systems changes, interventions, partnerships, governance structures and learning systems that might need to be in place to help achieve the goals.

Good evaluations often start with understanding pathways of impacts. For example, the whole essence of theory-of-change approaches to evaluation question is the humble question: How?  

I work in development/global health, with a majority of my work in the past three years in India. While there is a clear focus on indicators both in the SDG documents and planned interventions/systems change efforts, there continue to be ambiguities on the roles that evaluation and evaluators can play in helping achieve such spectacular ambitions.   

In this blog I raise questions related to the role evaluations need to play in helping achieve such grand goals. How do we move from a position of being passive responders to a more active stance that demonstrates leadership for enhancing the likelihood of achieving such grand visions?  

Getting Real

Understanding the cumulative, dynamic pressures of COVID, debts, discrimination and climate change become especially important as we seek to locate our role. Consider this recent article from April 2021:

“Without climate finance, poor countries face a bleak future of extreme weather, water and food shortages, and climate-driven migration, which all threaten to reverse decades of progress in lifting people out of poverty. Many governments are also being wooed by fossil-fuel developers eager to exploit coal, oil or gasfields in exchange for cash. The problem is compounded by new waves of the Covid-19 pandemic, and poor countries are seeing debts soar and the cost of borrowing rise.“

It is important that responding to such challenges are not seen as problems confined to the developed world. Many of the syndemic problems of racism, inequities and fractured civic ecologies are very much a reality in the ‘developed North.  Consider this Guardian article from February 2021:  

“For Black Americans, Covid-19 is another brutal reminder of the racist legacy of the American healthcare system. A disproportionate number of the 500,000 Americans who have died of coronavirus are Black. Yet African Americans and other people of color have struggled to access vaccines.  Racism corrupts every facet of the US healthcare system. Discriminatory practices barred the entry of Black candidates into medical, nursing, dental, pharmacy and other professional healthcare programs. Segregated medical facilities and unjust treatment within medical facilities continue to plague Black patients.“

As the evaluation community seeks to find and assert its voice in contributing to SDGs, it is important to take stock of naming the processes that contribute to amplifying the problems. In my experience, our focus tends to be the “intervention” or the “project”: we can get caught up in the project trap. We tend to focus less on the systems and syndemic processes that generate the conditions of poverty and inequities.  It is within the purview of evaluation and evaluative thinking to promote realism about the context and support structures and bring a clearer focus on systems and dynamic processes that will help achieve the vision of “no one left behind.”

As a field, we need to both recognize the formidable nature of the tasks and move toward action. As Ibram Kendi has brilliantly shown in How to be an Antiracist addressing the impacts of racism requires active commitment and action to reversing inequities. Lazily claiming to be a “non-racist” or “neutral” and “above the fray” or dismissing opportunities to push an agenda that takes inequities seriously with ‘this intervention has little to do with inequities’ is hardly enough. How do we move as a field beyond our passive focus on projects and interventions?  Should taking a more disruptive stance towards provoking clarity around the pathways, seriousness of addressing inequities, and realism about the complexity of the change process be part of how we approach our role and responsibility in helping achieve the SDGs?

Questions Galore

I briefly expand on these above issues by raising questions that I think are important as we locate our role in addressing the SDGs. My questions that seek to clarify roles are in four domains:

  1. Building capacities and sustainable impacts
  2. Multi-dimensional contexts and trajectories of outcomes
  3. Understanding the coherence and the disruptive powers of interventions
  4. What does it take to “leave no one behind”?
1) Building system-level capacities and sustainable impacts

It’s important to recognize that much of evaluators’ focus on impacts has not been matched with ensuring that these impacts are sustained in the longer run, after the funding of interventions has ended.  As example, Jindra Cekan has been a leader in calling out how few evaluations focus on issues of emerging and sustained impacts. Along with colleagues, I have argued elsewhere that we need to integrate planning for sustainability into our thinking of theories of change as we understand  how interventions/systems can bring about long-term change. The interface between interventions and whether these innovations are absorbed by systems capable of implementing the interventions in a sustainable way still needs deeper exploration. In a similar vein, there’s a need to pay attention to system level capacities including ‘adaptive’ and ‘absorptive’ capacities of systems.  As we focus on the interface of interventions and systems, some questions arise: What is a theory of change that focusses on sustainable impacts? How do we operationalize systems’/institutions’ abilities to absorb interventions? How can understanding mechanisms of sustainability occur at multiple levels?  How can concepts and methods from complexity sciences help inform our theorizing and measurement of sustainable impacts (see Zenda Ofir’s blog on viewing development as a complex adaptive system)?  The challenges of understanding development from a complexity lens goes beyond measurement: How can ideas of complexity sciences inform our view of what constitutes evidence?

2) Multi-dimensional contexts and trajectories of indicators

As an evaluation community we could be more focussed in understanding what are the multi-dimensional contexts that make changes (and reaching SDG targets) harder in some contexts than others. Much of the results frameworks that I see still have anticipations of success that are based on aspirations of ‘mechanical,’ ‘aspirations’ of results often based on linear extrapolations of past patterns. How can we incorporate contexts more deeply into our planning for what changes are realistic in different settings? We need a broader set of dialogues on what contexts matter and also how changes in context and planning for interventions in especially resource-deprived contexts need to inform our expectations of success. Further, we need better examples of how progressive evaluation approaches such as developmental evaluation, realist evaluation, and principles-focused evaluation help build a better understanding/intuition of what are realistic changes especially in very “sticky,” difficult to change conditions.

A focus on multidimensional contexts can also shed light on the anticipated kinetics of systems change.  I have found our approaches and tools remarkably limited for understanding anticipated timelines, trajectories and rates of change (see Michael Woolcock’s work for a refreshing introduction to the importance of these concepts).

3) Understanding the coherence and the disruptive powers of interventions

One of the significant aspects of the newly developed Development Assistance Criteria (DAC) is its focus on coherence as one of the evaluation criteria. How can evaluators help in mapping out the alignment between different levels of interventions to promote their coherence and disruptive powers? The mechanisms by which coherence matters and the contexts in which coherence matters need to be more clearly understood. Under what conditions should interventions be coherent/aligned with the rest of its local ecology/context? Alternatively, under what conditions can a lack of coherence add to the disruptive powers of an intervention in leading to favourable changes? Under what conditions should interventions not try to fit in with existing interventions, especially in contexts in which the contexts itself needs changes? Thinking about the coherence of multiple interventions forces us to explore design considerations.

4) What does it take to “leave no one behind”?

What could the evaluation community be doing to help operationalize strategies to population segments that correspond to intersectional categories of multiple disadvantages? Working in the international development/global health space, I continue to be surprised at the shallow nature of most theories of change focussed on inequities. The same focus that the evaluation community has spent on deciding what is a good, rigorous evaluation design we have not spent in answering what is a good enough theory of change. This problem is especially acute when we consider interventions that have the aspirations of addressing problems of inequities. What is a good enough theory — and by this, I simply mean a theory that can seriously aid translation/implementation — to address inequities?  Addressing these questions will help position our field to be more relevant in any dialogue that focuses on making a difference in the lives of individuals in need.

Looking Ahead

The SDGs provide fundamental challenges and amazing opportunities for the evaluation community to go beyond standard, stale ways of valuing. Addressing some of the above questions will have implications for how we think about evaluation competencies and how we approach building evaluation capacities. There is a need for the evaluation community to engage more deeply with both conceptual and evaluative issues involved with sustainability, inequity, contexts, disruption and coherence.

Addressing some of the above questions will require new partnerships, not just between evaluators of different stripes but between the evaluation and a number of other communities/sectors including ecologists, public health and nutrition, financial services for the poor, epidemiologists, etc. We might also need to consider how diverse evaluation approaches can be combined with other measurement opportunities (such as routine surveillance, etc.) to help build useful ecologies of evidence.

This is an exciting time to be an evaluator and an exciting time to rethink evaluation competencies and capacities.

What Gets Sustained When We Think About Sustainability?

May 24, 2021 / / 0 Comments

Naïvely I used to think that sustainability was just about a program finding another funding source when the previous funding was coming to an end. The only other examples that I recall about sustainability were stories about international development projects that, for example, built a water pump in a village in a far-away country and then after the project implementers left, the water pump eventually broke but no one in the community knew how to fix or maintain the pump — so the benefits from the project were not sustained. Such examples hinted at the notion that gains had been made, but they could not be sustained over time, partially due to poor planning.

I finally got a clearer, fuller picture of what sustainability is about while writing a paper in 2019,

‘Till Time (and Poor Planning) Do Us Part: Programs as Dynamic Systems Incorporating Planning of Sustainability into Theories of Change’

Canadian Journal of Evaluation, 2019

This blog is about some of the lessons we learned in thinking about sustainability and the roles for evaluation in helping better ensure that gains made by projects/initiatives can be sustained.

What basically is sustainability?

Sustainability can be thought of in three ways: 1) sustaining components of the intervention, 2) sustaining impacts after the program ends, and 3) mainstreaming or incorporating active ingredients of the intervention into other programs.

Program vs Client Point-of-Views

In my observation, evaluators, planners, program managers and staff tend to conceive of the impacts of a program very much from the viewpoint of the program – in terms of its activities and outputs that presumably lead to the intended outcomes. But if we consider a program’s impact from the frame of a client/ intended program recipient, most clients intersect only at points with a program while the rest of the time they are outside from the program living their lives. The “impact journey” typically encountered by a client rarely takes the form of a linear trajectory through one program but is one of a “rugged landscape” where many hills need to be climbed, circuitous pathways navigated, rivers crossed to reach the promised land of impacts. The program may only partially help progress the client along their impact journey. In planning programs, one needs to consider what else may be needed for clients to eventually reach the promised land, or at least what else is needed to sustain the gains achieved.

In evaluation terms, this translates to a change in focus from more direct and immediate program impacts to more sustained impacts that take into account the complex and heterogeneous needs of clients/program recipients.  

Risk Landscapes of Clients  

The metaphors of heterogeneous landscapes and multiple hills that are involved in clients’ journeys toward achieving goals/outcomes are useful reminders of the risk landscapes that remain for clients to contend with even after the intervention ends. In our experience, programs typically end without sufficient attention paid in advance to the provision of resources or continuity of care to clients, or to implementing strategies to better ensure that the outcomes achieved during the intervention can be maintained after the program ends. Knowledge of clients’ heterogeneous impact journeys and what context of supports, capacities, opportunities, and motivational incentives they might need after the program ends to at least retain the gains achieved and hopefully stay on the journey can inform strategies that envision what needs to be in place. For example, what needs to be in place might include partnerships with other agencies that have the capacity to carry on, or equipping participants with needed capabilities and resources to carry on themselves. In some scenarios, it might be preferable for a program to divert funds from serving a larger number of clients and taking them halfway across the river, to reallocating resources in order to take a fewer number of clients all the way across the river.

What changes when planning for sustainability?

When you plan for sustainability, your intervention’s theory of change and program activities will be different than if you plan only for outcomes within the intervention’s sphere of control and influence. For example, there have been multitudes of exercise programs studied in the literature that show good results of attendance, compliance, improved health outcomes, etc., but after the program ends the participants stop exercising, so maintenance of the healthy gains dissipates. Some of the reasons given by participants include: no gyms or green spaces in their neighbourhood, lack of transportation to get to gyms, other exercise class times are not convenient, not feeling safe to exercise alone, and loss of motivation. If we know all of this, then a typical exercise program should be changed from simply leading participants through the best set of exercises to also teaching participants how to exercise on their own, helping them to understand what affects motivation, structuring exercise sessions so that endorphins (natural feel-good chemicals in the brain) kick in, reinforcing how good exercise feels so that participants start associating exercising with feeling good and will be more driven to continue. The intervention changes based on considerations of what clients will typically encounter in their journey beyond the intervention and what it would take to sustain them in that journey.

Planning for sustainability should be an integral part of what we consider to be a useful theory of change. Incorporating planning for sustainability can change the nature of the program itself (Sridharan & Gillespie, 2004).

Finally, here are tips and considerations for developing a theory of change to incorporate sustainability planning:

  • Make explicit the uncertainties in the theory of change
  • Map the connections between the intervention and the rest of the organizational system
  • Explore the heterogeneities of client-level needs
  • Explore key barriers that individuals will face in the intended impact journey
  • Explore the potential timelines of impact
  • Explore the dynamic context of supports needed both during and post-funding stages to assist with capabilities, opportunities, and motivational incentives
  • Explore the role of boundary partners

Insights from webinar four “Leaving no one behind”

April 9, 2021 / / 0 Comments

TECCHI’S final webinar in the Sustainable Development Goals (SDGs) series was focused on leaving nobody behind. “Leaving no one behind” a phrase commonly used in the global health and international development sectors was unpacked during the webinar, with a focus on how it relates to the field of evaluation. The topic was examined from multiple points of view with some speakers focused on the importance of understanding local contexts and power dynamics while trying to ensure that nobody gets left behind. Another speaker approached the topic from a different lens, by describing how the burden to sustain interventions is often placed on countries that are impoverished and likely lacking in capacity to sustain program outcomes. She debated whether or not “leaving nobody behind” is a reasonable goal to have given the political and economic structures that currently govern our world. This blog post will focus on the dialogue taking place between the webinar participants who engaged in discussions through Zoom’s chat feature. Below is a list of some comments that stood out to me from the conversation:

“The point being raised about trying to understand different perceptions is equally valid in developed economies eg in evaluating programmes for sex workers or drug dependents and possible non uptake of services provided – equally need to search for their understandings and reasons of why there is no take up”  
“Why should be burden the community?  can [we] have a model where the state is held accountable for services that are sensitive to social and cultural realities.”  
“Prettiness of no one left behind isn’t matched”
“Finally, we don’t fund, design, implement, monitor or evaluate for sustainability or consider longer-trajectories post-project. Those are seen as the problems of the countries themselves…. who are poor enough to qualify for aid and often powerless and capital poor who can’t sustain multi-million dollar outcomes…  Just a warning,  ‘complexity’ can be hidden behind – it’s too complex to address, no?”
“I do think the methodologies and methods exist, but are not applied i.a. because of the political economy of evaluation, and of development.”

The first comment talks about the need to understand contexts and power dynamics from the participant’s perspective. While there is a growing realization of the importance of contextual considerations while developing, implementing and evaluating programs, it is important to be reflexive about our position and humble about our contextual knowledge (and lack of it). When working in the international development field there is a lot one might not know with regards to the context they’re working in, and it’s critical to seek out knowledge from program participants and other local actors to understand their perspectives about a given program and how it fits into the fabric of the local community. This comment also reminded me about a striking observation that Dr. Das made during the webinar. He hinted at how the field of development often attempts to fit a poor person into the western “modern” perspective. In his presentation, he talked about the promotion of institutional delivery in Odisha. While the program being implemented defined success as an increase in the number of women giving birth in an institutional facility, it didn’t take into account the fact that many of these same women didn’t even have proper houses to live in.

While the second comment is intriguing, I have to kindly disagree with this statement (“Why should be burden the community?  can [we] have a model where the state is held accountable for services that are sensitive to social and cultural realities”). While it is important for states to have policies and frameworks that take into account social and cultural realities, it is also important for individuals at the community level to play a role in developing programs that are sensitive to their needs. Subsequently, actors at the local level also need to be held accountable to the promises that they make. Operating at the community level enables an organization to build intimate connections with those whose needs they’re serving, and consequently get a pulse on what reality is like from the program participant lens. It enables one to occupy a unique position where you do have the opportunity to listen with humility and lead change that is sensitive to the local context.

Another comment that arose from the discussion is the need to acknowledge the systematic confines that program implementers and evaluators are working within. This includes the fact that funding cycles are generally short-term and often narrowly focused on impacts from the funders’ perspective. Given these circumstances, there might be a failure to acknowledge the work done in obscurity before empirically measurable impacts can be observed. In some cases, this fixation on achieving impacts can hinder experimentation, as the room for failure is very small, and taking risks through experimentation could lead to outcomes that are less than optimal.

While “leaving nobody behind” is extremely important from a human rights perspective, we are aware that our reality is a stark contrast, with inequities growing deeper and the poor being stuck in a vicious cycle that is hard to break through. Given that our words don’t align with our reality, and our inequities are deeply embedded in economic and political systems, how do we work towards reconciling this difference?

Summary of Presentations | No One Left Behind

April 8, 2021 / / 0 Comments

Abhijit Das

  • Why is it a big deal to be discussing the goal of “Leaving No One Behind”?
  • Historically, most goals have achieved only modest achievements and are unable to show that the people who needed them most have actually been helped
  • Slow progress and modest gains are not enough – We need to ask:
    • Who gains more?  Who gains less?  Who doesn’t gain?  Are there any losers?
  • Development changes relationships – disrupts power structures
    • We need to pay attention to these social power structures in these extremely hierarchical power structures
  • Heterogeneities – will the same range of services work for all?

Understanding Lived Realities

  • Understanding people and communities – What is their self-concept or self-determination? What are their aspirations?
  • State-citizen relationships – What kinds of legal protections are available? What kinds of accountability mechanisms exist?
  • Is marginalization (economic/geographic) the same as social exclusion (stigma/discrimination)?
  • How much diversity of lived health-related experiences have we really understood?
  • Do we understand community concerns and consequences?

Consequences of Poor Services

  • Who pays the costs?  Who is accountable?

The Ethical Conundrum: Who do evaluators serve?

  • Donors, international agencies, governments, or those who often get “left behind”?

Zenda Ofir

Four Points: Beyond the Rhetoric?

  1. Two Sides to the Ethical Imperative of “Leave No One Behind”:
  2. Sometimes certain groups must be left behind – at least for a while. 
    1. It has become a comfortable rhetoric to leave no one behind – has become meaningless
    1. “It’s not about your project! It’s about my country! (Oumoul Ba Tal, Mauritania, 2007)
    1. It may be unethical to expect countries to ensure no one is left behind – far more complex than we may think
    1. Burden of sustaining “no one left behind” efforts inevitably falls on the poorest countries in the Global South – who can’t afford it
  • Development and Evaluation Specialists buy into questionable narratives and assumptions:
  • Examples of questionable narratives about development:
    • Development from the start has to be based on “inclusive growth” and NOLB. Development trajectories of countries and their stage of development do not matter.
    • Funding and effort have to be concentrated on extreme poor countries/areas/communities/groups
    • Progress can be measured – in snapshots – by the performance of the weakest, without considering the sustainability of the progress
  • There is a Particularly Dark Side to Aid-driven ‘Development’ and Evaluation – What Counts as ‘Success’?
  • In the aid world, there is little consideration – in a ‘country’/’society’ context – of:
    • The long-term
    • Development trajectories
    • Poverty traps
    • Skillful policy and practice adaptation to changing conditions
    • What complex systems concepts tell us about how the (social) world works
  • In Both Design & Evaluation, We Need to Pay More Attention to Nuance, to Patterns in Systems, and to the Application of Complex Systems Concepts – especially:
    • Systematic, directed, sequenced experimentation
    • Co-evolution
    • Cascading ripple effects
    • Trade-offs
  • Impressive application of complex systems concepts
    • “How China Escaped the Poverty Trap”, Yuen Yuen Ang

Orielle Solar

  • What challenges for equity need to be addressed to ensure the Sustainable Development Goal of “No One Left Behind”?

Weakness to Address Inequity Problem:

  • Restrictive vision of health and wellbeing
    • Absence of analysis and action to address social determinants of health
    • Limited consideration of socioeconomic-political context
    • Lack of recognition of heterogeneity of sub-populations

Equity Solutions:

  • Social basis of health
    • Social determinants addressed
    • Application of life-course approach
    • Prioritization with equity approach
    • Proportional universalism
    • Multi-sector action
    • Community engagement

Key Challenges for Processes of Regional Adaptation:

  • Enhancing guidance for action on health inequities
    • Integrating multi-sectoral action and social participation to address inequities
    • Supporting capacity-building/strengthening and exchanges
    • Strengthening the equity lens in M&E

Recommendations – Challenge of Enhancing guidance for action on health inequities

  • Strengthening orientations for action on equity
    • Social determinants framework – underlying mechanisms that must be addressed
    • Incorporate equity challenges specific to the region – ethnicity, migration, violence, etc.
    • Strengthen the gender perspective
    • Focus on structural social determinants using the life-course perspective
  • Establishing criteria to set equity-based priorities
    • Where to start (which groups, settings, problems, interventions)?
  • Intersectoral action and social participation – cornerstones to achieving greater equity
  • Participation as an “end” not just a “means”

Recommendations – Challenge of Strengthening the Equity Lens in M&E

  • Measuring progress in indicators of equity and evaluating impact is fundamental
    • Promote and support evaluation thinking in national plans beyond the indicator and monitoring framework
    • Follow a set of indicators that account for inequalities between social groups in terms of outcomes, relevant social determinants, and key implementation processes
    • Stratify indicators as an essential requirement, not an option 

Recommendations – Challenge of Supporting capacity-building/strengthening and exchanges

  • Build capacities of professionals, academics and civil society actors to understand and act upon the social determinants to address equity:
    • Promote forums, courses, and activities related to equity-oriented design, implementation, and M&E of interventions
    • Develop a platform and repository to share and update good practice, learning, and evidence from regional and national experiences
    • Promote a research agenda on policy/intervention evidence

Mel Mark

Many Meanings of “Leave No One Behind”:

  • Absolute goals (e.g., “End poverty”, “End hunger”)
  • Reduce overall disparities (e.g., “reduce inequalities within and between countries”)
  • Reduce/eliminate group-based disparities (e.g., “achieve gender equality and empower all women and girls”)
  • Prioritization of the least well off

Seeming Trends in SDG Evaluation regarding “Leave No One Behind”

  • Disaggregation, attention to distributional/subgroup effects
  • Participatory, inclusive approaches
  • Maybe a move towards evaluation policies

Lots of Action on the Statistical Front

  • UN Economic Commission for Europe, 5th expert meeting on statistics for SDGs
  • Some of it related to LNOB
  • Session 2 – Impact of COVID-19 on SDGs – “Leave No One Behind” Perspective

Tentative Suggestions from 30,000 Feet

  • On the Ecology of Evaluation – Ecological Niche of Evaluation Approaches
    • The ecological niche consists of the role an evaluation approach plays in its environment, including: where it is conducted, what resources it takes, what it does and what its intended and actual consequences are, how it interacts with other evaluation approaches, policy processes, those in its environment
    • Alternative Roles for Different Evaluation Approaches – no single approach sufficient
    • A varied and inclusive ecology of evaluation
      • Can we effectively advocate for multiple evaluation approaches, each used in potentially valuable ways?
  • The Use and Influence of Evaluation
    • The steps from and evaluation to its ultimate consequences – involve multiple levels, processes, often emergent, issues of power, motivation, relational issues, etc.
    • Pathways of evaluation influence – planned and emergent sequences of how evaluation gets from here to there
      • Will we strengthen attention to such pathways and share learning?
      • Motivation, political will may be critical – what is the possible role of evaluation in increasing will?
  • In the Face of Spectacular Ambitions
    • What are the ways that we can supercharge learning (including replication, broadly speaking), dissemination, cultural/contextual adaptation?
    • Don’t let 2030 be an enemy of 2040, 2045, … – What’s the long-term perspective?
    • How do we hold high aspirations regarding evaluation and SDGs, while retaining appropriate modesty?

Insights from webinar three “Measuring Coherence of Complex Interventions”

March 26, 2021 / / 0 Comments

TECCHI’s third webinar in the Sustainable Development Goals (SDGs) series was centered on the role that evaluations could play in measuring the coherence of interventions. Coherence can be defined as “how well an intervention fits” within a given setting. As a recent graduate, I’ve been surrounded by many classmates who are interested in the notions of disruption, innovation, and bringing new ideas to the solution space. However, too often our courses in university are structured so that students with similar academic backgrounds are in the same classes. Many times the lack of cross-faculty spaces for collaboration have come in the way of engaging with ideas using a lens of “coherence”. How do ideas generated in the field of health interact with those in political science, business and the environment? How do these different fields work together to solve problems in the real-world? The webinar on coherence helped me think more deeply about how ideas and interventions fit within a given context. Below are a few quotes that were mentioned in the chat during the webinar that stuck out to me:

“The relevance of formal vs, informal institutions and their overlay may need more attention.  Linking back to [the] interest in how this overlay will be relevant in coherence in a systems’ perspective as well.”
“I’d like to hear more about the coherence of coherence. Coherence with what, in what contexts? I’ve long identified limitations of silo thinking and the myth that any single intervention can have meaningful impacts, in isolation”
“I would also like to hear about the linkage of coherence with sustainability. Also in my opinion, measurement could be key specifically in situation wherein there are different partners and programmes work in tandem”
“’coherence’ isn’t always a ‘good’. […] Its a bit about the argument about institutionalisation as always good!? It depends on coherence for what purpose which I think is what you are implying”
“Decision processes are surely key interfacing formal and informal institutions (or culture)”
“In some humanitarian contexts, “coherence” with national policies and strategies are sometimes not desirable.”
“I would like to support [a] comment on the difference between policy design and evaluation and claiming the need for more inclusion of evaluators in policy design. Not only formative evaluation but the promotion of old fashion ex-ante evaluation which we don’t seem to see much nowadays. Also this relates to how do we cumulate knowledge in program evaluation as discipline… and the needs for evaluators to be less reactive and more proactive (not to that easy of course!)”
“I think it’s interesting that conversations about implementation and evaluation often happen separately – I am not an evaluator but an implementer and we talk about coherence in different language and in different ways. I think involving evaluators in implementation/program design and vice versa would be beneficial to address a lot of this issues with coherence”
“For me, an important part of coherence is the idea of resonance — the idea of adding to what already happening and gaining (non linear) momentum in the process. This links us back to the idea of contributing to trajectories we talked about a couple of weeks ago – trajectories consisting of systems of actors interacting with each other, technology, institutions, etc that deliver coherent sets of outcomes over time. The outcomes can be good or bad, and trajectories may work to resist change.”
“that’s why I was arguing that the choice of unit of analysis is key – I wold [sic] argue it’s the cluster of interacting polices we should be looking at!”
“Isn’t the point that we need to choose what to be coherent with, and being more explicit and transparent about that choice?”

Coherence with what?

When thinking about coherence, many participants in the webinar noted the importance of stating up-front what policies, programs, and values a new program aims to be coherent with. In any given context, there are formal institutions and informal actors present, these different groups might have values that are similar, but also those that are divergent. By stating which aspects of a given context a new program aims to align with, program implementers could move towards building connections with different partners (such as community organizations or policy actors) that are working on similar projects or program areas. These new relationships can then be leveraged to work towards shared goals and ambitions around solving a problem.

Fred Carden, the first speaker in the webinar, discussed different types of coherence – internal, intra-organizational, inter-organizational and international coherence. The idea of coherence can be thought about on many different levels, and perhaps as one aims to increase the scope of organizations and national priorities one wants their program to be coherent with, the difficulty of the task increases. With an increased number of stakeholders, organizations and policies to consider, a larger diversity in views and values are present at the table. This in turn increases the difficulty of designing and implementing a program that is coherent with the views of a broad range of factors.

Is coherence always desirable?

During the webinar, some participants posed questions that sought to understand when coherence is desirable, and when it is not. Are there certain aspects of a program that we hope are coherent with a given context, and other aspects that are aiming to bring a more disruptive energy to a solution space? When there is disruptive change, are appropriate change management tasks undertaken so that individuals are able to understand the change that is being brought, and how their role is shifting as a result? How can disruption and the notion of coherence be reconciled? Although coherence might not be desired in all aspects of a program, it is important to understand how the program fits in the larger socio-cultural, historical, environmental, political and economic contexts. How does the new program build on progress that is already made, and what new inputs does the program bring in? Are the users of the program ready to engage with the new change that is being proposed? Were the users part of the process while developing the new program?

Measuring Coherence

Measuring coherence is also another theme that was brought up multiple times during our discussion. Given that in any given context there are often multiple interventions taking place simultaneously, and that different programs and services synergistically build on each other, how do we measure if a program is coherent in a given context? How do you measure contributions when there are many partners involved? What about the unseen partners who might be playing a key role? Another question that comes to mind is the role of evaluation in looking at problems as a whole, and understanding how multiple programs funded by different organizations are working in tandem towards solutions. Is there value in funding evaluations that aim to understand how an ecology of policies, programs, and stakeholders at multiple levels work together towards solving an issue that is of importance to them?

Summary of Presentations | Measuring Coherence of Complex Interventions

March 25, 2021 / / 0 Comments

Fred Carden

Coherence: A Principles-Focused Approach

  • What is Coherence?
    • OECD 1991 – initial focus on policy coherence
    • European Commission 1992 – put into law
    • Focus has been on global issues
    • Enshrined in SDGs
    • No accepted definition
    • OECD defines coherence as: an approach and policy tool to systematically integrate the economic, social and environmental dimensions of development at all stages of domestic and international policy analysis.
  • What is Coherence in a Country Program?
    • Development, Trade, Diplomacy – how are they coherent?
    • There are different levels of coherence: 1) Internal Coherence; 2) Intra-Organizational Coherence; 3) Inter-Organizational Coherence; 4) International Coherence
    • Focus of Global Affairs Canada in Columbia case was on first two levels of coherence
    • We first looked at the notion that coherence is not only about policy – what does coherence mean in the context of programming of global affairs in Columbia?
  •  Levels of Coherence
    • Policy Coherence
    • Organizational Coherence
    • Administrative Coherence
    • Knowledge Coherence
    • Informal Coherence
  • Measuring Coherence
    • Developed four-level rubric – no coherence to coherence
    • Breaking it up into different elements very helpful in discerning what was/was not coherent – Example: coherence across international assistance; coherence across international engagement
    • Found that there was pretty good policy coherence, but administratively weak

Lori Bell

  • In the context of UNHCR commissioned evaluations, coherence is always looked at through the lens of relevance – are we doing the right thing?
  • Presentation provides concrete examples of evaluations where coherence was looked at from an internal and external point of view

System-wide evaluation of cyclone Idai Mozambique 2019

External Coherence

  • Complexity, crowdedness of the humanitarian space
  • The divide and conquer of different agencies
  • How did we live up to our responsibilities and roles compared to what other actors did?
  • If you want to look at External Coherence, you can’t do it on your own – we need to understand the comparative advantage of UNICEF, etc.
  • We need to know a lot about other agencies operations and what the government is doing
  • Inter-agency joint evaluation, rather than a single agency evaluation

The Importance of Personalities, Perceptions, and Culture

  • Coherence boils down to individual behaviours and decision-making on the ground
  • We need to look at culture and perceptions

Elliot Stern

Aspects of Policy Coherence

  • ‘Policy Coherence’ in EU and OECD
    • Coherence now a “new” (2020) OECD criteria, but entered European policy vocabulary in 1992 Maastricht Treaty
    • Distinction between internal and external coherence – i.e., consistency of elements making up a policy action vs consistency between different policy actions (this distinction may be less clear-cut than 10 or 20 years ago)
  • 4 Aspects of Contemporary Policy Coherence
    • Unit of analysis
      • Growing understanding of complex and interdependent policy priorities (e.g., innovation, climate change mitigation, equity and justice)
      • Extending the Unit of Analysis: i) decontextualized policy interventions (coherent with policy inputs); ii) contextualized policy interventions (coherent with policy/non-policy inputs/contexts); iii) interdependent policies interventions (coherent between multiple policies in multiple contexts); iv) policy systems (many coherence possibilities)
      • Systems Mapping for complex policy evaluation
    • Coordination and governance – Coherence doesn’t happen on its own!
      • Multi-part and separate but interdependent policy interventions must be aligned – importance of coordination and governance arrangements
      • Success often depends on pre-existing community, sectoral/territorial networks; credibility of system leaders; capacities to use information and adapt; participatory engagement of stakeholders
      • Coordination and governance itself must be evaluated – part of ToCs, etc.
      • Encouraging those in governance and coordination roles to reflexively learn and generate and use M&E data a way of strengthening coherence
    • Timing: Re-setting the evaluation clock
      • Move away from short-run, ex-post evaluation at end of program/policy cycle to integrate evaluation into program planning and “up-valuing” mid-term/ongoing evaluations
      • Ex-post evaluations are moving long-term – look back over longer time scales on rolling basis
      • Mid-term evaluations increasingly inform reprogramming and “steering” and adaptive management
    • Importance of substantive theory
      • When we move from narrow focus on programs to recognize importance of complexity, context and interdependencies, it changes where we look for theory
      • Program theory is not enough
      • Becoming standard in UK and EU structural funds to look to academic research to identify building blocks of Theories of Change (e.g., Innovations Systems Theory)
      • Centrality of literature reviews and partnerships with domain experts

Vijayalakshmi Vadivelu

UNDP’s programming:

  • In past 3 decades, programs have looked at coherence in different program areas – support to livelihoods, peace building, etc. require synergies – intersecting elements to enhance coherence at the country level
  • UNDP’s consistent emphasis on coherence and partnerships – implemented with a range of partnerships – integrated role within UN system
  • Programming at national and subnational levels enables synergies/linkages

Coherence a key dimension of program performance

  • Assessed at multiple levels: Internal, External, Intra-organizational
  • Mapping of areas where maximum synergies are possible 
  • Also look at intersecting areas in response to crises

External Coherence      

  • UNDP collaborates with other actors to contribute to humanitarian response
  • Determine pathways that link, map key actors working in the area, identify intersecting process indicators and strength of partnerships
  • In bringing together different actors, coherence assessed using sector data
    • Increasing household income, increased investment = indicators for policy coherence

Different types of Coherence – horizontal and vertical pathways

  • We try to determine the level of coherence
  • Questions regarding linkages to sustainability
  • Use causal pathways through the Theory of Change – coherence is one we try to examine
  • Multiple methods – determining coherence is more challenging, especially subjectivity in identifying some of the indicators, even where synergies are happening
  • Can be helpful to disaggregate and then develop a composite score
  • For each level, we use a different rubric for mapping
  • Determining national sector policy coherence is often much simpler, but determining the contribution of an agency/program can be extremely challenging

Insights from webinar two “Contexts, Trajectories, and Assessing Progress Towards Goals”

March 12, 2021 / / 0 Comments

TECCHI’s webinar, “Contexts, Trajectories, and Assessing Progress Towards Goals”, gave rise to a lot of discussion on the topic of timelines and trajectories for understanding programs and their impacts. Below is a table of some insightful comments from different webinar participants as they engaged with the ideas being presented.

“ex-post we find things that ‘fail’ at final, sometimes succeed later, tho’ most of the time those that succeed (and we ASSUME sustained impacts) ‘fail’ a couple of years later.”
“People seem to think that funders and program designers are foolish to look for simple short term outcomes. I think a case can be made that such behavior is rational and the most effective tactic for addressing important issues.
“the short-term is, in my view, irrelevant to our participants and partners who need truly sustainable development that’s locally driven/ sustainable (listen to Time to listen by CDA/Mary Anderson’s crew)
“I agree the short-term view is problematic. I’m only saying that there are good reasons for it. If we don’t appreciate those reasons, progress will never happen.
“Surely the problem Jon Doe, is the preoccupation by traditional evaluators on projects/programmes because that’s who pays us. If you focus on policy and social change then the whole logic if short term impacts becomes absurd!
“Frankly I think we need Theories of Sustainability designed by those tasked with sustaining the outcomes, not just donors ToC during fixed lifetimes.
“I have been thinking about this. It is possible to know some relevant aspects of context in advance, but impossible to know how context affects outcomes except retrospectively.
“measuring multi-dimensional facets of contexts and trajectories should not force to add and burden the questionnaire/respondents…
“context, people and system matter! great said! How do we ensure the linkages to craft our evaluation more practical in day to day application? The human centered evaluation could lead us further towards reality of contexts?
“I always ask the evaluators in training to apply the concepts that we are learning to a program that the learners are working on or passionate about. Situational and Interpersonal Competencies Domains are so important to evaluators. There is more than the technical, reflective and management competencies domains that evaluators need.

Some of these comments shed light on the tensions that are present when thinking about timelines for impact. What are outcomes that we can expect to see in the short term? What is the nature of these outcomes, and how are they different from the long-term impacts that we hope to see in the future? How do short-term outcomes and long-term sustained impacts cohabitate in the same space of program planning?

When establishing timelines for impact, it is important to think about the assumptions that we are making for an impact to be achieved within a given timeline. It’s also perhaps important to reflect on how we go about creating these timelines in which we hope to observe positive change. Sandra Albert, one of the presenters in the webinar, hinted at how interventions are often planned around naive assumptions about the cultural practices in a given context. Given that contextual realities are often like murky water and there are multiple intersecting factors that play a role in the success or failure of interventions, how do we establish “reasonable” timelines for programs? Michael Woolcock, one of the presenters, spoke about the non-linear nature of trajectories and how evaluating an intervention at different points in time could lead to very different understandings on whether a program is successful or not. He also spoke about doing activities in obscurity before change can be observed. These actions often go unnoticed. Some of the webinar participants also mentioned how funding cycles are oftentimes built around achieving short-term wins. What is the value of these short-term gains? Perhaps, achieving these short-term wins that are less complex in nature could help create momentum and generate excitement about a project, and also bring a sense of hope that change is possible. However, it is important to balance this view with a lens of sustainability and helping communities develop the capacity to sustain impacts even once the official funding runs out. Given that observing change takes a lot of time, how do we go about creating theories of sustainability from the onset of a project?

Moving from sustainable impacts and outcome trajectories, to learning systems, I thought that the comments from the webinar participants on “human centered evaluation” and not “burdening” participants with completing questionnaires, quite intriguing. How do we make the process of giving feedback something that participants want to engage with? In order to feel motivated and empowered to provide feedback, and engage in the process of learning, program participants should feel a sense of ownership of the program, and should feel like the input that they are providing will get fed back into the project, ultimately making their experience interacting with the program more pleasurable. As we engage with these questions, it might be beneficial for program implementers and evaluators to collaborate with individuals across other fields such as design, urban planning, psychology etc. to understand how they engage with processes of continuous learning around the interactions of humans with their environment.

Summary of Presentations | Contexts, Trajectories, and Assessing Progress Towards Goals

March 11, 2021 / / 0 Comments

Michael Woolcock

Understanding Trajectories

  • Change is rarely linear and can take a long time
  • Example – Pace of Change for 5 different social events/processes over last 250 years:
    • In case of interracial marriage, we see that change took forever (nearly 200 years). But even in examples like same-sex marriage that seem to have had a rapid uptake in approval ratings and legality, if we look at the history, we see that people were campaigning for decades before any change really happened.
  • Centuries worth of change still unfolding for certain issues (e.g., gender equality); others occur at a much faster rate.
  • We need to incorporate these realities much more routinely into how we think about development interventions

Why Trajectories Matter

  • We must Take Time Seriously – it is a crucial component to how we understand change
  • Example of punctuated equilibrium (e.g., same-sex marriage).  Yet other interventions where the situation gets worse before it gets better (e.g., work on democracy reform, human rights)
  • Also “Shooting Star” projects – example where pilot studies yield wonderful results, but when scaled up achieved much more modest gains.
  • Baseline and follow-up not enough – need a much more measured sense of when and how things are being achieved
  • We can’t expect the full plate of development tasks as rendered in the SDGs to be achieved within 10 years or even 20 years. The presumption that all of this can be achieved by 2030 is entirely sensible for some goals, but wildly unrealistic for others.

Rethinking Theories of Change

  • Assessing impact claims against Counterfactuals and ‘Counter-temporals’
    • Focusing on Mechanisms & Processes – how impacts are achieved for different people in different places over different time periods
    • When and how things are happening for whom

Sandra Albert

  • Context: North East Region (NER) of India, examples coming from Meghalaya
    • > 3.5 million people; 86% Indigenous/Scheduled Tribes
    • Matrilineal society – children given maternal clan names, women inherit ancestral property, birth of daughters is celebrated (unlike much of rest of India)
  • Examples:
    • Pilot workshop on sexuality and reproductive health (SRH) for undergrads in new university in Meghalaya, assuming that SRH training would likely not be needed in such an evolved place. But responses to survey questions revealed otherwise – blaming the victim a prominent theme in responses.
    • SALT (Stimulate, Appreciate, Learn & Transfer) approach to improving immunization uptake in Assam.  Analysis found no effect on Full Immunization Coverage in 12- to 23-month-old children, but NGOs implementing insist a tangible difference is being observed. Is “tyranny of time” contributing to poor outcomes and measurement?
    • Behaviour change can also happen quite quickly – Example of practices related to hand hygiene, respiratory etiquette, and quarantine in Meghalaya pre- and post-COVID-19.
    • High unmet need for contraception – example of vasectomy figures in Meghalaya (0.1%) and reasons for men’s aversion to vasectomy (e.g., fear it would make them “fat and lazy” – associated with local practice of castrating pigs)
  • The Meghalaya Paradox:
    • Matrilineal Society (like Kerala)
    • But poor maternal and child health indicators persist
  • Context in the NER of India needs to be considered

Boru Douthwaite

“I Wouldn’t Start from Here: Making the Case for Outcome Trajectory Evaluation (OTE)”

  • The program Theory of Change is the wrong place to start!
    • Focuses too much on the program, assumes program has more causal power than it actually does
    • Doesn’t adequately take into account broader context in which the program operates, which also has causal power
  • Better place to start is the broader context – specifically the outcome trajectories that have produced program outcomes
    • Program outcomes of interest = arrangements of interlinking and self-reinforcing changes that emerge and evolve over time
    • Program outcomes emerge from outcome trajectories – interacting and co-evolving systems of actors, knowledge, technology, and institutions
  • Example: Cassava seed certification arrangement to control two viral diseases in Tanzania
  • How do we start from a better place?   
  • Outcome Trajectory Evaluation – Steps:
    • Step 1: Agree on program outcomes to evaluate
    • Step 2: Select published middle-range theory that best fits the types of change under scrutiny
    • Step 3: Use middle-range theory to help identify and describe the outcome trajectories
    • Step 4: Use middle-range theory as basis of case-specific theories of change, one for each outcome trajectory
    • Step 5: Validate outcome trajectory timelines & case-specific ToC with key stakeholders
    • Step 6: Use case-specific ToC and validated timelines to answer the evaluation questions
    • Step 7: Review of final evaluation report before publication
  • Comparison of OTE with Standard Program Evaluation (SPE):
    • Starting Point
      • OTE: program outcomes and the outcome trajectories from which they emerge
      • SPE: theory of change/results framework in program document
    • Use of Theory
      • OTE: use published theory as basis for ToC for each outcome trajectory
      • SPE: test whether ToC specified in Program Document was valid
    • Establishing Causation
      • OTE: by investigating whether: (i) the outcomes exist; (ii) they were generated by the respective trajectories; (iii) the program contributed to the trajectory
      • SPE: by investigating whether program activities contributed to outcomes without really knowing about the synergistic contribution of others. 
    • Learning from one evaluation to the next
      • OTE: possible because middle-range theory provides a common framework to do so
      • SPE: harder without a common framework

Shubham Gupta

“Context(s) Matter – Learning from Collaborative Evaluation in India”

  • Why do we need to take context seriously?
    • Human Component – Development programs are not mechanical
    • Micro vs Macro lens to program development and evaluation
    • Realistic expectation of program success
  • Outcome Trajectory – Taking the Contact Lens
    • Fascination with averages – leads us to think in a very linear way and miss out on significant small wins
    • Timelines of impact – a dominant context
      • Example: mapping outcome of a maternal health intervention
    • Making course correction based on contextual outcomes
  • What can be counted as context?
    • Often, we think of basic social demographics, some kind of political/policy priority, community practices, belief & attitudes
    • Non-traditional ways of understanding context need to be incorporated into the program thinking and evaluation thinking
      • Examples: organizational culture is a very important contextual factor that can drive change, leadership is another major contextual factor, crisis/ecological challenges
  • Curious case of infant mortality reduction: UP vs Kerala
    • 2018 snapshot – UP appears to be doing much worse than Kerala and worse than national average
    • When we look at the complete trajectory (1971-2018), the interpretation changes dramatically
    • Adding deeper understanding of context (not just time), and the interpretation changes even further (e.g., infant mortality rate across divisions, multiple kinds of disparities within districts)
  • Incorporating contexts in the program’s Theory of Change
  • Looking from below – collaborative learning with implementation teams
  • Exit strategy and sustainability – based on contextual success

Post Webinar Reflections on Sustainability Evaluation

March 4, 2021 / / 0 Comments

What happens once a program comes to an end? How do program implementers help build a system’s capacity to sustain impacts? How do we measure sustained impacts and how do metrics change over time to respond to the evolving needs of communities? These were some of the questions that were raised by participants during TECCHI’s webinar on Sustainability Evaluation. While the answers to these questions are complex and very dependent on the context of the program, this blogpost aims to reflect on some of the rich discussions that took place during the webinar that might offer some insights to these questions. However, this post might leave you having more questions than answers. Below are a few comments from the webinar participants that stood out to me:

“I just evaluated a complex health intervention here in [Canada] with absolutely stunning results, but everyone told us ‘Once you leave we won’t be able to continue this’”
“If we are such great systems thinkers, why not ponder the question of what the systems reasons for not doing follow-up evaluations?”
“In my work at [an international development organization] I was in the team screening proposals for funding, element of sustainability was weakest. When I visited communities after few months the funding ended, was frustrated to find most of the activities had stopped. When I asked the people, they said it was “NGO project” not ours.”
“Funding timelines and expectations are often sustainability killers even though they are project starters”
“Building ownership or behaviour change takes a long time because its about unlearning . For behaviour change in communities, the outsiders have to change. We have to shift as ‘experts’ with solutions to facilitators.”  
“In my work with nonprofits, they don’t really start coming up with concrete sustainability strategies until they’ve done a sort of systems map of all the stakeholders they never thought about who stand to care about/benefit from the work they do, and id new partners as a result”
“Perhaps greater intentionality around exploring the mechanisms that support the achievement of outcomes; and consider how to sustain those mechanisms.”  
“You know, we don’t kick our kids out of the house at 14, we slowly make sure they can handle money, handle a job, meet their school deadlines, etc. Maybe we should be planning exit strategies more intentionally in all programs, and in that period we would learn more about potential barriers to sustainability, what’s coming up that needs to be dealt with”

Community Ownership and Sustainable Impacts

The idea of sustainability can be thought about in many ways including sustaining components of the intervention; sustaining impacts after the program ends; and incorporating key ingredients of the intervention into other programs and services that exist within a community. Many participants in the webinar were interested in the idea of building capacity at the community level and facilitating an environment in which communities take ownership of programs, with the goal of sustaining impacts once the funding runs out. One of the participants mentioned that in her work “the team screening proposals for funding, element of sustainability was weakest. When I visited communities after few months the funding ended, was frustrated to find most of the activities had stopped. When I asked the people, they said it was “NGO project” not ours.” It is important to design programs that address a need that the community themselves recognize as something that matters to them. When there is a need that the community themselves recognizes as something that they want to address, they would perhaps be more likely to want to be a part of the design process, and as a result might not view the program as the “funder’s project”, but rather as their own project. The webinar participants also addressed the importance of unlearning our roles in the international development space, “Building ownership or behaviour change takes a long time because it’s about unlearning . For behaviour change in communities, the outsiders have to change. We have to shift as ‘experts’ with solutions to facilitators.” This shift in thinking of funders as facilitators of change rather than leaders of change shifts the dynamics of how programs are typically funded and run, and the power dynamics that are at play between a funder and the community. Often we think about co-designing projects with communities, but what if the area of interest of a funder doesn’t align with the community’s own interests? Who really gets to lead change?

A Timeline for Sustainability and Follow-up Evaluations

Another concept that was brought to light in our discussions was the idea of having a timeline for sustainability. One participant mentioned that often “there’s no sustainability ‘for how long’ plan”, and another noted that there is perhaps a need for “greater intentionality around exploring the mechanisms that support the achievement of outcomes; and consider how to sustain those mechanisms”. When thinking about sustainability, we need to identify what it is that we are we trying to sustain; the programs themselves or the impacts of the programs? What if the mechanisms change, do you still attribute the impact to a particular program? How do we account for the changing needs in a context? How is sustainability being measured? If certain components of a program that were considered to be “critical” still continue in a given context after the funding has run out, do we claim that the program was sustained?

As an example, a few years ago, TECCHI conducted an evaluation for a women’s entrepreneurship program that was funded by Ontario’s Poverty Reduction Strategy. The goal of the program was to help women start and run their own businesses, to help them increase their income. If, hypothetically, a few years after the program was concluded, a woman who had launched a business through the program was successful, and her business continues to grow and she has increased her income as a result, is this the only definition of a “sustained impact”? Suppose another woman realized that her business idea was not working out (for whatever reason – lack of a market for her product/lack of investment from external sources/ difficulty marketing etc.), and as a result she closed her business, and pursued something else. Is this really a failure of the program? In reality, not every business succeeds, and for some individuals having the opportunity to try opening a business, might be a personal success in and of itself (sense of fulfillment that you tried something that you always wanted to do, no matter what the ultimate outcome might be). Would this really be a failure of a sustained impact, just because her income didn’t increase as a result of the program? What about the experience gained from this exercise of opening a business? Wouldn’t it be worse to continue to invest (time, money etc.) into a business that didn’t become profitable? Although a business that is successful and continues to bring in profits after the program has concluded would be the poster child of a sustainable impact for this project, isn’t there also value in the knowledge and skills gained by those individuals whose businesses failed to do as well? What really is a sustainable impact for this program? What would a timeline look like for sustained impacts?

A related concept that was discussed during the webinar was the idea of conducting follow-up evaluations. One participant questioned “If we are such great systems thinkers, why not ponder the question of what the systems reasons for not doing follow-up evaluations?”, while another chimed in saying that “Funding timelines and expectations are often sustainability killers even though they are project starters”. If you are going to conduct an evaluation years after the program has come to an end, you are hoping that remnants of the program you funded still exist in the community at some functional level. If a follow-up evaluation gets conducted, how are we defining a “successful program”, and how would our ideas of “success” change one year post intervention as compared to four years post intervention?

Planning exit strategies

When interventions are being designed, many different considerations are being taken into account – from macro level factors such as the political and social climate in which one is implementing a project, to meso level factors such as the institutional values of the organization running the program, their resource capacities and constraints, as well as micro level factors such as the demographics of the participants who will interact with the program, the character of the program staff they will interact with etc. However, when thinking about the end of a program it is not often that exit strategies are planned out, and it is perhaps important to create exit strategies a priori to work towards ensuring that impacts can be sustained, and progress can be built upon the change that has been made. One of the webinar participants described, “You know, we don’t kick our kids out of the house at 14, we slowly make sure they can handle money, handle a job, meet their school deadlines, etc. Maybe we should be planning exit strategies more intentionally in all programs, and in that period we would learn more about potential barriers to sustainability, what’s coming up that needs to be dealt with”. Intentionally creating an exit strategy with program participants is perhaps helpful in increasing transparency and helping community members seek other resources or mechanisms through which they might be able to sustain impacts, if they believe that the program was useful to them and brought about positive change. Exit strategies should leverage the connections that were built through the program, and this could include community members, volunteers, potential local funders, and program champions who would continue to bring energy to the program once the original funders have left.

While thinking about sustaining impacts, it is important for all parties present to be reflexive and recognize the roles they play, the limitations of their knowledge, and the tensions that might be present given the different interests of the players present. When evaluating programs in the international development field, it is important to view sustainability under a critical lens, and recognize that there might be tensions between funders/program implementers and the community when trying to sustain programs and their impacts once the funding runs out. On one hand, the program implementer’s job depends on a program working well in a given context. Oftentimes, programs only need to work well enough so that positive changes are observed while the funding is present. However, if programs work “too well”, and communities can sustain positive impacts on their own, they break the cycle of dependency on funders. How do funding organizations stay relevant when the communities they are working with have built the capacity to sustain impacts once the program has come to an end?

« Older posts

All views and opinions expressed in this blog are personal and should not be considered the views of the Evaluation Centre or any other partner organisation.

This page was developed by

The Evaluation Centre for Complex Health Interventions 

© 2024 Evaluating Sustainable Development Goals

Theme by Anders Noren — Up ↑