The Aspiration and the Responsibility
The Sustainable Development Goals (SDGs) provide a spectacular vision for
Further, the SDGs have a sharp focus on “leaving no one behind”:
Such grand visioning and rhetoric need to be tempered by harsh realities how COVID, Climate Change, debts and discriminatory practices and structures have had cumulative and intersectional impacts on poor countries and poor people.
Change often happens through ambitions and aspirations to achieve the near impossible. While the cold gaze of measurement and design should not dampen the enthusiasm inherent in long-term ambitions, the evaluation community could ethically and responsibly provoke questions around the types of systems changes, interventions, partnerships, governance structures and learning systems that might need to be in place to help achieve the goals.
Good evaluations often start with understanding pathways of impacts. For example, the whole essence of theory-of-change approaches to evaluation question is the humble question: How?
I work in development/global health, with a majority of my work in the past three years in India. While there is a clear focus on indicators both in the SDG documents and planned interventions/systems change efforts, there continue to be ambiguities on the roles that evaluation and evaluators can play in helping achieve such spectacular ambitions.
In this blog I raise questions related to the role evaluations need to play in helping achieve such grand goals. How do we move from a position of being passive responders to a more active stance that demonstrates leadership for enhancing the likelihood of achieving such grand visions?
Getting Real
Understanding the cumulative, dynamic pressures of COVID, debts, discrimination and climate change become especially important as we seek to locate our role. Consider this recent article from April 2021:
It is important that responding to such challenges are not seen as problems confined to the developed world. Many of the syndemic problems of racism, inequities and fractured civic ecologies are very much a reality in the ‘developed North. Consider this Guardian article from February 2021:
As the evaluation community seeks to find and assert its voice in contributing to SDGs, it is important to take stock of naming the processes that contribute to amplifying the problems. In my experience, our focus tends to be the “intervention” or the “project”: we can get caught up in the project trap. We tend to focus less on the systems and syndemic processes that generate the conditions of poverty and inequities. It is within the purview of evaluation and evaluative thinking to promote realism about the context and support structures and bring a clearer focus on systems and dynamic processes that will help achieve the vision of “no one left behind.”
As a field, we need to both recognize the formidable nature of the tasks and move toward action. As Ibram Kendi has brilliantly shown in How to be an Antiracist addressing the impacts of racism requires active commitment and action to reversing inequities. Lazily claiming to be a “non-racist” or “neutral” and “above the fray” or dismissing opportunities to push an agenda that takes inequities seriously with ‘this intervention has little to do with inequities’ is hardly enough. How do we move as a field beyond our passive focus on projects and interventions? Should taking a more disruptive stance towards provoking clarity around the pathways, seriousness of addressing inequities, and realism about the complexity of the change process be part of how we approach our role and responsibility in helping achieve the SDGs?
Questions Galore
I briefly expand on these above issues by raising questions that I think are important as we locate our role in addressing the SDGs. My questions that seek to clarify roles are in four domains:
- Building capacities and sustainable impacts
- Multi-dimensional contexts and trajectories of outcomes
- Understanding the coherence and the disruptive powers of interventions
- What does it take to “leave no one behind”?
1) Building system-level capacities and sustainable impacts
It’s important to recognize that much of evaluators’ focus on impacts has not been matched with ensuring that these impacts are sustained in the longer run, after the funding of interventions has ended. As example, Jindra Cekan has been a leader in calling out how few evaluations focus on issues of emerging and sustained impacts. Along with colleagues, I have argued elsewhere that we need to integrate planning for sustainability into our thinking of theories of change as we understand how interventions/systems can bring about long-term change. The interface between interventions and whether these innovations are absorbed by systems capable of implementing the interventions in a sustainable way still needs deeper exploration. In a similar vein, there’s a need to pay attention to system level capacities including ‘adaptive’ and ‘absorptive’ capacities of systems. As we focus on the interface of interventions and systems, some questions arise: What is a theory of change that focusses on sustainable impacts? How do we operationalize systems’/institutions’ abilities to absorb interventions? How can understanding mechanisms of sustainability occur at multiple levels? How can concepts and methods from complexity sciences help inform our theorizing and measurement of sustainable impacts (see Zenda Ofir’s blog on viewing development as a complex adaptive system)? The challenges of understanding development from a complexity lens goes beyond measurement: How can ideas of complexity sciences inform our view of what constitutes evidence?
2) Multi-dimensional contexts and trajectories of indicators
As an evaluation community we could be more focussed in understanding what are the multi-dimensional contexts that make changes (and reaching SDG targets) harder in some contexts than others. Much of the results frameworks that I see still have anticipations of success that are based on aspirations of ‘mechanical,’ ‘aspirations’ of results often based on linear extrapolations of past patterns. How can we incorporate contexts more deeply into our planning for what changes are realistic in different settings? We need a broader set of dialogues on what contexts matter and also how changes in context and planning for interventions in especially resource-deprived contexts need to inform our expectations of success. Further, we need better examples of how progressive evaluation approaches such as developmental evaluation, realist evaluation, and principles-focused evaluation help build a better understanding/intuition of what are realistic changes especially in very “sticky,” difficult to change conditions.
A focus on multidimensional contexts can also shed light on the anticipated kinetics of systems change. I have found our approaches and tools remarkably limited for understanding anticipated timelines, trajectories and rates of change (see Michael Woolcock’s work for a refreshing introduction to the importance of these concepts).
3) Understanding the coherence and the disruptive powers of interventions
One of the significant aspects of the newly developed Development Assistance Criteria (DAC) is its focus on coherence as one of the evaluation criteria. How can evaluators help in mapping out the alignment between different levels of interventions to promote their coherence and disruptive powers? The mechanisms by which coherence matters and the contexts in which coherence matters need to be more clearly understood. Under what conditions should interventions be coherent/aligned with the rest of its local ecology/context? Alternatively, under what conditions can a lack of coherence add to the disruptive powers of an intervention in leading to favourable changes? Under what conditions should interventions not try to fit in with existing interventions, especially in contexts in which the contexts itself needs changes? Thinking about the coherence of multiple interventions forces us to explore design considerations.
4) What does it take to “leave no one behind”?
What could the evaluation community be doing to help operationalize strategies to population segments that correspond to intersectional categories of multiple disadvantages? Working in the international development/global health space, I continue to be surprised at the shallow nature of most theories of change focussed on inequities. The same focus that the evaluation community has spent on deciding what is a good, rigorous evaluation design we have not spent in answering what is a good enough theory of change. This problem is especially acute when we consider interventions that have the aspirations of addressing problems of inequities. What is a good enough theory — and by this, I simply mean a theory that can seriously aid translation/implementation — to address inequities? Addressing these questions will help position our field to be more relevant in any dialogue that focuses on making a difference in the lives of individuals in need.
Looking Ahead
The SDGs provide fundamental challenges and amazing opportunities for the evaluation community to go beyond standard, stale ways of valuing. Addressing some of the above questions will have implications for how we think about evaluation competencies and how we approach building evaluation capacities. There is a need for the evaluation community to engage more deeply with both conceptual and evaluative issues involved with sustainability, inequity, contexts, disruption and coherence.
Addressing some of the above questions will require new partnerships, not just between evaluators of different stripes but between the evaluation and a number of other communities/sectors including ecologists, public health and nutrition, financial services for the poor, epidemiologists, etc. We might also need to consider how diverse evaluation approaches can be combined with other measurement opportunities (such as routine surveillance, etc.) to help build useful ecologies of evidence.
This is an exciting time to be an evaluator and an exciting time to rethink evaluation competencies and capacities.
