Embracing Complexity : Making Evaluations Matter

Category: Webinar 4

Insights from webinar four “Leaving no one behind”

TECCHI’S final webinar in the Sustainable Development Goals (SDGs) series was focused on leaving nobody behind. “Leaving no one behind” a phrase commonly used in the global health and international development sectors was unpacked during the webinar, with a focus on how it relates to the field of evaluation. The topic was examined from multiple points of view with some speakers focused on the importance of understanding local contexts and power dynamics while trying to ensure that nobody gets left behind. Another speaker approached the topic from a different lens, by describing how the burden to sustain interventions is often placed on countries that are impoverished and likely lacking in capacity to sustain program outcomes. She debated whether or not “leaving nobody behind” is a reasonable goal to have given the political and economic structures that currently govern our world. This blog post will focus on the dialogue taking place between the webinar participants who engaged in discussions through Zoom’s chat feature. Below is a list of some comments that stood out to me from the conversation:

“The point being raised about trying to understand different perceptions is equally valid in developed economies eg in evaluating programmes for sex workers or drug dependents and possible non uptake of services provided – equally need to search for their understandings and reasons of why there is no take up”  
“Why should be burden the community?  can [we] have a model where the state is held accountable for services that are sensitive to social and cultural realities.”  
“Prettiness of no one left behind isn’t matched”
“Finally, we don’t fund, design, implement, monitor or evaluate for sustainability or consider longer-trajectories post-project. Those are seen as the problems of the countries themselves…. who are poor enough to qualify for aid and often powerless and capital poor who can’t sustain multi-million dollar outcomes…  Just a warning,  ‘complexity’ can be hidden behind – it’s too complex to address, no?”
“I do think the methodologies and methods exist, but are not applied i.a. because of the political economy of evaluation, and of development.”

The first comment talks about the need to understand contexts and power dynamics from the participant’s perspective. While there is a growing realization of the importance of contextual considerations while developing, implementing and evaluating programs, it is important to be reflexive about our position and humble about our contextual knowledge (and lack of it). When working in the international development field there is a lot one might not know with regards to the context they’re working in, and it’s critical to seek out knowledge from program participants and other local actors to understand their perspectives about a given program and how it fits into the fabric of the local community. This comment also reminded me about a striking observation that Dr. Das made during the webinar. He hinted at how the field of development often attempts to fit a poor person into the western “modern” perspective. In his presentation, he talked about the promotion of institutional delivery in Odisha. While the program being implemented defined success as an increase in the number of women giving birth in an institutional facility, it didn’t take into account the fact that many of these same women didn’t even have proper houses to live in.

While the second comment is intriguing, I have to kindly disagree with this statement (“Why should be burden the community?  can [we] have a model where the state is held accountable for services that are sensitive to social and cultural realities”). While it is important for states to have policies and frameworks that take into account social and cultural realities, it is also important for individuals at the community level to play a role in developing programs that are sensitive to their needs. Subsequently, actors at the local level also need to be held accountable to the promises that they make. Operating at the community level enables an organization to build intimate connections with those whose needs they’re serving, and consequently get a pulse on what reality is like from the program participant lens. It enables one to occupy a unique position where you do have the opportunity to listen with humility and lead change that is sensitive to the local context.

Another comment that arose from the discussion is the need to acknowledge the systematic confines that program implementers and evaluators are working within. This includes the fact that funding cycles are generally short-term and often narrowly focused on impacts from the funders’ perspective. Given these circumstances, there might be a failure to acknowledge the work done in obscurity before empirically measurable impacts can be observed. In some cases, this fixation on achieving impacts can hinder experimentation, as the room for failure is very small, and taking risks through experimentation could lead to outcomes that are less than optimal.

While “leaving nobody behind” is extremely important from a human rights perspective, we are aware that our reality is a stark contrast, with inequities growing deeper and the poor being stuck in a vicious cycle that is hard to break through. Given that our words don’t align with our reality, and our inequities are deeply embedded in economic and political systems, how do we work towards reconciling this difference?

Summary of Presentations | No One Left Behind

Abhijit Das

  • Why is it a big deal to be discussing the goal of “Leaving No One Behind”?
  • Historically, most goals have achieved only modest achievements and are unable to show that the people who needed them most have actually been helped
  • Slow progress and modest gains are not enough – We need to ask:
    • Who gains more?  Who gains less?  Who doesn’t gain?  Are there any losers?
  • Development changes relationships – disrupts power structures
    • We need to pay attention to these social power structures in these extremely hierarchical power structures
  • Heterogeneities – will the same range of services work for all?

Understanding Lived Realities

  • Understanding people and communities – What is their self-concept or self-determination? What are their aspirations?
  • State-citizen relationships – What kinds of legal protections are available? What kinds of accountability mechanisms exist?
  • Is marginalization (economic/geographic) the same as social exclusion (stigma/discrimination)?
  • How much diversity of lived health-related experiences have we really understood?
  • Do we understand community concerns and consequences?

Consequences of Poor Services

  • Who pays the costs?  Who is accountable?

The Ethical Conundrum: Who do evaluators serve?

  • Donors, international agencies, governments, or those who often get “left behind”?

Zenda Ofir

Four Points: Beyond the Rhetoric?

  1. Two Sides to the Ethical Imperative of “Leave No One Behind”:
  2. Sometimes certain groups must be left behind – at least for a while. 
    1. It has become a comfortable rhetoric to leave no one behind – has become meaningless
    1. “It’s not about your project! It’s about my country! (Oumoul Ba Tal, Mauritania, 2007)
    1. It may be unethical to expect countries to ensure no one is left behind – far more complex than we may think
    1. Burden of sustaining “no one left behind” efforts inevitably falls on the poorest countries in the Global South – who can’t afford it
  • Development and Evaluation Specialists buy into questionable narratives and assumptions:
  • Examples of questionable narratives about development:
    • Development from the start has to be based on “inclusive growth” and NOLB. Development trajectories of countries and their stage of development do not matter.
    • Funding and effort have to be concentrated on extreme poor countries/areas/communities/groups
    • Progress can be measured – in snapshots – by the performance of the weakest, without considering the sustainability of the progress
  • There is a Particularly Dark Side to Aid-driven ‘Development’ and Evaluation – What Counts as ‘Success’?
  • In the aid world, there is little consideration – in a ‘country’/’society’ context – of:
    • The long-term
    • Development trajectories
    • Poverty traps
    • Skillful policy and practice adaptation to changing conditions
    • What complex systems concepts tell us about how the (social) world works
  • In Both Design & Evaluation, We Need to Pay More Attention to Nuance, to Patterns in Systems, and to the Application of Complex Systems Concepts – especially:
    • Systematic, directed, sequenced experimentation
    • Co-evolution
    • Cascading ripple effects
    • Trade-offs
  • Impressive application of complex systems concepts
    • “How China Escaped the Poverty Trap”, Yuen Yuen Ang

Orielle Solar

  • What challenges for equity need to be addressed to ensure the Sustainable Development Goal of “No One Left Behind”?

Weakness to Address Inequity Problem:

  • Restrictive vision of health and wellbeing
    • Absence of analysis and action to address social determinants of health
    • Limited consideration of socioeconomic-political context
    • Lack of recognition of heterogeneity of sub-populations

Equity Solutions:

  • Social basis of health
    • Social determinants addressed
    • Application of life-course approach
    • Prioritization with equity approach
    • Proportional universalism
    • Multi-sector action
    • Community engagement

Key Challenges for Processes of Regional Adaptation:

  • Enhancing guidance for action on health inequities
    • Integrating multi-sectoral action and social participation to address inequities
    • Supporting capacity-building/strengthening and exchanges
    • Strengthening the equity lens in M&E

Recommendations – Challenge of Enhancing guidance for action on health inequities

  • Strengthening orientations for action on equity
    • Social determinants framework – underlying mechanisms that must be addressed
    • Incorporate equity challenges specific to the region – ethnicity, migration, violence, etc.
    • Strengthen the gender perspective
    • Focus on structural social determinants using the life-course perspective
  • Establishing criteria to set equity-based priorities
    • Where to start (which groups, settings, problems, interventions)?
  • Intersectoral action and social participation – cornerstones to achieving greater equity
  • Participation as an “end” not just a “means”

Recommendations – Challenge of Strengthening the Equity Lens in M&E

  • Measuring progress in indicators of equity and evaluating impact is fundamental
    • Promote and support evaluation thinking in national plans beyond the indicator and monitoring framework
    • Follow a set of indicators that account for inequalities between social groups in terms of outcomes, relevant social determinants, and key implementation processes
    • Stratify indicators as an essential requirement, not an option 

Recommendations – Challenge of Supporting capacity-building/strengthening and exchanges

  • Build capacities of professionals, academics and civil society actors to understand and act upon the social determinants to address equity:
    • Promote forums, courses, and activities related to equity-oriented design, implementation, and M&E of interventions
    • Develop a platform and repository to share and update good practice, learning, and evidence from regional and national experiences
    • Promote a research agenda on policy/intervention evidence

Mel Mark

Many Meanings of “Leave No One Behind”:

  • Absolute goals (e.g., “End poverty”, “End hunger”)
  • Reduce overall disparities (e.g., “reduce inequalities within and between countries”)
  • Reduce/eliminate group-based disparities (e.g., “achieve gender equality and empower all women and girls”)
  • Prioritization of the least well off

Seeming Trends in SDG Evaluation regarding “Leave No One Behind”

  • Disaggregation, attention to distributional/subgroup effects
  • Participatory, inclusive approaches
  • Maybe a move towards evaluation policies

Lots of Action on the Statistical Front

  • UN Economic Commission for Europe, 5th expert meeting on statistics for SDGs
  • Some of it related to LNOB
  • Session 2 – Impact of COVID-19 on SDGs – “Leave No One Behind” Perspective

Tentative Suggestions from 30,000 Feet

  • On the Ecology of Evaluation – Ecological Niche of Evaluation Approaches
    • The ecological niche consists of the role an evaluation approach plays in its environment, including: where it is conducted, what resources it takes, what it does and what its intended and actual consequences are, how it interacts with other evaluation approaches, policy processes, those in its environment
    • Alternative Roles for Different Evaluation Approaches – no single approach sufficient
    • A varied and inclusive ecology of evaluation
      • Can we effectively advocate for multiple evaluation approaches, each used in potentially valuable ways?
  • The Use and Influence of Evaluation
    • The steps from and evaluation to its ultimate consequences – involve multiple levels, processes, often emergent, issues of power, motivation, relational issues, etc.
    • Pathways of evaluation influence – planned and emergent sequences of how evaluation gets from here to there
      • Will we strengthen attention to such pathways and share learning?
      • Motivation, political will may be critical – what is the possible role of evaluation in increasing will?
  • In the Face of Spectacular Ambitions
    • What are the ways that we can supercharge learning (including replication, broadly speaking), dissemination, cultural/contextual adaptation?
    • Don’t let 2030 be an enemy of 2040, 2045, … – What’s the long-term perspective?
    • How do we hold high aspirations regarding evaluation and SDGs, while retaining appropriate modesty?