I was fortunate to be a discussant of Thomas Schwandt’s presentation titled “Indications of Post-Normal Evaluation” organized by the International Evaluation Academy in May 2021.  The presentation built on Schwandt’s far-reaching ideas on ‘post-normal evaluation’ developed in his 2019 article. This is how Schwandt introduces the idea of ‘post-normal’ evaluation: “Post-normal evaluation can be seen, on the one hand, as a result of the failure of normal evaluation to rationalize the world and, on the other hand, as the amplification of the inevitability of and capacity for constant change which can only be managed. We will gauge the success of the future practice of evaluation in terms of whether it is resilient enough to adapt to the ontological realities of complexity, uncertainty, and contradiction in ways other than being methodologically innovative. That resilience is largely an ethical matter, of evaluators taking full responsibility for the choices they make in framing and bounding the evaluations they conduct with the public”.”

The need for a post-normal evaluation predates the COVID-19 pandemic.  The urgency and need for a post-normal evaluation that comes to terms with the uncertainties and incompleteness of knowledge to inform action has only grown with the cold realities of COVID-19.  The challenges of COVID-19 highlight Schwandt’s call for coming to terms with unpredictability, instability, and incompleteness in our knowledge in value determination.

Over the past three years I have been working in India on a range of interventions focused on improving Maternal, Newborn and Child Health.  A number of the interventions focus on strengthening health systems.  Working in the interface between interventions and systems, I have been interested in how evaluations can help build capacities and stronger systems.  I have found the traditional set of evaluation skills on design and individual-level surveys inadequate in understanding system-level constructs and the complex dynamics of system-level changes. 

The goal of this blog is to raise questions that can help sharpen focus on addressing SDGs and COVID-19.  My hope is that these questions will precipitate thinking on the types of skills evaluators need to have in order to contribute to responding to the ongoing pandemic and enhancing local and global responses to SDGs.

Responding to COVID-19

It is hard to think about a post-normal evaluation without asking how evaluation as a field could have contributed more to avert the worst impacts of COVID-19?  This blog is being written at a time when India and many parts of South America are being ravaged by COVID-19.

Despite the number of very insightful blogs written on the implications of COVID-19 for evaluative practice, there is a need for more focused dialogue on how evaluations could have been be more useful during the pandemic.  One guiding question is: In what ways is evaluation as a field contributing to a search for an architecture of a post-normal COVID-19 world?

There are three issues that have emerged as I reflect on systemic responses during COVID-19:

  1. As we witness the collapse of systems in multiple settings during the pandemic, how can the evaluator’s gaze move from a focus on projects and interventions towards system-level capacities and resilience? In my experience, most evaluations focus primarily on projects and interventions; there is limited focus on understanding system and system-level resilience.
  2. There is also a need to pay greater attention to dynamics.  Evaluation’s focus typically tends to make judgements over the short run.  As a field, we have paid limited attention to problems of dynamics.  While there is growing attention to evaluating complex interventions, there is more limited focus on understanding anticipated dynamics of interventions.  COVID-19 has highlighted the need to understand non-linearities in systems change.  What can we do as a field to elevate the focus on dynamics?   
  3. Perhaps most importantly, our preoccupation with saying what works or doesn‘t work has often failed to represent and communicate uncertainties that might exist in our knowledge.  As a field, I think we could do a much better job representing, estimating and communicating uncertainties.  COVID-19 has highlighted the ubiquity of uncertainities. What role can evaluators play in creating greater comfort around uncertainties in social decision-making and social navigation?  Given the uncertainties that exist in responding to almost every facet of the pandemic, how can we as a field play a role in promoting greater comfort in navigating uncertainties?    

Embracing Understanding of Contexts More Deeply

As a realist evaluator, I am interested in understanding the contexts, mechanisms and outcomes associated with interventions.  One insight from realist evaluation is that it’s not interventions that bring about change; rather, it’s interventions under the right context, conditions and support structures that bring about change.  Yet, what strikes me as I work on challenging problems of health and nutrition in a range of settings is that, for the most part, we can do far more as a field to focus on multiple dimensions of contexts and support structures in the planning, implementation, and sustainability of interventions. 

Focusing on Inequities, Sustainability and Heterogeneities

Interventions and system-reform efforts focused on achieving the SDGs will need to pay attention to problems of inequities, heterogeneities of contexts, and finding solutions that can have sustainable impacts.  Addressing problems of inequities will require a focus on what ‘works for whom under what conditions’ and also attention to the heterogenous needs of different intersectional segments.  My sense is that, as a field, we have not done enough to bring a sufficient focus on problems of inequities, sustainability and heterogeneous solutions.  Questions that I have raised in other settings include: What is a theory of change that is serious about addressing inequities?  How does a theory of change focused on sustainable impacts differ from a theory of change of immediate or intermediate impacts?  How do we both represent and translate insights on heterogeneities of solutions that would be needed to address the diverse needs across different contexts?

Towards an Ecology of Evidence

We also need to ask tough questions on what types of evidence are useful.  It’s perhaps not enough to simply say that we’re moving from evidence-driven interventions towards evidence-informed interventions.  I think we need to raise deeper question around the ecology of evidence needed to implement and sustain systems and interventions over time.  What are the types of evidence needed to make a difference in inequities?  To help plan for sustainability?  Given the multiple definitions of complexity including contextual, dynamic, and multi-components, how do we promote a view of evaluation that generates knowledge on the contexts, processes, and impacts associated with systems, programs, and policies?  

Evaluation as Way Finding: Towards an Ecology of Solutions

An important concept that Schwandt and others have raised is that of evaluation as ‘way-finding.’  Schwandt raised this as an issue of being focused on solving problems, not just being preoccupied with specific solutions.  Developmental economist Lant Pritchett’s insight is especially helpful here: “We should be in the business of solving problems, not just of selling solutions”.  Addressing challenging problems like maternal mortality, hunger, and nutrition will require more than a singular solution; an ecology of solutions tailored to the needs of specific contexts may be needed.  Further, such solutions have an essential dynamic long-term aspect to them because it’s unlikely that solutions to difficult problems like inequities, maternal mortality, and malnutrition can be found solely with quick fixes.  As a field, evaluation needs to more clearly explore what does evaluation as way-finding look like—we need to shift our focus to ‘what does it take to solve a problem in specific contexts’, rather than the more commonly addressed question ‘does intervention ’x’ work’.  Relatively recent approaches like Developmental Evaluation and Principles-Focused Evaluation offer great promise in providing insights on evaluation as way finding.  

Questioning the Roles of Evaluators: Understanding Influence

In my view, any focus on a ‘post-normal’ needs to address the role that evaluators play and the changing roles of evaluators.  If one accepts that we should not only pay attention to interventions/projects, but also incorporate a broader understanding of a range of systems, what should be the role of evaluations and evaluators in building the capacities of systems?  Could evaluation as a field have done more to promote a focus on the ‘known unknowns’ and ‘unknown unknowns’ of societal responses to COVID-19? While I think there has been a focus on and around the different roles of evaluations, we can do more to discuss the multiple roles of evaluation in navigating and discovering solutions for complex problems. Much of our discussions have been about solutions in specific contexts.  As evaluation seeks to have influence in the technical, social, and political environments in which it is located, there is a need to reflect not just on the pathways of influence of a single evaluation, but also on how a variety of evaluative products/approaches can help move understanding and solutioning of a problem.  I find the concept of a ‘fractured information ecology’ especially useful in helping locate both the role of evaluations, as well as understanding how evaluations can have influence: “However, there have been even more worrying signs of a fracturing information ecology at the interface of science, policy and public discourse.”

We need better narratives on the boundaries of our influence and how evaluations can have influence despite the fractured information ecologies in a number of decision-making environments.

Urgency in Action

There needs to be urgency in rethinking our role as evaluators and how we can be both simultaneously disruptive and constructive as we respond to the fractured information ecology by promoting collaborative problem-solving.  “The damaged information ecology in which global and national responses to COVID-19 are currently playing out should be of deep concern to all those working towards the adoption of more sustainable policies, economics and ways of life.”  As a field, we need to spur focus on actual examples of how evaluators have navigated changes in fractured contexts, built an ecology of evidence, and also navigated the boundaries of influence as the contexts of learning have changed, shrunk, or expanded.