Evaluating Complexity

Rethinking Evaluation Competencies in 2021: Questions to Focus Our Efforts

3 June 2021 / Sanjeev Sridharan / 0 Comments

Competencies can be defined as “clusters of related knowledge, skills, abilities, and other requirements necessary for successful job performance.” This definition taken from the Unites Nations Evaluation group suggests a concept of competency that is defined at the individual level. One challenge as we look ahead is to also explore how organizational and system-level supports can help enhance evaluation “job performance.”

The Development Monitoring Evaluation Office (DMEO) of Niti Aayog in India had organized a panel titled, “Systemic Essentials for Strengthening Government Evaluation Capacities” in June 2021. In this blog I reflect on some questions on evaluation competencies that can help sharpen our focus.

Learning from other discussions on evaluation competencies

A good starting point is to explore some examples of evaluation competencies. The UN Evaluation Group in 2016, for example, in its focus in achieving Sustainable Development Goals, Gender, Equality and Human Rights, listed the following five domains of evaluation competencies:

Professional Foundation – these include ethics and integrity, evaluation norms and standards
Technical Evaluation Skills – these include evaluation approaches, methods and data analysis
Management Skills – work planning, coordination, and supervision
Interpersonal skills – these include communication, facilitation, and negotiation
Promoting a Culture of Learning for Evaluation – these include focus on utilization and integration of evaluation into policy and programs

The American Evaluation Association in 2018 focus is on the following five domains:

Professional Practice
Methodology
Context
Planning and Management
Interpersonal

Any policy maker or evaluator interested in what evaluation capacities matter is also advised to explore other evaluation competencies including the Aotearoa New Zealand Evaluation Association competency document from a decade ago (2011). One focus here is on paying attention to cultural values in shaping competencies: “… our intention is to ensure that cultural competency is not treated like a peripheral or marginalised aspect, rather a central component of the development of our framework of evaluation competencies and practice standards.” They go on to add: “Cultural competency is central to the validity of evaluative conclusions as well as the appropriateness of the evaluation practice.”

Similarly, the article by Jean King and co-authors from 2001, titled, “Toward a Taxonomy of Essential Evaluation Competencies” is also a must-read for anyone interested in the challenge of defining evaluation competencies. Jean is a friend, and we continue to collaborate and communicate. I know what continues to engage her is: How can systems and organizations sustain evaluation capacities over time? The big challenge as I see it is not about competencies at the individual level, but what it would take to ensure that the capacities of the broader ecosystem and organizations are sustainable over time.

Questions to precipitate action

All of the above serves as a backdrop of what I think are the essential questions that the field of evaluation and organizations like the DMEO will need to address:

The Opportunity

1) Focusing on Sustainable-Capacities Systems and Organizations: What programs of work can help build evaluation capacities in the broader ecosystem? How can it help build capacities at the organizational level in a way that capacities are sustained even if key individuals leave organizations? A good starting point will be to recognize we don’t have good answers to these questions.

2) Embracing the Sustainable Development Goals (SDGs): The SDGs provide a remarkable opportunity for the entire field of evaluation to prove its salience and utility. What kinds of evaluation skills can help practitioners promote an understanding of system dynamics, coherence across interventions, focus on sustainable impacts and “no one left behind” to help contribute to enhancing evaluation’s contributions to achieving the SDGs? The relevance of each of these concepts were discussed in a remarkable set of webinars hosted by the Evaluation Centre for Complex Health Interventions in Toronto.

3) Competencies for Evaluators in the Public Sector – The enormous opportunity that the dialogue led by DMEO presents is to help raise questions around what specific competencies are needed for evaluators working within governments. How can a program of work on building evaluation capacities help an evaluator working within the Central or State government do their jobs better?

Some Contemporary Challenges

4) Focus on Diversity, Equity, and Inclusion — Much of our dialogue around evaluation capacities and capabilities are occurring at a time when we are facing deep discussions and divides around inequities, hierarchies and privilege: How can evaluators create greater voice and inclusiveness in understanding the impacts of programs and policies? I do think that as a field we need to more clearly understand evaluation’s role in addressing inequities and promoting inclusion.

5) Understanding the Architecture of Complex Programs and Policies — My view is that most evaluators have a poor understanding of representing the complexities of social programs and policies. Our tools to understand the architectures of complex programs and policies are quite limited. How do we promote understanding of theories of change of complex interventions? How do we build competencies in understanding the program mechanisms/processes that make a difference to impacts? How do we more clearly represent and understand contexts and its vital role in the success of policies and programs?

6) Enhancing Interpersonal Skills: I find evaluation as a profession both in the North and the South still focused primarily on the technical. I am unsure that we have cracked how to build interpersonal skills. How does one build a program of work that enhances the interpersonal skills of evaluators?

7) Towards Complexity-Informed M&E: Models of Continuous Improvement – While much of our preoccupation has been on ‘what works’ and ‘what doesn’t work,’ I’m unconvinced that we have paid as much attention to how systems and organizations improve in an ongoing manner. Most interventions, and organizations are ‘complex systems thrust amidst complex systems.’ What are programs of evaluation capacity building that can help inform more complexity-informed monitoring and evaluation?

8) Skill Sets to Synthesize Evidence and Build an Ecology of Evidence – Another area that I find missing is an ability to synthesize very disparate sources of information towards creating a policy environment in which an ecology of evidence is used to make decisions. Despite all of our focus on mixed methods approaches, I’m not convinced that we have trained individuals to synthesize evidence that can help build an ecology of evidence.

Looking ahead

I do want to end by reiterating that this is an exciting time to raise questions around evaluation competencies. With the focus on SDGs and recent debates on what evaluation criteria matter, this rich dialogue around competencies can lead towards a more vibrant field of evaluation both in India and globally. It is perhaps fitting to end with remembering one of the founders of the field of modern-day evaluation, Donald Campbell. His vision of evaluation was that of a “mutually monitoring, disputatious community.” It’s important to recognize that our work can lead towards evolution of knowledge and systems and mostly importantly improved lives and a more sustainable world. There will be differences and disputes as we argue about different views of progress. How do we promote an ecosystem of evaluators that have both confidence and grace while disputing diverse views of what constitutes progress?

Accelerating towards a Post-Normal Evaluation?

20 May 2021 / Sanjeev Sridharan / 0 Comments

I was fortunate to be a discussant of Thomas Schwandt’s presentation titled “Indications of Post-Normal Evaluation” organized by the International Evaluation Academy in May 2021. The presentation built on Schwandt’s far-reaching ideas on ‘post-normal evaluation’ developed in his 2019 article. This is how Schwandt introduces the idea of ‘post-normal’ evaluation: “Post-normal evaluation can be seen, on the one hand, as a result of the failure of normal evaluation to rationalize the world and, on the other hand, as the amplification of the inevitability of and capacity for constant change which can only be managed. We will gauge the success of the future practice of evaluation in terms of whether it is resilient enough to adapt to the ontological realities of complexity, uncertainty, and contradiction in ways other than being methodologically innovative. That resilience is largely an ethical matter, of evaluators taking full responsibility for the choices they make in framing and bounding the evaluations they conduct with the public”.”

The need for a post-normal evaluation predates the COVID-19 pandemic. The urgency and need for a post-normal evaluation that comes to terms with the uncertainties and incompleteness of knowledge to inform action has only grown with the cold realities of COVID-19. The challenges of COVID-19 highlight Schwandt’s call for coming to terms with unpredictability, instability, and incompleteness in our knowledge in value determination.

Over the past three years I have been working in India on a range of interventions focused on improving Maternal, Newborn and Child Health. A number of the interventions focus on strengthening health systems. Working in the interface between interventions and systems, I have been interested in how evaluations can help build capacities and stronger systems. I have found the traditional set of evaluation skills on design and individual-level surveys inadequate in understanding system-level constructs and the complex dynamics of system-level changes.

The goal of this blog is to raise questions that can help sharpen focus on addressing SDGs and COVID-19. My hope is that these questions will precipitate thinking on the types of skills evaluators need to have in order to contribute to responding to the ongoing pandemic and enhancing local and global responses to SDGs.

Responding to COVID-19

It is hard to think about a post-normal evaluation without asking how evaluation as a field could have contributed more to avert the worst impacts of COVID-19? This blog is being written at a time when India and many parts of South America are being ravaged by COVID-19.

Despite the number of very insightful blogs written on the implications of COVID-19 for evaluative practice, there is a need for more focused dialogue on how evaluations could have been be more useful during the pandemic. One guiding question is: In what ways is evaluation as a field contributing to a search for an architecture of a post-normal COVID-19 world?

There are three issues that have emerged as I reflect on systemic responses during COVID-19:

As we witness the collapse of systems in multiple settings during the pandemic, how can the evaluator’s gaze move from a focus on projects and interventions towards system-level capacities and resilience? In my experience, most evaluations focus primarily on projects and interventions; there is limited focus on understanding system and system-level resilience.
There is also a need to pay greater attention to dynamics. Evaluation’s focus typically tends to make judgements over the short run. As a field, we have paid limited attention to problems of dynamics. While there is growing attention to evaluating complex interventions, there is more limited focus on understanding anticipated dynamics of interventions. COVID-19 has highlighted the need to understand non-linearities in systems change. What can we do as a field to elevate the focus on dynamics?
Perhaps most importantly, our preoccupation with saying what works or doesn‘t work has often failed to represent and communicate uncertainties that might exist in our knowledge. As a field, I think we could do a much better job representing, estimating and communicating uncertainties. COVID-19 has highlighted the ubiquity of uncertainities. What role can evaluators play in creating greater comfort around uncertainties in social decision-making and social navigation? Given the uncertainties that exist in responding to almost every facet of the pandemic, how can we as a field play a role in promoting greater comfort in navigating uncertainties?

Embracing Understanding of Contexts More Deeply

As a realist evaluator, I am interested in understanding the contexts, mechanisms and outcomes associated with interventions. One insight from realist evaluation is that it’s not interventions that bring about change; rather, it’s interventions under the right context, conditions and support structures that bring about change. Yet, what strikes me as I work on challenging problems of health and nutrition in a range of settings is that, for the most part, we can do far more as a field to focus on multiple dimensions of contexts and support structures in the planning, implementation, and sustainability of interventions.

Focusing on Inequities, Sustainability and Heterogeneities

Interventions and system-reform efforts focused on achieving the SDGs will need to pay attention to problems of inequities, heterogeneities of contexts, and finding solutions that can have sustainable impacts. Addressing problems of inequities will require a focus on what ‘works for whom under what conditions’ and also attention to the heterogenous needs of different intersectional segments. My sense is that, as a field, we have not done enough to bring a sufficient focus on problems of inequities, sustainability and heterogeneous solutions. Questions that I have raised in other settings include: What is a theory of change that is serious about addressing inequities? How does a theory of change focused on sustainable impacts differ from a theory of change of immediate or intermediate impacts? How do we both represent and translate insights on heterogeneities of solutions that would be needed to address the diverse needs across different contexts?

Towards an Ecology of Evidence

We also need to ask tough questions on what types of evidence are useful. It’s perhaps not enough to simply say that we’re moving from evidence-driven interventions towards evidence-informed interventions. I think we need to raise deeper question around the ecology of evidence needed to implement and sustain systems and interventions over time. What are the types of evidence needed to make a difference in inequities? To help plan for sustainability? Given the multiple definitions of complexity including contextual, dynamic, and multi-components, how do we promote a view of evaluation that generates knowledge on the contexts, processes, and impacts associated with systems, programs, and policies?

Evaluation as Way Finding: Towards an Ecology of Solutions

An important concept that Schwandt and others have raised is that of evaluation as ‘way-finding.’ Schwandt raised this as an issue of being focused on solving problems, not just being preoccupied with specific solutions. Developmental economist Lant Pritchett’s insight is especially helpful here: “We should be in the business of solving problems, not just of selling solutions”. Addressing challenging problems like maternal mortality, hunger, and nutrition will require more than a singular solution; an ecology of solutions tailored to the needs of specific contexts may be needed. Further, such solutions have an essential dynamic long-term aspect to them because it’s unlikely that solutions to difficult problems like inequities, maternal mortality, and malnutrition can be found solely with quick fixes. As a field, evaluation needs to more clearly explore what does evaluation as way-finding look like—we need to shift our focus to ‘what does it take to solve a problem in specific contexts’, rather than the more commonly addressed question ‘does intervention ’x’ work’. Relatively recent approaches like Developmental Evaluation and Principles-Focused Evaluation offer great promise in providing insights on evaluation as way finding.

Questioning the Roles of Evaluators: Understanding Influence

In my view, any focus on a ‘post-normal’ needs to address the role that evaluators play and the changing roles of evaluators. If one accepts that we should not only pay attention to interventions/projects, but also incorporate a broader understanding of a range of systems, what should be the role of evaluations and evaluators in building the capacities of systems? Could evaluation as a field have done more to promote a focus on the ‘known unknowns’ and ‘unknown unknowns’ of societal responses to COVID-19? While I think there has been a focus on and around the different roles of evaluations, we can do more to discuss the multiple roles of evaluation in navigating and discovering solutions for complex problems. Much of our discussions have been about solutions in specific contexts. As evaluation seeks to have influence in the technical, social, and political environments in which it is located, there is a need to reflect not just on the pathways of influence of a single evaluation, but also on how a variety of evaluative products/approaches can help move understanding and solutioning of a problem. I find the concept of a ‘fractured information ecology’ especially useful in helping locate both the role of evaluations, as well as understanding how evaluations can have influence: “However, there have been even more worrying signs of a fracturing information ecology at the interface of science, policy and public discourse.”

We need better narratives on the boundaries of our influence and how evaluations can have influence despite the fractured information ecologies in a number of decision-making environments.

Urgency in Action

There needs to be urgency in rethinking our role as evaluators and how we can be both simultaneously disruptive and constructive as we respond to the fractured information ecology by promoting collaborative problem-solving. “The damaged information ecology in which global and national responses to COVID-19 are currently playing out should be of deep concern to all those working towards the adoption of more sustainable policies, economics and ways of life.” As a field, we need to spur focus on actual examples of how evaluators have navigated changes in fractured contexts, built an ecology of evidence, and also navigated the boundaries of influence as the contexts of learning have changed, shrunk, or expanded.

Some Reflections on Learning from the Pandemic: Reimagining the role of Community Organizations During the Pandemic

25 July 2020 / Diane Walter

Diane Walter, Executive Director, Margaret's Housing and Community Support Services

Margaret’s Housing and Community Support Services is a multi-service agency that provides a continuum of housing options and wrap-around supports to women living with mental illness and substance use, as well as community-based support services including several drop-in centres and respite services in Toronto for individuals experiencing intersectional issues tied to mental health challenges and homelessness

The Evaluation Centre posed a number of questions for me to reflect on. I share some brief reflections below. My hope is that such reflections from both me and others on my team and partners at the Evaluation Centre lead to systemic learning and system-level solutions in the future.

At such a difficult time, what has kept you going?

What keeps me going more than anything else since the pandemic is the knowledge that some of the folks that we’re serving really have no options. Recognizing that we can make a difference in someone’s life by making sure that a staff member delivers some groceries, or there’s a wellness check-in, for example, keeps me going. Also, recognizing that although I’m a minor cog, we might still be helpful during this pandemic in somebody’s life, this gives me energy even when I don’t have energy.

What have you learned during the pandemic?

The generosity of my staff, some partners and Torontonians
The generosity of Torontonians with donations of cash and PPE and both my team and our partners who have worked closely and well inspires me. What I have discovered is there are many staff who weren’t stars necessarily before the pandemic; they did their jobs adequately and met their deliverables; but during the pandemic they have shone and seem to thrive in the chaos – you can call them at any time, and they just step up and deliver in spades.
Leadership stepping up to the plate
Some leaders in Toronto and Ontario have pleasantly surprised me. I’ve seen some leaders during the pandemic lead with compassion and empathy. I have also grown to respect some leaders’ ability to communicate to the public in a near daily basis that they have the backs of Ontarians, whatever the challenges are.
This pandemic spotlights the need for housing as a solution
We can only hope that this pandemic is galvanizing all three levels of government in the need for housing. I hope the government can do an accounting to see if the pandemic has cost them more than if they had addressed the homeless situation 4-5 years ago. For example, the City has opened up multiple hotel programs — rented hotel rooms to put homeless people who have tested positive or are waiting on results and those from encampments. If they had funneled those funds into building the real bricks and mortar of housing, I think things would have been significantly different. Had housing been thought about in a more deliberate solution-based fashion, this pandemic would not have been so costly.
The Importance of Prevention: Reimagining health care
I think most health care is delivered in the community, but hospitals are the recipients of the bulk of our healthcare dollars. If we were to re-imagine what a healthcare system is — not an acute care system or a sickness system, but a healthcare system — where more people in the community are trained to do simple tasks so that folks would not end up in the hospital system, given the fact that when people end up in the hospital it’s often because all the prevention and all the incremental interventions did not occur to prevent them from ending up in an expensive hospital bed, They could have been treated in the community had there been adequate prevention and adequate interventions, even something simple before prevention.

What recommendations do you have in response to this pandemic?

We have to get ready for the next wave. Historically pandemics often have a second wave. Some questions we as a system need to answer: Is it going to be worse than the first one? Is it going to be more decimating? We don’t know. But for us, we certainly have learned some things that we’re preparing ourselves should there be a second wave. For example, we’re making sure that our inventory of PPEs will take us into and perhaps beyond the second wave; some of the protocols around IPAC (Infection Prevention and Control) we are now making sure they’re ramped up, and we’re aggressively following those infection IPAC standards.
There are learnings around how a pandemic like COVID-19 could so comprehensively affect how we operate. For example, within the drop-in respite setting, we were shifting from using Styrofoam to reusable utensils because we wanted to reduce our footprint. We were really gung ho about going Green, but now that is not on the agenda for awhile; it’s totally off the table. We’re back to Styrofoam because clients are more likely to mistakenly pick up and use a reusable utensil or coffee cup that another client has used rather than one that gets thrown out. We’re looking into whether we can use other materials, but the other materials are so expensive.
We can learn some things from this pandemic that are actually positive. In terms of guidelines, we are adhering to those around PPE and making sure staff are spaced — it will affect some of the ways we work in terms of front-facing, client-facing service delivery. But we are seeing the value of virtual interaction – however, we haven’t gotten to that place where we really take a deep dive and assess it. These are things that are just coming up, and where we are thinking, “That we can amend. This we can modify.” For example, we can integrate the virtual piece into our work seeing clients one-on-one. The nurse that we use and the psychiatrist have realized that some people prefer the virtual interaction with the psychiatrist than the one-on-one
The need for learning and developing system-level solutions
My view is that what we desperately need is to learn from the experiences of community organizations providing care during this time of the pandemic. Evaluation teams like the Evaluation Centre have a role in precipitating dialogues that can lead to co-constructed systemic solutions across multiple levels of government. The partners of organizations serving the homeless need to become learning organizations that better understand the importance of coordinated systemic solutions in addressing problems of chronic homelessness. It is simply not sufficient to say, “we are all in this together.” We need to move beyond rhetoric –what is needed is a coordinated system in which all partners feel valued and supported.

Towards an Ecology of Care during the Pandemic: Questions to help Frame System-Level Responses for Chronic Homelessness

25 July 2020 / April Nakaima and Rachael Gibson

Staff of the Evaluation Centre in Collaboration with Our Partners at Margaret’s Housing and Community Support Services and other Community Organizations

As part of an evaluation that the Evaluation Centre is conducting with Margaret’s Housing and Community Support Services, and through speaking to other providers of care in the community sector serving marginalized individuals, a number of questions has surfaced during this pandemic that can help with improving systems to create solutioning for individuals who face multiple disadvantages at the intersection of housing and mental health. The Evaluation Centre is presently working on a full-fledged report that discusses the implications of these questions for both understanding problems and generating solutions.

Some of these questions will need to be addressed by conducting summative evaluation approaches in the near future. However, most of these questions will require developmental evaluation approaches that need to be co-constructed between the organizations working to provide an ecology of care for individuals who face co-occurring conditions of homelessness, mental illness, and other issues such as substance use, gender identity discrimination, racial discrimination, and disabilities.

Understanding syndemic processes: An important issue that surfaced in our dialogues with multiple community stakeholders was the conceptual need to better understand mutually reinforcing vicious cycles that might connect chronic homelessness to poor health. The pandemic might further serve to exacerbate these connections by both reducing capacities of community organizations as well as by worsening health conditions for marginalized individuals. How will an understanding of the syndemic processes of chronic homelessness inform better planning for the ongoing and future pandemics?
Learning from successful drop-in/respite sites: Not all Drop-in/Respite sites have seen problems during the pandemic. Some have had outbreaks; others have had no cases. Can a comparative analysis of such facilities help us better understand factors associated with successful responses to the pandemic?
More deeply understanding the role of community organizations in preventing and responding to the pandemic: In our multiple dialogues what has clearly surfaced is the great need to better value the role of community organizations in an ecology of care in addressing the problems of chronic homelessness and persistent mental health challenges. What role does research and evaluation play in creating better awareness of the value of community organizations in such an ecology of care?
Equity in distribution of PPE. A related set of basics questions that emerged from multiple dialogues with community organizations has included a focus on equity in distribution of PPE and testing kits: How can there be a clearer prioritization of the importance of community organizations in responding to the pandemic? What are more equitable ways of distributing testing kits, PPE, etc.?
Managing fatigue: A number of organizations working on homelessness are already overstretched, under-funded, and even during normal times the staff are potentially over-worked and stressed. Combined with the fear and the potential chaotic disruption caused by the pandemic, important questions for both systems and organization leaders working on chronic homelessness include: How does one manage fatigue/stress among frontline staff addressing homelessness given the long hours and extreme stress that they are facing during the pandemic? Going forward how does one build surplus capacity in community organizations given the potential organizational disruptions caused by the pandemic?
Leveraging political and social opportunities to find solutions to chronic homelessness: Stakeholders also noted that the pandemic also has created a political climate in which key actors who perhaps may not have been cognitively and politically sensitive to recognizing the need for multi-sector approaches to chronic homelessness are becoming more understanding and supportive of the interconnections between social and economic factors that drive chronic homelessness. Stakeholders noted that such favourable changes in a recognition of the multilevel complex drivers of chronic homelessness can be seen in the three levels of government. Can more be done to leverage such changes in political climate to create policy and practice solutions that are aligned across the three levels of government?
Generating client-sensitive solutions: Another point that surfaced in our multiple dialogues was the need to respond to the pandemic in ways that recognizes the lived realities of clients. For example, providing hotel rooms to isolate individuals who have tested positive or are awaiting test results might not work for some individuals whose mental health may de-compensate in such an environment. How do we think more systemically about solutions that are more client-sensitive and especially acknowledge challenges of anxiety and depression among some marginalized populations?
Having better established communication protocols: A point raised by multiple community stakeholders in the initial stages of the pandemic was the need to have clearer protocols/guidelines when faced with the disruptions caused by the pandemic. The lack of clear protocols/guidelines was deeply problematic for at least a few clients. Some of the issues that were raised in terms of a lack of communication included: lack of clarity on how long testing would take, lack of clarity of where individuals needed to be housed or stop-gap solutions that respected client contexts and client choices. What are processes going forward to develop better communication protocols between the hospitals/public health sector and the community sector?
Creating systems of coordinating and supporting community care: Another point that surfaced in multiple discussions was the experience of some community support and housing organizations that felt very much alone as they responded to the pandemic. Some of the issues that surfaced pointed to a lack of system coordination including: the need for more thoughtful systems of alerts, the need for data sharing, and the need for clearer protocols and guidelines mentioned earlier. Some of the stakeholders we spoke to did not feel supported in their efforts to thwart the pandemic or felt that there was not a system of care to support them. How does one create a more caring, coordinated system of care in which organizations don’t feel alone in responding to a pandemic?
Learning solutions from the pandemic: Some stakeholders we spoke to also spoke about some of the lessons that they have learned about the power of virtual interaction for some clients. For example, some clients have accessed more time with their psychiatrist virtually and have preferred the online interactions to their previous face-to-face interactions. Going forward, are there hybrid models in which face-to-face solutions can be combined with online solutions for some clients?

We believe that addressing the above points both through individual evaluations of community care organizations’ experiences and through generating solutions systemically perhaps through a systems-focused developmental evaluation process might be needed. It’s important we do not treat the issues raised above as just realities affecting one organization; in our work evaluating multiple community interventions during the pandemic and dialogue through the webinars that the Evaluation Centre have organized, we hear very similar concerns being raised by community organizations in countries like Chile, the UK, and India. Any solutions that we arrive at within Ontario will in all likelihood have relevance in other settings.

Valuing Innovations: Evaluating the Online Platform of Dancing with Parkinson’s

14 May 2020 / April Nakaima and Rachael Gibson

The Evaluation Centre and Dancing with Parkinson’s (DWP) started collaborating in 2015 through a project funded by the Ontario Brain Institute to evaluate DWP’s in-person dance classes. Our relationship with DWP was only formal for the first year; however, over the past 4 years our collaboration has continued, as Sarah and her team continue to challenge colleagues at the Evaluation Centre to think creatively about how evaluation can help community organizations, such as DWP, better tell their contribution story. The need for such stories has only grown given the present pandemic. Last month we started an evaluation of DWP’s most recent innovation: live, online dance classes. This evaluation provides an opportunity to further develop this mode of delivery, build connections between this platform and other interventions, and explore client experiences over time.

Targeting Social Isolation

Parkinson’s Disease (PD) is a neurodegenerative condition with no cure to date. Most often diagnosed in people at around age 60 or older, PD manifests as a movement disorder affecting balance, ability to initiate voluntary movement like walking and talking, and a host of other symptoms including fatigue and depression. As the disease progresses, people with PD tend to isolate themselves due to concerns that they might, for example, fall and get injured while out, have difficulty communicating with others, or that the outing would be too exhausting. Isolation is familiar for people with PD.

Research tells us that living in isolation (outside of this pandemic) is associated with poorer health outcomes – physical, cognitive, mental, and emotional. Conversely, individuals with greater social support (e.g., close family and friends around) tend to have better health outcomes and tend to recover more quickly from illness, injury, and surgery. In the UK the Campaign to End Loneliness provides examples from research of how detrimental isolation can be for health: “Loneliness, living alone and poor social connections are as bad for your health as smoking 15 cigarettes a day. (Holt-Lunstad, 2010) … Lonely people are more likely to suffer from dementia, heart disease and depression. (Valtorta et al, 2016) (James et al, 2011) (Cacioppo et al, 2006).” In addition, two years ago the UK government appointed a Minister of Loneliness. Aware of the negative aspects of loneliness, one of DWP’s core missions has been to bring people with PD out of isolation. In the unique context of COVID-19 where isolation is associated with better health outcomes, many of us are experiencing isolation at home, regardless of our demographic, socio-economic, geographic, or health profile. But long before COVID-19 necessitated isolation, people with PD and large numbers of seniors were already experiencing isolation. An evaluation of DWP can offer important learnings about the management of isolation. How, for example, can the “positives” of isolation be amplified and the negatives of isolation dampened?

It becomes important to evaluate this program for multiple reasons:

Imagining and Conceptualizing the Intervention

The innovation as an organizing device. One of the challenges of living with the pandemic, or a disease like PD that pushes one toward isolation, is that time can become blurred. The DWP online dance class is offered daily for 20 minutes. One area that may be valuable to explore is whether and how this innovation helps organize the day for the participants. One mechanism by which the online dance class can benefit participants is as an organizing device that helps bring structure to their daily activities. Exploring this mechanism will help understand one aspect of how an online platform can benefit participants. What can we learn about a brief online class as a structuring device to help participants organize the day?

Displacement from social isolation: Building community. The second mechanism by which an online class can potentially work is that it can serve to address issues of loneliness while simultaneously building social connections to community. On any given day, the class might have 70 or more participants. One of the features of the online class is that for 5-10 minutes before and after the actual dance portion of the class, there is time open for dialogue with the instructor and class participants. The vast majority of participants take advantage of this opportunity to dialogue and build connections with others. When we consider the mechanism of addressing social isolation and building community through online platforms it is important to recognize that individuals without access to digital devices, such as computers, will be excluded from such communities and may feel further isolated as a result. Acknowledging this as a critical issue, Sarah and her team are currently investigating potential options for providing digital devices to those in need. Important evaluation questions include: How does the online environment build connections between the participants? Can the platform be further enhanced to build community? How can the online intervention include the most marginalized individuals?

Building theories of change across different settings. One of the deeper aspects of dance as an intervention for PD is that a comprehensive theory by which dancing can impact the health of Parkinson’s patients is unclear. While some mechanisms about how dance can provide benefits are known, other mechanisms, such as the role of building social connections and reducing isolation, have yet to be fully explored. Further, it remains an open question as to how dance can best be combined with other neurological interventions. The issue is not dance vs. neurological interventions; rather, the question is: What would a comprehensive package of dance alongside other interventions look like? This web-based class provides an opportunity to explore the interactions of mechanisms in a very different context from the in-person delivery. Following a realist evaluation frame, we believe that there is value in exploring feedback from multiple participants (at different stages of the disease, with different abilities, in varying contexts of living arrangements) regarding how different mechanisms might work. There is also a need to better understand the extent to which such a modified package of innovations could be successful given that the mechanisms of delivering the dance class online miss some key aspects, such as touch – hugs, for example, are often shared between teachers/volunteers and participants, physical contact is made during partnership dances like the Waltz and Tango, gentle corrections to posture and alignment are made through touch during the class, and participants hold hands in the final closing circle. During such moments of touch, the hormone oxytocin is released and is associated with feelings of trust, bonding, social connection. What would a theory of change that explores multiple mechanisms by which dance can impact brain health look like? In what ways, does DWP’s original theory of change differ when using this very different delivery platform in the context of COVID-19?

Emergent possibilities. The present innovation provides an opportunity to define what success means from the perspective of participants with PD. This is a deep issue given that PD is a neurodegenerative condition and the expected trajectory of the disease leads towards a decline in one’s ability to perform the activities of daily living as well as a decline in health outcomes more broadly. As such, success is not easily defined even in the best of circumstances. Given that these online classes were designed in response to a crisis situation, participants themselves may be discovering how they view the success of such an innovation. There are multiple emergent possibilities that might happen as such an innovation unfolds. Some of the unfolding discovery incorporates the possibility that even after the intensity of the pandemic ends and in-person classes resume there will be a need to provide a version of such an online class. How can evaluation help map out how clients view success in a more dynamic way?

The importance of aesthetics: One of the big gaps in the literature involves understanding the role of the aesthetics of dance, art, and music, as well as how the arts can be used as interventions to improve health. In a beautiful article titled “Feeling Lovely: An Examination of the Value of Beauty for People Dancing with Parkinson’s”, Sara Houston has argued for the value added of aesthetics through creating a mechanism of “feeling lovely.” The DWP evaluation needs to explore whether the aesthetics still come through in the online platform in the context of participants’ own living rooms. Exploring the linkage between aesthetics and health and the possibilities of delivering dance online at a large scale has enormous ramifications for how we can think about the ‘solutions space’ of dance-based interventions for brain disorders. How does one stay true to principles of aesthetics even in an online platform? How does one establish the impacts of aesthetics on health? Does such a linkage exist?

The Role of Developmental Evaluation

The innovation as a developmental process. From a developmental evaluation frame, like a number of disruptions caused by the pandemic, it’s important to see this online dance intervention as a developmental process. As with any innovation, there is uncertainty facing the sustained delivery of this online dance innovation. The evaluation provides an opportunity to learn and help develop the intervention further. What needs further development? How can the evaluation help with such development?

Principles that guide the innovation. Related also to a developmental evaluation process, there is a need to explore basic principles to inform delivery of such an intervention. What are key principles that guide the delivery? Are these principles static? What do different clients at different stages of the disease want from the class, and what are the principles that drive them?

Understanding Experiences and Scaling-up

Understanding experiences and impacts over time. Another important focus of the evaluation is to explore the impact of this innovation. In what ways are the clients being impacted? How do client experiences change over time? In addition to impacts, it is important to explore the dynamics of such experiences. For example, it is possible that such classes are exciting initially but that the enthusiasm wanes over time; however, it is equally possible that enthusiasm for the classes grow over time and they become part of the daily regimen of the clients. What are the impacts of such an online platform on clients? Ho do client experiences of the online platform change over time?

Scaling Up. One of the great benefits of the online model of delivery is that it might be relatively easier to scale than a face-to-face delivery model. The evaluation needs to explore the kinds of constraints that need to be addressed to scale up such a dance-based intervention widely. Over the last few weeks alone, the participations have moved from 15 to 400+. How can the evaluation help with the process of scaling up?

Looking Ahead: The Need for Stories

As we enter a brave new world of both constraints on past ways of doing things and emerging possibilities with newer platforms, evaluation has a role to play to help further develop such innovations and also explore some of the initial impacts of such platforms. The online dance intervention that Sarah and her team has developed is both a labour of love and also a creative solution that pays careful attention to the multiple realities and constraints of teaching online. Teaching online will come with a diminished role of key mechanisms, like touch, but it may have other compensatory mechanisms. The evaluation provides an opportunity to document how this online platform evolves and also relay the stories from clients about how this platform makes a difference in their lives.

Adapting a Dance Intervention in the Time of COVID-19

14 May 2020 / Sarah Robichaud

Dancing With Parkinson’s (DWP) is a Toronto-based not-for-profit that has been designing and delivering specialized dance classes for people with Parkinson’s disease (PD) since 2008. It is a program that is based on providing in-person dance classes. Until recently, it was offering 15 weekly dance classes throughout the Greater Toronto Area (GTA). But on March 12, 2020 DWP had to suspend all of its dance classes due to COVID-19.

As the Founder and Executive Director of DWP, I was devastated. My mind was actually quite paralyzed for the first week or so after closing our doors. To be honest, I thought this meant the end of DWP as an organization. Part of this was based on my assumptions about the program; I truly believed that the program could only be effective if it was delivered in person. Key ingredients in our model, such as touch, for example, simply could not be implemented if we were unable to meet with participants face to face. The other part was practical. DWP has very limited operating costs and relies heavily on its annual dance-a-thon fundraiser held every Spring as well as regular class fees. So, I really thought the pandemic signaled the end of DWP.

Once the shock had subsided a bit, I still didn’t have a clear idea of what to do. But I knew one thing: I didn’t want to give up on the core values and mission that had driven me to create DWP in the first place – a belief that bringing seniors out of isolation and creating a safe, welcoming space for connection through dance could make a difference in their lives. I also knew that the pandemic and the social distancing measures put in place to stop it could leave seniors even more isolated than ever. I wanted to find a way, really quickly, to keep our seniors connected.

This line of thinking led me to pilot short online dance classes for our existing DWP community. The online classes were so well-received that I quickly began offering 20-minute online dance classes seven days per week. Class participants started asking me if they could share the class link with their sister, brother, and friends. My friends asked if their mom, dad, aunt, or uncle could join. I soon realized that all seniors, not just those with PD, could benefit from these classes, especially at a time when so many people were feeling the physical and emotional strains of social isolation. There are now more than 400 seniors on our online class roster and these participants login from all across the country. We even have some participants from outside of Canada. The participant numbers are growing daily.

Challenges of Adapting to Online Delivery

Have there been challenges? Sure. For one, I am not very technologically inclined. So, this has been a bit of a learning curve for me. I am still working on improving the quality of the classes in terms of sound and picture, for example. There have also been a handful of seniors that needed some support in setting up and using the online platform, so we walked them through this offline. But, for the most part, participants have accessed the online classes fairly seamlessly. In fact, some seniors have even shared tips with me about specific features of the technology! I think the biggest challenge and the worst part of adapting the program to an online delivery method is that some seniors don’t have access to the technology needed to join the classes – and these are the people who most likely would benefit the most. This is something that needs to be addressed.

Some Surprises and Learnings

Innovating the program in this way has been surprising on a number of fronts. First of all, I am surprised that seniors are still able to make meaningful connections with each other and feel energized, uplifted, physically conditioned, hopeful, and so on. Many of the same outcomes we tried to bring about before in our in-person classes are happening now through the online technology. Do I think touch is still important? Yes. And I will be super excited to give everyone a big hug when we are allowed to resume our in-person dance classes. But maybe it is not the primary way of connecting people; there are other ways.

Second, I have received initial feedback that some seniors find the online classes meet their needs better than the in-person classes. This goes beyond the convenience aspect of the online method, which is not terribly surprising (for example, it makes sense that seniors with PD would find it easier to attend a class in their own homes rather than organizing transportation and spending the time and effort to make it to a geographically distant class location). For the online classes, I spotlight video myself, which makes me really big on the screen. This may make the class feel more intimate or one-on-one in nature. Some participants say that this helps them focus better on the class exercises, as they are not distracted by the movements of others in their peripheral vision. That participants could actually benefit more from the online version of the class came as a big surprise to me. This is a pretty deep learning.

Finally, I am shocked by how much this is touching me – how emotionally connected I feel to all of the dance participants and the new friends I am making. I am meeting new seniors all the time and quickly feeling very attached to them. There is something very raw and human about this whole experience. This fits with DWP’s philosophy – come as who you are in the present moment. We accept you and you accept us. People are being so vulnerable and open and this is a very scary and uncertain time. But DWP is still able to create this safe and joyful place to go. I am excited about the fact that DWP has a life beyond what the plan was. I am excited to be innovating again and making some new plans that might end up impacting more people than we ever thought we could.

Adaptations and Nimbleness at the Time of Crisis: Some Questions for Evaluators

5 May 2020 / Sanjeev Sridharan

Adaptiveness as a Clunky Dance

The pandemic provides a reminder of how dynamic the world can be. It continues to surprise and challenge the creativity of even the most experienced program implementer working in community settings. In almost all of the public health interventions I know, the pandemic has forced a re-think of the implementation plans. The pandemic also has posed critical challenges for how evaluation as a field can be adaptive and nimble when faced with flux and the need to drastically change the playbook. The pandemic provides an opportunity for us to ask ourselves how evaluations can be adapted to be helpful at a time of crisis. Over the past month, I have had a chance to connect online with multiple implementers working both in community and policy settings. Despite the severity of the crisis, the creativity of many organizations in adapting to the pandemic has been striking.

One program implementer refers to her organization’s adaptiveness as a “clunky dance.” Another individual refers to ‘muddling towards an authentic response.’ This blog is written from the perspective that evaluators have a chance to learn from our implementation partners’ nimbleness and also incorporate their adaptiveness into the frameworks we use to value interventions. Given the challenges some community organizations are experiencing, evaluators might have a role to play in creating an enabling environment by raising questions around changing needs and the need for systemic coordinated responses.

Some Questions

Based on multiple dialogues with policy and community partners, I raise six sets of questions:

1. Narratives of programmatic adaptations		4. Re-thinking theories of systemic change
2. Measuring nimbleness and adaptiveness		5. Minimal components needed for an intervention to work
3. Deeper learnings about strengthening systems		6. Intersectionalities and cracks in the network

1. Narratives of programmatic adaptations

Most interventions have had to adapt during the pandemic. One somewhat straightforward area in which evaluation can be responsive is to highlight through simple, succinct narratives the adaptations that have occurred, the drivers of such adaptations, and if these adaptations were successful in responding to the emerging crisis. A number of the programs I am aware of are already documenting such adaptations. This can be both straightforward and challenging; but it requires coordination between program and evaluation teams. Attention needs to be paid to the constraints facing the programs, the timeframes the programs had to make the adaptations, the doors that were closed on the earlier service delivery platforms as a result of the pandemic and how new platforms were created to still provide the services. Evaluation teams need to leverage such narratives and help programs tell their stories systematically. What are exemplars of good evaluation stories of the adaptiveness/nimbleness of specific interventions?

2. Measuring nimbleness and adaptiveness

Few evaluation frameworks include adaptiveness and nimbleness as criteria to judge the success of interventions. As an example, it is interesting that the newly revised DAC evaluation criteria (by the OECD’s Development Assistance Committee; https://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm does not focus on either nimbleness or adaptiveness as desired attributes of interventions and criteria for evaluation.

Here is my attempt at defining these terms. Adaptiveness can be defined as the extent to which an intervention morphs to respond to changes in its context. In my judgement, adaptiveness is related to the contextual awareness of an intervention. Nimbleness can be defined as the agility with which an intervention responds to changes in perceived needs; there is a temporal aspect to nimbleness. In my judgment, based on a quick scan of the literature, neither of these constructs has been a key focus of evaluation frameworks. The pandemic provides an opportunity to learn from how programs have both conceptualized and operationalized adaptiveness and nimbleness. The pandemic reminds us that the needs of individuals are dynamic, and the opportunity for community, food, social protection, and health systems to respond to such needs are constrained by multiple factors. I have found dialogues with community organizations helpful in surfacing critical ideas on how we can measure adaptiveness and nimbleness. I raise this issue because evaluators have an opportunity to elevate the salience of adaptiveness and nimbleness as important criteria in evaluating interventions. What are innovative examples of measures of adaptiveness/nimbleness of programmatic responses during the pandemic

3. Deeper learnings about strengthening systems

Interventions are embedded within broader systems. No intervention is an island. This pandemic has highlighted the need to better understand the connections between the intervention and its underlying systemic contexts/supportive structures. One of the most striking insights arising from dialogues with policy and community partners has been that this crisis can sometimes potentially accelerate systems and coordinated responses to health and social problems. There is a literature around crisis-driven learning: https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1468-5973.2009.00578.x

Some stakeholders have mentioned that there has been a greater sharpening of focus of the systemic response during this crisis. Others have noted that the coordination between different organizations has actually been enhanced during this pandemic. However, other organizations have described how the crisis has paralyzed their ability to respond to needs and their limited capacities to coordinate with other organizations. My view is that evaluation as a field has a role to play in exploring if and how crisis-focused coordination can be enhanced. Even in the happy situation where coordination has been enhanced, evaluation can still play a role by exploring what is likely to happen when the intensity of the crisis recedes. Are there examples of evaluations that have taken a developmental approach to enhance coordination between organizations at this time of crisis?

4. Re-thinking theories of systemic change

Many stakeholders also conjectured that the crisis has helped them re-imagine what is possible with system-level efforts. A number of community organizations are beginning to re-think their own theories of change, the role of different partners and how their organization relates to the overall system. In some settings, the pandemic has served to highlight and build support for interventions focused on poverty and social isolation in a way that was not there before the pandemic. It may be argued that responding to the pandemic has provided a greater understanding of the needs of the vulnerable and a deeper lived experience of what social isolation means. If this is the case, what does this suggest for refining interventions that are focused on poverty and social isolation? Are there examples of intervention-level theories of change that have been refined as a response to adapting to the pandemic?

5. Minimal components needed for an intervention to work

Evaluations often focus on what an optimal set of components is needed for an intervention to produce favorable impacts. We rarely focus on the minimal set of components that are required for an intervention to function. An important insight from a few community organizations has been that this pandemic has taught them lessons about how to simplify their programs; it has also provided insights on a minimal set of components required to meet the needs of individuals, especially the most disadvantaged. In the Behavioral Medicine literature, there is the concept of MINC (minimum intervention needed to produce change; https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3958586/). Once the intensity of COVID-19 has abated, a focus on the minimal set of components needed might help with thinking about our social programs in a richer light, especially if the focus is on using limited resources to address the needs of disadvantaged stakeholders. Can a focus on a minimal set of components needed to produce change help enhance a focus on meeting the needs of the disadvantaged given limited resources?

6. Intersectionalities and cracks in the network

One consistent feedback from a number of community organizations is that COVID-19 has helped surface or highlight key cracks in the existing “networks of care”. There is a need to pay attention to individuals who fall through such cracks and also focus on individuals who fall in the ‘intersections’ of multiple categories of deprivation. A number of interventions have adapted to respond to such intersectionalities and perceived cracks in the network. Over time, there is a need to re-think how our strategies address the needs of individuals who fall through such cracks. While many interventions focus on the needs of disadvantaged individuals/groups, in my experience, the theories of change rarely incorporate (in a sharp focus) the mechanisms by which the interventions can address the needs of such individuals who fall in such intersections.

Similarly, there is also a need to move beyond static definitions of vulnerability towards paying attention to the dynamics of vulnerability. (https://www.ncbi.nlm.nih.gov/pubmed/23549696). The issue is not just that the interventions meet the needs of the vulnerable, but additionally, that they help disrupt the generative mechanisms that COVID-19 might have amplified (https://www.brookings.edu/blog/future-development/2017/06/19/pandemics-and-the-poor/). Now, more than ever, there is a need to pay attention to syndemic (https://www.thelancet.com/series/syndemics) processes between poverty and social problems like homelessness that might exacerbate problems over time. Some important evaluation questions to consider are: Do the proposed solutions incorporate the lived realities of individuals who fall through systems cracks? Are the systems, structures and processes that are being set up to address the needs of such individuals consistent with the needs and expectations of the clients whose needs are intended to be served? Does the proposed solution pay attention to the dynamics of vulnerability that might be especially acute for marginalized individuals during and post-pandemic?

Looking Ahead

As we go forward, one ethical principle that should guide our work is that measurement and valuing should not interfere with the programmatic response during this crisis.

The above set of issues should be seen as questions, opportunities and challenges for the field of evaluation. In my view, much of the recent measurement work during this time of crisis has surfaced the problem space of COVID-19. The pandemic’s impacts on individuals, especially on highly vulnerable individuals, and such a problem space have also helped raise deep questions about how our existing systems of care might not be adequate. However, accompanying this crisis-driven problem space there has been a creative response to address as best as possible within existing constraints how programs have had to adapt in a timely manner to respond to the growing sets of needs. As a field, evaluation needs to help tell the story of such emergent solutions. Such a focus on the solution space (https://www.sciencedirect.com/science/article/abs/pii/S0149718912000213) can help grow the salience of evaluations as a useful tool in bridging problems and solutions during times of crisis.

Innovations and adaptations during crisis times

23 April 2020 / April Nakaima

The Evaluation Centre for Complex Health Interventions was founded with a philosophy that evaluations of interventions need to adapt to the complexities driven both by changes in the intervention over time as well as changes across contexts. The pandemic provides remarkable challenges for both the program implementation and evaluation communities to be adaptive and responsive given unparalleled levels of disruption. This website is intended to serve as a space for blogs and other thought pieces by program implementers, policy makers, community workers, and evaluators to reflect on three aspects:

How have your interventions adapted in response to the pandemic?
How have your evaluations needed to adapt given the disruptions in the intervention?
How has what you VALUE as an eVALUator changed given the pandemic?