5 MAY 2020 / SANJEEV SRIDHARAN

Adaptiveness as a Clunky Dance

The pandemic provides a reminder of how dynamic the world can be.  It continues to surprise and challenge the creativity of even the most experienced program implementer working in community settings.  In almost all of the public health interventions I know, the pandemic has forced a re-think of the implementation plans.  The pandemic also has posed critical challenges for how evaluation as a field can be adaptive and nimble when faced with flux and the need to drastically change the playbook.  The pandemic provides an opportunity for us to ask ourselves how evaluations can be adapted to be helpful at a time of crisis.  Over the past month, I have had a chance to connect online with multiple implementers working both in community and policy settings.  Despite the severity of the crisis, the creativity of many organizations in adapting to the pandemic has been striking.

One program implementer refers to her organization’s adaptiveness as a “clunky dance.” Another individual refers to ‘muddling towards an authentic response.’  This blog is written from the perspective that evaluators have a chance to learn from our implementation partners’ nimbleness and also incorporate their adaptiveness into the frameworks we use to value interventions.  Given the challenges some community organizations are experiencing, evaluators might have a role to play in creating an enabling environment by raising questions around changing needs and the need for systemic coordinated responses.

Some Questions

Based on multiple dialogues with policy and community partners, I raise six sets of questions:

1. Narratives of programmatic adaptations		4. Re-thinking theories of systemic change
2. Measuring nimbleness and adaptiveness		5. Minimal components needed for an intervention to work
3. Deeper learnings about strengthening systems		6. Intersectionalities and cracks in the network

1. Narratives of programmatic adaptations

Most interventions have had to adapt during the pandemic.  One somewhat straightforward area in which evaluation can be responsive is to highlight through simple, succinct narratives the adaptations that have occurred, the drivers of such adaptations, and if these adaptations were successful in responding to the emerging crisis.  A number of the programs I am aware of are already documenting such adaptations.  This can be both straightforward and challenging; but it requires coordination between program and evaluation teams.  Attention needs to be paid to the constraints facing the programs, the timeframes the programs had to make the adaptations, the doors that were closed on the earlier service delivery platforms as a result of the pandemic and how new platforms were created to still provide the services.  Evaluation teams need to leverage such narratives and help programs tell their stories systematically.  What are exemplars of good evaluation stories of the adaptiveness/nimbleness of specific interventions?

2. Measuring nimbleness and adaptiveness  

Few evaluation frameworks include adaptiveness and nimbleness as criteria to judge the success of interventions.  As an example, it is interesting that the newly revised DAC evaluation criteria (by the OECD’s Development Assistance Committee; https://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm does not focus on either nimbleness or adaptiveness as desired attributes of interventions and criteria for evaluation.

Here is my attempt at defining these terms.  Adaptiveness can be defined as the extent to which an intervention morphs to respond to changes in its context.  In my judgement, adaptiveness is related to the contextual awareness of an intervention.  Nimbleness can be defined as the agility with which an intervention responds to changes in perceived needs; there is a temporal aspect to nimbleness.  In my judgment, based on a quick scan of the literature, neither of these constructs has been a key focus of evaluation frameworks. The pandemic provides an opportunity to learn from how programs have both conceptualized and operationalized adaptiveness and nimbleness.   The pandemic reminds us that the needs of individuals are dynamic, and the opportunity for community, food, social protection, and health systems to respond to such needs are constrained by multiple factors. I have found dialogues with community organizations helpful in surfacing critical ideas on how we can measure adaptiveness and nimbleness. I raise this issue because evaluators have an opportunity to elevate the salience of adaptiveness and nimbleness as important criteria in evaluating interventions.  What are innovative examples of measures of adaptiveness/nimbleness of programmatic responses during the pandemic

3. Deeper learnings about strengthening systems  

Interventions are embedded within broader systems.  No intervention is an island.  This pandemic has highlighted the need to better understand the connections between the intervention and its underlying systemic contexts/supportive structures.  One of the most striking insights arising from dialogues with policy and community partners has been that this crisis can sometimes potentially accelerate systems and coordinated responses to health and social problems.  There is a literature around crisis-driven learning: https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1468-5973.2009.00578.x

Some stakeholders have mentioned that there has been a greater sharpening of focus of the systemic response during this crisis.  Others have noted that the coordination between different organizations has actually been enhanced during this pandemic. However, other organizations have described how the crisis has paralyzed their ability to respond to needs and their limited capacities to coordinate with other organizations. My view is that evaluation as a field has a role to play in exploring if and how crisis-focused coordination can be enhanced.  Even in the happy situation where coordination has been enhanced, evaluation can still play a role by exploring what is likely to happen when the intensity of the crisis recedes.  Are there examples of evaluations that have taken a developmental approach to enhance coordination between organizations at this time of crisis?

4. Re-thinking theories of systemic change

Many stakeholders also conjectured that the crisis has helped them re-imagine what is possible with system-level efforts.  A number of community organizations are beginning to re-think their own theories of change, the role of different partners and how their organization relates to the overall system.  In some settings, the pandemic has served to highlight and build support for interventions focused on poverty and social isolation in a way that was not there before the pandemic. It may be argued that responding to the pandemic has provided a greater understanding of the needs of the vulnerable and a deeper lived experience of what social isolation means.  If this is the case, what does this suggest for refining interventions that are focused on poverty and social isolation? Are there examples of intervention-level theories of change that have been refined as a response to adapting to the pandemic?

5. Minimal components needed for an intervention to work

Evaluations often focus on what an optimal set of components is needed for an intervention to produce favorable impacts.  We rarely focus on the minimal set of components that are required for an intervention to function.  An important insight from a few community organizations has been that this pandemic has taught them lessons about how to simplify their programs; it has also provided insights on a minimal set of components required to meet the needs of individuals, especially the most disadvantaged.  In the Behavioral Medicine literature, there is the concept of MINC (minimum intervention needed to produce change; https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3958586/).  Once the intensity of COVID-19 has abated, a focus on the minimal set of components needed might help with thinking about our social programs in a richer light, especially if the focus is on using limited resources to address the needs of disadvantaged stakeholders.  Can a focus on a minimal set of components needed to produce change help enhance a focus on meeting the needs of the disadvantaged given limited resources?

6. Intersectionalities and cracks in the network

One consistent feedback from a number of community organizations is that COVID-19 has helped surface or highlight key cracks in the existing “networks of care”.  There is a need to pay attention to individuals who fall through such cracks and also focus on individuals who fall in the ‘intersections’ of multiple categories of deprivation.  A number of interventions have adapted to respond to such intersectionalities and perceived cracks in the network.  Over time, there is a need to re-think how our strategies address the needs of individuals who fall through such cracks.  While many interventions focus on the needs of disadvantaged individuals/groups, in my experience, the theories of change rarely incorporate (in a sharp focus) the mechanisms by which the interventions can address the needs of such individuals who fall in such intersections.

Similarly, there is also a need to move beyond static definitions of vulnerability towards paying attention to the dynamics of vulnerability. (https://www.ncbi.nlm.nih.gov/pubmed/23549696).  The issue is not just that the interventions meet the needs of the vulnerable, but additionally, that they help disrupt the generative mechanisms that COVID-19 might have amplified (https://www.brookings.edu/blog/future-development/2017/06/19/pandemics-and-the-poor/).  Now, more than ever, there is a need to pay attention to syndemic (https://www.thelancet.com/series/syndemics) processes between poverty and social problems like homelessness that might exacerbate problems over time.  Some important evaluation questions to consider are: Do the proposed solutions incorporate the lived realities of individuals who fall through systems cracks?  Are the systems, structures and processes that are being set up to address the needs of such individuals consistent with the needs and expectations of the clients whose needs are intended to be served?  Does the proposed solution pay attention to the dynamics of vulnerability that might be especially acute for marginalized individuals during and post-pandemic?

Looking Ahead

As we go forward, one ethical principle that should guide our work is that measurement and valuing should not interfere with the programmatic response during this crisis.

The above set of issues should be seen as questions, opportunities and challenges for the field of evaluation.  In my view, much of the recent measurement work during this time of crisis has surfaced the problem space of COVID-19.  The pandemic’s impacts on individuals, especially on highly vulnerable individuals, and such a problem space have also helped raise deep questions about how our existing systems of care might not be adequate.  However, accompanying this crisis-driven problem space there has been a creative response to address as best as possible within existing constraints how programs have had to adapt in a timely manner to respond to the growing sets of needs.  As a field, evaluation needs to help tell the story of such emergent solutions.  Such a focus on the solution space  (https://www.sciencedirect.com/science/article/abs/pii/S0149718912000213) can help grow the salience of evaluations as a useful tool in bridging problems and solutions during times of crisis.

Staff of the Evaluation Centre in Collaboration with Our Partners at Margaret’s Housing and Community Support Services and other Community Organizations
25 July 2020 / APRIL NAKAIMA AND RACHAEL GIBSON

As part of an evaluation that the Evaluation Centre is conducting with Margaret’s Housing and Community Support Services, and through speaking to other providers of care in the community sector serving marginalized individuals, a number of questions has surfaced during this pandemic that can help with improving systems to create solutioning for individuals who face multiple disadvantages at the intersection of housing and mental health. The Evaluation Centre is presently working on a full-fledged report that discusses the implications of these questions for both understanding problems and generating solutions.

Some of these questions will need to be addressed by conducting summative evaluation approaches in the near future. However, most of these questions will require developmental evaluation approaches that need to be co-constructed between the organizations working to provide an ecology of care for individuals who face co-occurring conditions of homelessness, mental illness, and other issues such as substance use, gender identity discrimination, racial discrimination, and disabilities.

1. Understanding syndemic processes:  An important issue that surfaced in our dialogues with multiple community stakeholders was the conceptual need to better understand mutually reinforcing vicious cycles that might connect chronic homelessness to poor health. The pandemic might further serve to exacerbate these connections by both reducing capacities of community organizations as well as by worsening health conditions for marginalized individuals. How will an understanding of the syndemic processes of chronic homelessness inform better planning for the ongoing and future pandemics?
2. Learning from successful drop-in/respite sites: Not all Drop-in/Respite sites have seen problems during the pandemic. Some have had outbreaks; others have had no cases. Can a comparative analysis of such facilities help us better understand factors associated with successful responses to the pandemic?
3. More deeply understanding the role of community organizations  in preventing and responding to the pandemic: In our multiple dialogues what has clearly surfaced is the great need to better value the role of community organizations in an ecology of care in addressing the problems of chronic homelessness and persistent mental health challenges. What role does research and evaluation play in creating better awareness of the value of community organizations in such an ecology of care?   
4. Equity in distribution of PPE. A related set of basics questions that emerged from multiple dialogues with community organizations has included a focus on equity in distribution of PPE and testing kits: How can there be a clearer prioritization of the importance of community organizations in responding to the pandemic? What are more equitable ways of distributing testing kits, PPE, etc.?
5. Managing fatigue:  A number of organizations working on homelessness are  already overstretched, under-funded, and even during normal times the staff are potentially over-worked and stressed. Combined with the fear and the potential chaotic disruption caused by the pandemic, important questions for both systems and organization leaders working on chronic homelessness include: How does one manage fatigue/stress among frontline staff addressing homelessness given the long hours and extreme stress that they are facing during the pandemic? Going forward how does one build surplus capacity in community organizations given the potential organizational disruptions caused by the pandemic?
6. Leveraging political and social opportunities to find solutions to chronic homelessness:  Stakeholders also noted that the pandemic also has created a political climate in which key actors who perhaps may not have been cognitively and politically sensitive to recognizing the need for multi-sector approaches to chronic homelessness are becoming more understanding and supportive of the interconnections between social and economic factors that drive chronic homelessness. Stakeholders noted that such favourable changes in a recognition of the multilevel complex drivers of chronic homelessness can be seen in the three levels of government. Can more be done to leverage such changes in political climate to create policy and practice solutions that are aligned across the three levels of government?
7. Generating client-sensitive solutions:  Another point that surfaced in our multiple dialogues was the need to respond to the pandemic in ways that recognizes the lived realities of clients. For example, providing hotel rooms to isolate individuals who have tested positive or are awaiting test results might not work for some individuals whose mental health may de-compensate in such an environment. How do we think more systemically about solutions that are more client-sensitive and especially acknowledge challenges of anxiety and depression among some marginalized populations?
8. Having better established communication protocols:  A point raised by multiple community stakeholders in the initial stages of the pandemic was the need to have clearer protocols/guidelines when faced with the disruptions caused by the pandemic. The lack of clear protocols/guidelines was deeply problematic for at least a few clients. Some of the issues that were raised in terms of a lack of communication included: lack of clarity on how long testing would take, lack of clarity of where individuals needed to be housed or stop-gap solutions that respected client contexts and client choices. What are processes going forward to develop better communication protocols between the hospitals/public health sector and the community sector?
9. Creating systems of coordinating and supporting community care:  Another point that surfaced in multiple discussions was the experience of some community support and housing organizations that felt very much alone as they responded to the pandemic. Some of the issues that surfaced pointed to a lack of system coordination including: the need for more thoughtful systems of alerts, the need for data sharing, and the need for clearer protocols and guidelines mentioned earlier. Some of the stakeholders we spoke to did not feel supported in their efforts to thwart the pandemic or felt that there was not a system of care to support them. How does one create a more caring, coordinated system of care in which organizations don’t feel alone in responding to a pandemic?
10. Learning solutions from the pandemic:  Some stakeholders we spoke to also spoke about some of the lessons that they have learned about the power of virtual interaction for some clients. For example, some clients have accessed more time with their psychiatrist virtually and have preferred the online interactions to their previous face-to-face interactions. Going forward, are there hybrid models in which face-to-face solutions can be combined with online solutions for some clients?

We believe that addressing the above points both through individual evaluations of community care organizations’ experiences and through generating solutions systemically perhaps through a systems-focused developmental evaluation process might be needed. It’s important we do not treat the issues raised above as just realities affecting one organization; in our work evaluating multiple community interventions during the pandemic and dialogue through the webinars that the Evaluation Centre have organized, we hear very similar concerns being raised by community organizations in countries like Chile, the UK, and India. Any solutions that we arrive at within Ontario will in all likelihood have relevance in other settings.  

Diane Walter, Executive Director, Margaret's Housing and Community Support Services

Margaret’s Housing and Community Support Services is a multi-service agency that provides a continuum of housing options and wrap-around supports to women living with mental illness and substance use, as well as community-based support services including several drop-in centres and respite services in Toronto for individuals experiencing intersectional issues tied to mental health challenges and homelessness

The Evaluation Centre posed a number of questions for me to reflect on. I share some brief reflections below. My hope is that such reflections from both me and others on my team and partners at the Evaluation Centre lead to systemic learning and system-level solutions in the future.

At such a difficult time, what has kept you going?

What keeps me going more than anything else since the pandemic is the knowledge that some of the folks that we’re serving really have no options. Recognizing that we can make a difference in someone’s life by making sure that a staff member delivers some groceries, or there’s a wellness check-in, for example, keeps me going.  Also, recognizing that although I’m a minor cog, we might still be helpful during this pandemic in somebody’s life, this gives me energy even when I don’t have energy.

What have you learned during the pandemic?

The generosity of my staff, some partners and Torontonians
The generosity of Torontonians with donations of cash and PPE and both my team and our partners who have worked closely and well inspires me. What I have discovered is there are many staff who weren’t stars necessarily before the pandemic; they did their jobs adequately and met their deliverables; but during the pandemic they have shone and seem to thrive in the chaos – you can call them at any time, and they just step up and deliver in spades. 

Leadership stepping up to the plate
Some leaders in Toronto and Ontario have pleasantly surprised me. I’ve seen some leaders during the pandemic lead with compassion and empathy. I have also grown to respect some leaders’ ability to communicate to the public in a near daily basis that they have the backs of Ontarians, whatever the challenges are.  

This pandemic spotlights the need for housing as a solution
We can only hope that this pandemic is galvanizing all three levels of government in the need for housing.  I hope the government can do an accounting to see if the pandemic has cost them more than if they had addressed the homeless situation 4-5 years ago. For example, the City has opened up multiple hotel programs — rented hotel rooms to put homeless people who have tested positive or are waiting on results and those from encampments. If they had funneled those funds into building the real bricks and mortar of housing, I think things would have been significantly different. Had housing been thought about in a more deliberate solution-based fashion, this pandemic would not have been so costly.

The Importance of Prevention: Reimagining health care
I think most health care is delivered in the community, but hospitals are the recipients of the bulk of our healthcare dollars. If we were to re-imagine what a healthcare system is — not an acute care system or a sickness system, but a healthcare system — where more people in the community are trained to do simple tasks so that folks would not end up in the hospital system, given the fact that when people end up in the hospital it’s often because all the prevention and all the incremental interventions did not occur to prevent them from ending up in an expensive hospital bed, They could have been treated in the community had there been adequate prevention and adequate interventions, even something simple before prevention.

What recommendations do you have in response to this pandemic?

We have to get ready for the next wave. Historically pandemics often have a second wave. Some questions we as a system need to answer: Is it going to be worse than the first one? Is it going to be more decimating? We don’t know. But for us, we certainly have learned some things that we’re preparing ourselves should there be a second wave. For example, we’re making sure that our inventory of PPEs will take us into and perhaps beyond the second wave; some of the protocols around IPAC (Infection Prevention and Control) we are now making sure they’re ramped up, and we’re aggressively following those infection IPAC standards.

There are learnings around how a pandemic like COVID-19 could so comprehensively affect how we operate. For example, within the drop-in respite setting, we were shifting from using Styrofoam to reusable utensils because we wanted to reduce our footprint. We were really gung ho about going Green, but now that is not on the agenda for awhile; it’s totally off the table. We’re back to Styrofoam because clients are more likely to mistakenly pick up and use a reusable utensil or coffee cup that another client has used rather than one that gets thrown out. We’re looking into whether we can use other materials, but the other materials are so expensive.

We can learn some things from this pandemic that are actually positive. In terms of guidelines, we are adhering to those around PPE and making sure staff are spaced — it will affect some of the ways we work in terms of front-facing, client-facing service delivery. But we are seeing the value of virtual interaction – however, we haven’t gotten to that place where we really take a deep dive and assess it. These are things that are just coming up, and where we are thinking, “That we can amend. This we can modify.” For example, we can integrate the virtual piece into our work seeing clients one-on-one. The nurse that we use and the psychiatrist have realized that some people prefer the virtual interaction with the psychiatrist than the one-on-one

The need for learning and developing system-level solutions
My view is that what we desperately need is to learn from the experiences of community organizations providing care during this time of the pandemic. Evaluation teams like the Evaluation Centre have a role in precipitating dialogues that can lead to co-constructed systemic solutions across multiple levels of government.  The partners of organizations serving the homeless need to become learning organizations that better understand the importance of coordinated systemic solutions in addressing problems of chronic homelessness.  It is simply not sufficient to say, “we are all in this together.”  We need to move beyond rhetoric –what is needed is a coordinated system in which all partners feel valued and supported.