Creating an Open Space of Ideas: Inviting all Evaluators | Rethinking & Reshaping eValuations at a Time of Crisis | Making Evaluations Matter

Author: April Nakaima and Rachael Gibson

Towards an Ecology of Care during the Pandemic: Questions to help Frame System-Level Responses for Chronic Homelessness

Staff of the Evaluation Centre in Collaboration with Our Partners at Margaret’s Housing and Community Support Services and other Community Organizations

As part of an evaluation that the Evaluation Centre is conducting with Margaret’s Housing and Community Support Services, and through speaking to other providers of care in the community sector serving marginalized individuals, a number of questions has surfaced during this pandemic that can help with improving systems to create solutioning for individuals who face multiple disadvantages at the intersection of housing and mental health. The Evaluation Centre is presently working on a full-fledged report that discusses the implications of these questions for both understanding problems and generating solutions.

Some of these questions will need to be addressed by conducting summative evaluation approaches in the near future. However, most of these questions will require developmental evaluation approaches that need to be co-constructed between the organizations working to provide an ecology of care for individuals who face co-occurring conditions of homelessness, mental illness, and other issues such as substance use, gender identity discrimination, racial discrimination, and disabilities.

  1. Understanding syndemic processes:  An important issue that surfaced in our dialogues with multiple community stakeholders was the conceptual need to better understand mutually reinforcing vicious cycles that might connect chronic homelessness to poor health. The pandemic might further serve to exacerbate these connections by both reducing capacities of community organizations as well as by worsening health conditions for marginalized individuals. How will an understanding of the syndemic processes of chronic homelessness inform better planning for the ongoing and future pandemics?
  2. Learning from successful drop-in/respite sites: Not all Drop-in/Respite sites have seen problems during the pandemic. Some have had outbreaks; others have had no cases. Can a comparative analysis of such facilities help us better understand factors associated with successful responses to the pandemic?
  3. More deeply understanding the role of community organizations  in preventing and responding to the pandemic:  In our multiple dialogues what has clearly surfaced is the great need to better value the role of community organizations in an ecology of care in addressing the problems of chronic homelessness and persistent mental health challenges. What role does research and evaluation play in creating better awareness of the value of community organizations in such an ecology of care?   
  4. Equity in distribution of PPE. A related set of basics questions that emerged from multiple dialogues with community organizations has included a focus on equity in distribution of PPE and testing kits: How can there be a clearer prioritization of the importance of community organizations in responding to the pandemic? What are more equitable ways of distributing testing kits, PPE, etc.?
  5. Managing fatigue:  A number of organizations working on homelessness are  already overstretched, under-funded, and even during normal times the staff are potentially over-worked and stressed. Combined with the fear and the potential chaotic disruption caused by the pandemic, important questions for both systems and organization leaders working on chronic homelessness include: How does one manage fatigue/stress among frontline staff addressing homelessness given the long hours and extreme stress that they are facing during the pandemic? Going forward how does one build surplus capacity in community organizations given the potential organizational disruptions caused by the pandemic?
  6. Leveraging political and social opportunities to find solutions to chronic homelessness:  Stakeholders also noted that the pandemic also has created a political climate in which key actors who perhaps may not have been cognitively and politically sensitive to recognizing the need for multi-sector approaches to chronic homelessness are becoming more understanding and supportive of the interconnections between social and economic factors that drive chronic homelessness. Stakeholders noted that such favourable changes in a recognition of the multilevel complex drivers of chronic homelessness can be seen in the three levels of government. Can more be done to leverage such changes in political climate to create policy and practice solutions that are aligned across the three levels of government?
  7. Generating client-sensitive solutions:  Another point that surfaced in our multiple dialogues was the need to respond to the pandemic in ways that recognizes the lived realities of clients. For example, providing hotel rooms to isolate individuals who have tested positive or are awaiting test results might not work for some individuals whose mental health may de-compensate in such an environment. How do we think more systemically about solutions that are more client-sensitive and especially acknowledge challenges of anxiety and depression among some marginalized populations?
  8. Having better established communication protocols:  A point raised by multiple community stakeholders in the initial stages of the pandemic was the need to have clearer protocols/guidelines when faced with the disruptions caused by the pandemic. The lack of clear protocols/guidelines was deeply problematic for at least a few clients. Some of the issues that were raised in terms of a lack of communication included: lack of clarity on how long testing would take, lack of clarity of where individuals needed to be housed or stop-gap solutions that respected client contexts and client choices. What are processes going forward to develop better communication protocols between the hospitals/public health sector and the community sector?
  9. Creating systems of coordinating and supporting community care:  Another point that surfaced in multiple discussions was the experience of some community support and housing organizations that felt very much alone as they responded to the pandemic. Some of the issues that surfaced pointed to a lack of system coordination including: the need for more thoughtful systems of alerts, the need for data sharing, and the need for clearer protocols and guidelines mentioned earlier. Some of the stakeholders we spoke to did not feel supported in their efforts to thwart the pandemic or felt that there was not a system of care to support them. How does one create a more caring, coordinated system of care in which organizations don’t feel alone in responding to a pandemic?
  10. Learning solutions from the pandemic:  Some stakeholders we spoke to also spoke about some of the lessons that they have learned about the power of virtual interaction for some clients. For example, some clients have accessed more time with their psychiatrist virtually and have preferred the online interactions to their previous face-to-face interactions. Going forward, are there hybrid models in which face-to-face solutions can be combined with online solutions for some clients?

We believe that addressing the above points both through individual evaluations of community care organizations’ experiences and through generating solutions systemically perhaps through a systems-focused developmental evaluation process might be needed. It’s important we do not treat the issues raised above as just realities affecting one organization; in our work evaluating multiple community interventions during the pandemic and dialogue through the webinars that the Evaluation Centre have organized, we hear very similar concerns being raised by community organizations in countries like Chile, the UK, and India. Any solutions that we arrive at within Ontario will in all likelihood have relevance in other settings.  

Valuing Innovations: Evaluating the Online Platform of Dancing with Parkinson’s

The Evaluation Centre and Dancing with Parkinson’s (DWP) started collaborating in 2015 through a project funded by the Ontario Brain Institute to evaluate DWP’s in-person dance classes.  Our relationship with DWP was only formal for the first year; however, over the past 4 years our collaboration has continued, as Sarah and her team continue to challenge colleagues at the Evaluation Centre to think creatively about how evaluation can help community organizations, such as DWP, better tell their contribution story.  The need for such stories has only grown given the present pandemic.  Last month we started an evaluation of DWP’s most recent innovation:  live, online dance classes.  This evaluation provides an opportunity to further develop this mode of delivery, build connections between this platform and other interventions, and explore client experiences over time.

Targeting Social Isolation

Parkinson’s Disease (PD) is a neurodegenerative condition with no cure to date.  Most often diagnosed in people at around age 60 or older, PD manifests as a movement disorder affecting balance, ability to initiate voluntary movement like walking and talking, and a host of other symptoms including fatigue and depression.  As the disease progresses, people with PD tend to isolate themselves due to concerns that they might, for example, fall and get injured while out, have difficulty communicating with others, or that the outing would be too exhausting.  Isolation is familiar for people with PD.

Research tells us that living in isolation (outside of this pandemic) is associated with poorer health outcomes – physical, cognitive, mental, and emotional.  Conversely, individuals with greater social support (e.g., close family and friends around) tend to have better health outcomes and tend to recover more quickly from illness, injury, and surgery.  In the UK the Campaign to End Loneliness provides examples from research of how detrimental isolation can be for health: “Loneliness, living alone and poor social connections are as bad for your health as smoking 15 cigarettes a day. (Holt-Lunstad, 2010) … Lonely people are more likely to suffer from dementia, heart disease and depression. (Valtorta et al, 2016) (James et al, 2011) (Cacioppo et al, 2006).” In addition, two years ago the UK government appointed a Minister of Loneliness. Aware of the negative aspects of loneliness, one of DWP’s core missions has been to bring people with PD out of isolation.  In the unique context of COVID-19 where isolation is associated with better health outcomes, many of us are experiencing isolation at home, regardless of our demographic, socio-economic, geographic, or health profile.  But long before COVID-19 necessitated isolation, people with PD and large numbers of seniors were already experiencing isolation.  An evaluation of DWP can offer important learnings about the management of isolation.  How, for example, can the “positives” of isolation be amplified and the negatives of isolation dampened?

It becomes important to evaluate this program for multiple reasons:  

Imagining and Conceptualizing the Intervention

  • The innovation as an organizing device.  One of the challenges of living with the pandemic, or a disease like PD that pushes one toward isolation, is that time can become blurred.  The DWP online dance class is offered daily for 20 minutes.  One area that may be valuable to explore is whether and how this innovation helps organize the day for the participants.  One mechanism by which the online dance class can benefit participants is as an organizing device that helps bring structure to their daily activities.  Exploring this mechanism will help understand one aspect of how an online platform can benefit participants.  What can we learn about a brief online class as a structuring device to help participants organize the day?
  • Displacement from social isolation: Building community.  The second mechanism by which an online class can potentially work is that it can serve to address issues of loneliness while simultaneously building social connections to community.  On any given day, the class might have 70 or more participants.  One of the features of the online class is that for 5-10 minutes before and after the actual dance portion of the class, there is time open for dialogue with the instructor and class participants.  The vast majority of participants take advantage of this opportunity to dialogue and build connections with others.  When we consider the mechanism of addressing social isolation and building community through online platforms it is important to recognize that individuals without access to digital devices, such as computers, will be excluded from such communities and may feel further isolated as a result.  Acknowledging this as a critical issue, Sarah and her team are currently investigating potential options for providing digital devices to those in need.  Important evaluation questions include:  How does the online environment build connections between the participants?  Can the platform be further enhanced to build community?  How can the online intervention include the most marginalized individuals?
  • Building theories of change across different settings.  One of the deeper aspects of dance as an intervention for PD is that a comprehensive theory by which dancing can impact the health of Parkinson’s patients is unclear.  While some mechanisms about how dance can provide benefits are known, other mechanisms, such as the role of building social connections and reducing isolation, have yet to be fully explored.  Further, it remains an open question as to how dance can best be combined with other neurological interventions. The issue is not dance vs. neurological interventions; rather, the question is: What would a comprehensive package of dance alongside other interventions look like?  This web-based class provides an opportunity to explore the interactions of mechanisms in a very different context from the in-person delivery.  Following a realist evaluation frame, we believe that there is value in exploring feedback from multiple participants (at different stages of the disease, with different abilities, in varying contexts of living arrangements) regarding how different mechanisms might work.  There is also a need to better understand the extent to which such a modified package of innovations could be successful given that the mechanisms of delivering the dance class online miss some key aspects, such as touch – hugs, for example, are often shared between teachers/volunteers and participants, physical contact is made during partnership dances like the Waltz and Tango, gentle corrections to posture and alignment are made through touch during the class, and participants hold hands in the final closing circle. During such moments of touch, the hormone oxytocin is released and is associated with feelings of trust, bonding, social connection.  What would a theory of change that explores multiple mechanisms by which dance can impact brain health look like?  In what ways, does DWP’s original theory of change differ when using this very different delivery platform in the context of COVID-19?
  • Emergent possibilities.  The present innovation provides an opportunity to define what success means from the perspective of participants with PD.  This is a deep issue given that PD is a neurodegenerative condition and the expected trajectory of the disease leads towards a decline in one’s ability to perform the activities of daily living as well as a decline in health outcomes more broadly.  As such, success is not easily defined even in the best of circumstances.  Given that these online classes were designed in response to a crisis situation, participants themselves may be discovering how they view the success of such an innovation.  There are multiple emergent possibilities that might happen as such an innovation unfolds.  Some of the unfolding discovery incorporates the possibility that even after the intensity of the pandemic ends and in-person classes resume there will be a need to provide a version of such an online class.  How can evaluation help map out how clients view success in a more dynamic way?
  • The importance of aesthetics:  One of the big gaps in the literature involves understanding the role of the aesthetics of dance, art, and music, as well as how the arts can be used as interventions to improve health.  In a beautiful article titled “Feeling Lovely: An Examination of the Value of Beauty for People Dancing with Parkinson’s”, Sara Houston has argued for the value added of aesthetics through creating a mechanism of “feeling lovely.”  The DWP evaluation needs to explore whether the aesthetics still come through in the online platform in the context of participants’ own living rooms.   Exploring the linkage between aesthetics and health and the possibilities of delivering dance online at a large scale has enormous ramifications for how we can think about the ‘solutions space’ of dance-based interventions for brain disorders.  How does one stay true to principles of aesthetics even in an online platform?  How does one establish the impacts of aesthetics on health?  Does such a linkage exist?

The Role of Developmental Evaluation

  • The innovation as a developmental process.  From a developmental evaluation frame, like a number of disruptions caused by the pandemic, it’s important to see this online dance intervention as a developmental process.  As with any innovation, there is uncertainty facing the sustained delivery of this online dance innovation.  The evaluation provides an opportunity to learn and help develop the intervention further.  What needs further development?  How can the evaluation help with such development?
  •  Principles that guide the innovation.  Related also to a developmental evaluation process, there is a need to explore basic principles to inform delivery of such an intervention.  What are key principles that guide the delivery?  Are these principles static?  What do different clients at different stages of the disease want from the class, and what are the principles that drive them?

Understanding Experiences and Scaling-up

  • Understanding experiences and impacts over time.  Another important focus of the evaluation is to explore the impact of this innovation.  In what ways are the clients being impacted?  How do client experiences change over time?  In addition to impacts, it is important to explore the dynamics of such experiences.  For example, it is possible that such classes are exciting initially but that the enthusiasm wanes over time; however, it is equally possible that enthusiasm for the classes grow over time and they become part of the daily regimen of the clients.  What are the impacts of such an online platform on clients? Ho do client experiences of the online platform change over time?
  • Scaling Up.  One of the great benefits of the online model of delivery is that it might be relatively easier to scale than a face-to-face delivery model.  The evaluation needs to explore the kinds of constraints that need to be addressed to scale up such a dance-based intervention widely.  Over the last few weeks alone, the participations have moved from 15 to 400+.  How can the evaluation help with the process of scaling up?

Looking Ahead: The Need for Stories

As we enter a brave new world of both constraints on past ways of doing things and emerging possibilities with newer platforms, evaluation has a role to play to help further develop such innovations and also explore some of the initial impacts of such platforms.  The online dance intervention that Sarah and her team has developed is both a labour of love and also a creative solution that pays careful attention to the multiple realities and constraints of teaching online. Teaching online will come with a diminished role of key mechanisms, like touch, but it may have other compensatory mechanisms.  The evaluation provides an opportunity to document how this online platform evolves and also relay the stories from clients about how this platform makes a difference in their lives.