Our Publications (2016/17)
Sridharan, S., Goering, P., Streiner, D.L., van Draanen, J., Veldhuizen, S., Nichols, J. and Nakaima, A. (2017). The Importance of Surfacing Anticipated Timelines for Evaluations of Complex Community Health Interventions: A Case Study from a Research Demonstration Project Focused on Homelessness and Mental Health. Paper under revision.
Abstract
One of the more surprising realities of implementing evaluation designs is there is often a lack of clarity on how long an intervention will take to impact outcomes. This paper discusses an approach that involves key stakeholders in programme planning and implementation in developing a timeline of impact at the outset of an intervention. This method has been trialed through a case study of a large national multi-site intervention focused on mental health and homelessness. The goal for the research study was to generate evidence regarding practical services and systems to help support homeless people with mental illness in Canada. The data for this case study were collected at a national cross-site meeting that involved national project leads, site implementers, and evaluators from the five sites. Respondents completed a structured survey and also discussed their responses to open-ended questions.. This article demonstrates that explicit thinking about timelines of impacts can help with reflections on programme mechanisms and also provides a frame for discussions around the heterogeneous understanding of programme mechanisms across multiple contexts. In our study, ideas about mechanisms were remarkably varied, and the discussion exposed differences in understanding of the programme mechanisms across stakeholders that would not otherwise have come to light. Developing an anticipated timeline of impact before evaluation designs are implemented can help with understanding expectations of stakeholders, help surface/clarify the theory of change, highlight differences in theories and mechanisms across sites due to contextual differences, influence data analysis, and improve the state of the craft of conducting realist evaluations.
Sridharan, S., Nakaima, A., Gibson, R., Williams, C., Blair, K. and Bondy, M. (2017). Incorporating Syndemics and Ecology of Care into Evaluations of Interventions of Co-Morbidities: A Theory-Driven Approach. Paper under review.
Abstract
Background
This paper discusses lessons learned about heterogeneous needs and expectations of clients from a theory-driven evaluation of a drop-in centre in an urban setting that serves individuals, many homeless, and experiencing complex and often co-occurring disorders, including mental illness, disability, and addiction. We argue that such heterogeneous expectations have implications for both planning future interventions and evaluations of drop-ins.
Methods
This paper builds on the work of Van Belle et al. (2010), which proposes a six-step framework for the design and implementation of a theory-driven evaluation approach. We have adapted the framework to better fit the purpose of our evaluation, which focuses on issues regarding the heterogeneity of expectations of individuals, leverage of the intervention, and implications from the program level learning to a system level theory of change. Four evaluation team researchers interviewed 101 drop-in clients in teams over five weeks at the drop-in using a semi-structured questionnaire.
Results
The vast majority of clients had little expectation that the drop-in could “improve” their health or living situation. Instead what they valued and attributed to the drop-in were: the daily meals and coffee, opportunities to socialize with others, the space for hanging out and getting out of cold weather, opportunities to volunteer and help others, stability that the drop-in seemed to help them maintain, moments of joy and peace. Many of the clients were either unaware or disinterested in the services provided (e.g., housing referral, harm reduction) or connection to services. Most clients exhibited a lack of an entitlement attitude and the vast majority, if not all, greatly appreciated the drop-in.
Conclusions
The mutually reinforcing dynamics by which intersections of potential problems can dynamically amplify over time and create a vicious cycle — should not be underestimated. Taking a syndemic perspective that pays attention to the ‘synergy’ of co-occuring epidemics helps us become more aware about the challenges of a drop-in to reverse these patterns. An integrated approach involving other actors in the health, social and political systems with shared accountability is needed.
Sridharan, S., Mayne, J., Maplazi, J., Nakaima, A., Le, K., Saneinejad, S., Slutsky, A., Rotstein, O. and Laupacis, A. (2017). A Contribution Analysis Approach for Assessing Research Impacts: A Case Study of a Knowledge Institute in Toronto, Canada. Paper under review
Abstract
This paper describes a series of experiments conducted to understand how a very diverse group of scientists based in a Knowledge Institute in Toronto Canada could learn about the impact of their research on a range of social, health and economic outcomes. The experiments were informed by a theory-driven evaluation approach called Contribution Analysis. These experiments will help develop a monitoring and evaluation platform that can help the leadership assess progress of the knowledge institute towards the longer-term mission of the Institute to impact positively on peoples’ lives. Three pathways to impact — research/discovery, experiential/practice and capacity building—were uncovered. We plan to link measures directly to the impact pathways. As a result of these experiments we propose a metrics architecture that consists of a core set of monitoring measures, self-reflective impact narratives and evaluation of strategic areas.
Gibson, R. and Robichaud, S. (2017, Forthcoming). Evaluating Dancing With Parkinson’s: Reflections from the Perspective of a Community Organization. Evaluation and Program Planning.
Abstract
In 2015, Dancing With Parkinson’s (DWP), a Toronto-based community organization, participated in the Ontario Brain Institute’s (OBI) newly launched Evaluation Support Program. This paper reflects on that experience. In particular, we identify the key lessons derived from the OBI initiative, discuss how these lessons have informed DWP practice going forward, and highlight what we consider to be the most valuable aspects of the Evaluation Support Program. While we now recognize the need to establish an evaluation culture within DWP, we find that there are significant challenges associated with both building and sustaining evaluation capacity in the context of a small community-based organization. Whereas DWP has built considerable strengths in terms of informal evaluation capacity, on its own, such capacity is insufficient to, for example, demonstrate DWP’s impact to outside audiences or successfully scale up the program.
Nylen, K. and Sridharan, S. (2017, Forthcoming). Experiments in Evaluation Capacity Building: Enhancing Brain Disorders Research Impact in Ontario. Evaluation and Program Planning.
Abstract
This paper is the introductory paper on a forum on evaluation capacity building for enhancing impacts of research on brain disorders. It describes challenges and opportunities of building evaluation capacity among community-based organizations in Ontario involved in enhancing brain health and supporting people living with a brain disorder. Using an example of a capacity building program called the “Evaluation Support Program”, which is run by the Ontario Brain Institute, this forum discusses multiple themes including evaluation capacity building, evaluation culture and evaluation methodologies appropriate for evaluating complex community interventions. The goal of the Evaluation Support Program is to help community-based organizations build the capacity to demonstrate the value that they offer in order to improve, sustain, and spread their programs and activities. One of the features of this forum is that perspectives on the Evaluation Support Program are provided by multiple stakeholders, including the community-based organizations, evaluation team members involved in capacity building, thought leaders in the fields of evaluation capacity building and evaluation culture, and the funders.
Nakaima, A. and Sridharan, S. (2017, Forthcoming). Reflections on Experiential Learning in Evaluation Capacity Building with a Community Organization, Dancing With Parkinson’s. Evaluation and Program Planning.
Abstract
This paper discusses what was learned about evaluation capacity building with community organizations who deliver services to individuals with neurological disorders. Evaluation specialists engaged by the Ontario Brain Institute Evaluation Support Program were paired with community organizations, such as Dancing With Parkinson’s. Some of the learning included: relationship building is key for this model of capacity building; community organizations often have had negative experiences with evaluation and the idea that evaluations can be friendly tools in implementing meaningful programs is one key mechanism by which such an initiative can work; community organizations often need evaluation most to be able to demonstrate their value; a strength of this initiative was that the focus was not just on creating products but mostly on developing a learning process in which capacities would remain; evaluation tools and skills that organizations found useful were developing a theory of change and the concept of heterogeneous mechanisms (informed by a realist evaluation lens).
Sridharan, S. and Nakaima, A. (2017, Forthcoming). Valuing and Embracing Complexity: How an Understanding of Complex Interventions needs to Shape our Evaluation Capacities Building Initiatives. Evaluation and Program
Abstract
Planning.
Part of a larger forum on evaluation capacity building, this paper describes some of the main challenges of evaluating complex interventions, as well as the implications of such challenges for evaluation capacity building. It discusses lessons learned from a case study of an evaluation of Dancing with Parkinson’s, an organization that provides dance classes to people with Parkinson’s disease in Toronto, Canada. These implications are developed from a realist evaluation lens. Key lessons include the need to develop certain skills to understand program mechanisms and contexts, recognize multiple models of causality, apply mixed method designs, and ensure the successful scaling up and spread of an intervention.
Sridharan, S., Dey, A., Seth, A., Chandurkar, D., Singh, K., Hay, K. and Gibson, R. (2017, Forthcoming). Towards an Understanding of the Multilevel Factors Associated with Maternal Health Care Utilization in Uttar Pradesh, India. Global Health Action.
Abstract
Background: This paper explores the multilevel factors associated with maternal health utilization in India’s most populous state, Uttar Pradesh. Based on 5,666 household surveys, it examines factors at the district and individual levels as well as three key utilization practices: whether the respondent’s pregnancy was registered, whether the woman obtained any antenatal care, and whether the delivery was a home birth. The data were obtained as part of a baseline evaluation of the Uttar Pradesh Technical Support Unit (UPTSU) program. This intervention aims to assist the Government of Uttar Pradesh in increasing the efficiency, effectiveness, and equity of service delivery across a continuum of reproductive, maternal, child, and adolescent health (RMNCH+A) outcomes.
Methods
The paper employs multilevel models that control for individuals being nested within districts in order to understand the predictors of maternal health care utilization. It relies on data obtained from a household survey, which was conducted as part of the evaluation.
Results
The study identifies several important individual-level predictors of health care utilization, including: the literacy of the woman, the husband’s schooling, age at first marriage, and socio-economic factors. With respect to pregnancy registration, key predictors include the husband’s schooling (OR 1.49, 95% CI 1.26-1.76), having a bank account (OR 1.36, 95% CI 1.11-1.68) and owning a house (OR 2.28, 95% CI 1.85-2.80). Decreased odds of pregnancy registration are associated with the respondent’s marriage age at 15 years or younger (OR 0.84, 95% CI 0.74-0.96) and increased density of people per sleeping room (OR 0.77, 95% CI 0.62-0.96). Several factors are associated with receiving antenatal care, including the woman’s literacy (OR 1.49, 95% CI 1.28-1.73), the respondent having had a job outside the home in the last year (OR 1.39, 95% CI 1.10-1.77), and owning a house (OR 2.83, 95% CI 2.27-3.53). Reduced odds of antenatal care are associated with lower caste/social class (OR 0.73, 95% CI 0.58-0.92), marriage age of 15 and younger (OR 0.71, 95% CI 0.63-0.81), and increased age at interview (OR 0.97, 95% CI 0.96-0.98). Home delivery tends to be associated with lower caste/social class (OR 1.69, 95% CI 1.30-2.21) and marriage age of 15 and younger (OR 1.48, 95% CI 1.26-1.73). Factors that decrease the odds of having a home birth include the woman’s literacy (OR 0.62, 95% CI 0.54-0.72), having a bank account (OR 0.79, 95% CI 0.65-0.95), and increases in the wealth index (OR 0.96, 95% CI 0.94-0.99).
Implications
Changing patterns of inequities suggests that the intervention needs to disrupt the existing patterns of the health gradient. In order to successfully implement the intervention, it is necessary to understand the mechanisms by which the intervention can disrupt the unequal utilization patterns and target the various domains of disadvantage. Knowledge of key predictors of utilization can aid in the implementation of complex interventions incorporating equity considerations in healthcare utilization.
Zhao, K., Sridharan, S., Ingabire,M.-G., Yu, M., Nakaima,A., Li, M., Xiao, Y., Chen, E. (2017, Forthcoming). An Experiment on Building Evaluation Capacity to Address Health Inequities in China. In Sridharan, S., Zhao, K. and Nakaima, A. (Eds.), Building Capacities to Evaluate Health Inequities: Some Lessons Learned from Evaluation Experiments in China, India and Chile. New Directions for Evaluation.
Abstract
This paper describes an evaluation experiment conducted in China between 2013 and 2016 to use evaluative thinking and evaluation approaches to help build the salience of health equity as a performance measure for health systems. This project was informed by a realist evaluation approach that sought to understand the context, mechanisms and outcomes underlying health inequities. This chapter describes a theory of change that includes descriptions of the key actors involved in the project, the mechanisms of impact, and short- and long-term outcomes. Key questions that could help assess the impact of this project are also discussed. This paper contributes to the literature on building evaluation capacity for health inequities.
Zhao, K., Nakaima, A., Wudong,G., Qiu, S., Sridharan, S. (2017, Forthcoming). Evaluation at the Time of Health Systems Reform: The Needs of Chinese Policymakers for a Robust System of Evaluations to Assess Progress in the Implementation of Health System Reform Efforts. In Sridharan, S., Zhao, K. and Nakaima, A. (Eds.), Building Capacities to Evaluate Health Inequities: Some Lessons Learned from Evaluation Experiments in China, India and Chile. New Directions for Evaluation.
Abstract
In 2014 the authors of this paper, evaluators from China and Canada, jointly interviewed 12 policymakers in China at the national and provincial levels. This paper describes the needs of policymakers and how they view the evaluation capacity building needs of the health system. The learnings from the policymaker dialogues informed the evaluation capacity building efforts at three pilot sites in China. This paper focuses on the evaluation capacities needed by policymakers and implementers specifically to address inequities in the health sector, unlike most publications that focus on evaluation capacities of researchers
Xiao, Y., Zhao, K., Sridharan, S. and Cao, X. (2017, Forthcoming). Conceptual Indicators Framework for the Chinese Health System Strengthening. In Sridharan, S., Zhao, K. and Nakaima, A. (Eds.), Building Capacities to Evaluate Health Inequities: Some Lessons Learned from Evaluation Experiments in China, India and Chile. New Directions for Evaluation.
Abstract
In 2009 the national government of China launched massive health reforms, together with other in social and economic reforms. For the first time, evaluation was included in the draft national plan health reforms and development—the so-called “12th Five-Year National Plan for Health Development (2011-2015)”. The Chinese and Canadian researchers, with the support of the International Development Research Centre, Canada, helped to facilitate a deliberative process by various actors by conceptualizing an indicators system and mapping out key questions to be addressed by evaluating implementation of the national health plan. The conceptual indicators system serves as a platform for users and implementers of evaluation program to understand needs for evaluation better and sharpen focus on more prominent dimensions such as equity and contextual analysis.
Sridharan,S, Zhao,K., Nakaima, A., Maplazi, J., Yu, M., and Qiu, Y. (2017, Forthcoming). Towards a Structured Process to Evaluate Health Inequities: Lessons Learned from Developing and Implementing Evaluation Guidelines to Address Health Inequities. In Sridharan, S., Zhao, K. and Nakaima, A. (Eds.), Building Capacities to Evaluate Health Inequities: Some Lessons Learned from Evaluation Experiments in China, India and Chile. New Directions for Evaluation.
Abstract
This paper discusses the development and implementation of structured guidelines that contained a fixed set of questions as part of a project in building evaluation capacity for addressing health inequities in the health sector in China. The ambition of the guidelines was to test if a structured process of questions could aid teams that did not have a strong background in evaluation to complete equity-focused evaluations and also help raise the salience of health equity as an important goal of health systems reform among the participants of the project. The guidelines were informed by multiple perspectives that included the literature on social determinants of health, realist evaluation as well as utilization/ influence perspectives. One noteworthy aspect of this project was that the participants in this capacity building project were local and national policymakers, practitioners, and university researchers. Our original goal was to “test” these guidelines in equity-focused policy interventions in three separate provinces of China. One key insight from this project was that there is a need to move away from a testing perspective to a developmental approach in developing evaluation guidelines that can work across multiple organizational and country contexts and also closely reflect the needs of practitioners and policymakers in evaluation capacity building efforts.
Sridharan, S., Jones, B., Caudill, B. and Nakaiama, A. (2016). Steps towards incorporating heterogeneities into program theory: a case study of a data-driven approach. Evaluation and Program Planning.
Abstract
This paper describes a framework that can help refine program theory through data explorations and stakeholder dialogue. The framework incorporates the following steps: a recognition that program implementation might need to be multi-phased for a number of interventions, the need to take stock of program theory, the application of pattern recognition methods to help identify heterogeneous program mechanisms, and stakeholder dialogue to refine the program. As part of the data exploration, a method known as developmental trajectories is implemented to learn about heterogeneous trajectories of outcomes in longitudinal evaluations. This method identifies trajectory clusters and also can estimate different treatment impacts for the various groups. The framework is highlighted with data collected in an evaluation of an alcohol risk-reduction program delivered in a college fraternity setting. The framework discussed in the paper is informed by a realist focus on “what works for whom under what contexts.” The utility of the framework in contributing to a dialogue on heterogeneous mechanism and subsequent implementation is described. The connection of the ideas in paper to a ‘learning through principled discovery’ approach is also described
Sridharan, S., Maplazi, J., Shirodkar, A., Richardson, E. Nakaima, A. (2016). Incorporating Gender, Equity and Human Rights into the Action Planning Process: Moving from Rhetoric to Action. Global Health Action.
Abstract
BACKGROUND
Mainstreaming of gender, equity, and human rights (GER) is an important focus of the World Health Organization (WHO) and other UN organizations. This paper explores the role of action plans in mainstreaming GER. This paper is informed by a theory-driven evaluation lens.
DESIGN
A theory of change framework explored the following seven dimensions of how action plans can implement mainstreaming of GER: awareness of the foundations of GER; understanding of context; planning to impact GER; implementation for GER; monitoring, evaluation, and learning; planning for sustainability; agenda setting and buy-in. The seven dimensions were used to analyze the action plans. Reviewers also explored innovations within each of the action plans for the seven dimensions.
RESULTS
GER mainstreaming is more prominent in the foundation, background, and planning components of the plan but becomes less so along the theory of change including implementation; monitoring and evaluation; sustainability; and agenda setting and buy-in.
CONCLUSIONS
Our analysis demonstrates that much more can be done to incorporate GER considerations into the action planning process. Nine specific recommendations are identified for WHO and other organizations. A theory-driven approach as described in the paper is potentially helpful for developing clarity by which action plans can help with mainstreaming GER considerations.