The Evaluation Centre for Complex Health Interventions (TECCHI) at the Li Ka Shing Knowledge Institute at St. Michael’s Hospital, has been established to be a centre of excellence in evaluation. The growing focus on learning and accountability in Canada and internationally has resulted in an increasing need for evaluation approaches to understand if policies and programs work, how they work and what can be done in planning and implementation to make them work.
We define complex interventions as:
(i) interventions that have multiple components,
(ii) interventions that are dynamic (interventions change over time as they adapt to context and respond to learning);
(iii) interventions are heterogeneous across settings (the same intervention will look very different in different contexts).
Given the complexity of some of the health policies and programs, the designs and approaches required to evaluate them need to go beyond standard experimental or quasi-experimental approaches and will require the evaluator working with policy makers and program planners right at the policy/program formulation stage. Responding to the complexity of interventions requires a range of evaluation approaches and a mix of methods.
- We aim to ensure that health interventions are able to achieve intended health outcomes due to the integration of evaluation feedback right at the planning and implementation stages.
- We aim to build the evaluation capacities of all the partners that we work with.
- We aim to bring a mixed set of qualitative and quantitative methods to understand what works, for whom and how.
The Evaluation Centre contributes to an enhanced learning and accountability culture for improving health outcomes. We work with policy-makers, planners and implementers to integrate evaluation approaches and evaluative thinking to more comprehensively understand the impacts of policy and programmatic interventions. Ultimately such an enhanced evaluation culture can contribute to strengthened health systems and improved health outcomes.
Key areas of focus include:
- We appreciate that a majority of policy and community interventions are complex. By this we mean the interventions are likely to change over time both in response to changes in context as well as learning over time. Further interventions need to adapt to fit into local context and also respond to the diverse needs of individuals. Typically, complex interventions have multiple components. Complexity has implications for what constitutes rigor in evaluation design. We think discussions of gold standards of evaluation design that do not pay attention to the complexity of the interventions are plainly silly. The eventual test of rigor in evaluation is the ability to generate learning and bring meaningful difference to intervention participants.
- We work with partners to implement a range of methods to demonstrate accountability but we also partner to develop specific and strategic approaches to learning.
- Balancing accountability with learning when faced with complexity requires innovative evaluation approaches such as developmental and realist evaluation. We seek to combine a focus on strategic planning that approaches such as developmental evaluation offers with a focus on accountability that experimental and quasi experimental approaches provide.
- We work with organizations to move away from an exclusive focus on project level thinking towards building capacities within organizations for learning and accountability. We think organizations need to demonstrate both an accountability towards results and also an accountability towards learning.
- We think that the raison d’être of social and health programs is about making a difference in the lives of individuals who need it the most. This means that all evaluations need to specifically focus on equities and demonstrate how programs can balance efficiency, effectiveness and equities.
- We have worked with a number of institutions to build evaluation capacities. For example, we worked with the China National Health Development Research Centre and also the Ontario Brain Institute to build evaluation capacities to evaluate health equities. We seek to partner with other research academic and community organizations in building capacities to evaluate and improve programs over time.