TECCHI’S final webinar in the Sustainable Development Goals (SDGs) series was focused on leaving nobody behind. “Leaving no one behind” a phrase commonly used in the global health and international development sectors was unpacked during the webinar, with a focus on how it relates to the field of evaluation. The topic was examined from multiple points of view with some speakers focused on the importance of understanding local contexts and power dynamics while trying to ensure that nobody gets left behind. Another speaker approached the topic from a different lens, by describing how the burden to sustain interventions is often placed on countries that are impoverished and likely lacking in capacity to sustain program outcomes. She debated whether or not “leaving nobody behind” is a reasonable goal to have given the political and economic structures that currently govern our world. This blog post will focus on the dialogue taking place between the webinar participants who engaged in discussions through Zoom’s chat feature. Below is a list of some comments that stood out to me from the conversation:
“The point being raised about trying to understand different perceptions is equally valid in developed economies eg in evaluating programmes for sex workers or drug dependents and possible non uptake of services provided – equally need to search for their understandings and reasons of why there is no take up”
“Why should be burden the community? can [we] have a model where the state is held accountable for services that are sensitive to social and cultural realities.”
“Prettiness of no one left behind isn’t matched”
“Finally, we don’t fund, design, implement, monitor or evaluate for sustainability or consider longer-trajectories post-project. Those are seen as the problems of the countries themselves…. who are poor enough to qualify for aid and often powerless and capital poor who can’t sustain multi-million dollar outcomes… Just a warning, ‘complexity’ can be hidden behind – it’s too complex to address, no?”
“I do think the methodologies and methods exist, but are not applied i.a. because of the political economy of evaluation, and of development.”
The first comment talks about the need to understand contexts and power dynamics from the participant’s perspective. While there is a growing realization of the importance of contextual considerations while developing, implementing and evaluating programs, it is important to be reflexive about our position and humble about our contextual knowledge (and lack of it). When working in the international development field there is a lot one might not know with regards to the context they’re working in, and it’s critical to seek out knowledge from program participants and other local actors to understand their perspectives about a given program and how it fits into the fabric of the local community. This comment also reminded me about a striking observation that Dr. Das made during the webinar. He hinted at how the field of development often attempts to fit a poor person into the western “modern” perspective. In his presentation, he talked about the promotion of institutional delivery in Odisha. While the program being implemented defined success as an increase in the number of women giving birth in an institutional facility, it didn’t take into account the fact that many of these same women didn’t even have proper houses to live in.
While the second comment is intriguing, I have to kindly disagree with this statement (“Why should be burden the community? can [we] have a model where the state is held accountable for services that are sensitive to social and cultural realities”). While it is important for states to have policies and frameworks that take into account social and cultural realities, it is also important for individuals at the community level to play a role in developing programs that are sensitive to their needs. Subsequently, actors at the local level also need to be held accountable to the promises that they make. Operating at the community level enables an organization to build intimate connections with those whose needs they’re serving, and consequently get a pulse on what reality is like from the program participant lens. It enables one to occupy a unique position where you do have the opportunity to listen with humility and lead change that is sensitive to the local context.
Another comment that arose from the discussion is the need to acknowledge the systematic confines that program implementers and evaluators are working within. This includes the fact that funding cycles are generally short-term and often narrowly focused on impacts from the funders’ perspective. Given these circumstances, there might be a failure to acknowledge the work done in obscurity before empirically measurable impacts can be observed. In some cases, this fixation on achieving impacts can hinder experimentation, as the room for failure is very small, and taking risks through experimentation could lead to outcomes that are less than optimal.
While “leaving nobody behind” is extremely important from a human rights perspective, we are aware that our reality is a stark contrast, with inequities growing deeper and the poor being stuck in a vicious cycle that is hard to break through. Given that our words don’t align with our reality, and our inequities are deeply embedded in economic and political systems, how do we work towards reconciling this difference?
TECCHI’s third webinar in the Sustainable Development Goals (SDGs) series was centered on the role that evaluations could play in measuring the coherence of interventions. Coherence can be defined as “how well an intervention fits” within a given setting. As a recent graduate, I’ve been surrounded by many classmates who are interested in the notions of disruption, innovation, and bringing new ideas to the solution space. However, too often our courses in university are structured so that students with similar academic backgrounds are in the same classes. Many times the lack of cross-faculty spaces for collaboration have come in the way of engaging with ideas using a lens of “coherence”. How do ideas generated in the field of health interact with those in political science, business and the environment? How do these different fields work together to solve problems in the real-world? The webinar on coherence helped me think more deeply about how ideas and interventions fit within a given context. Below are a few quotes that were mentioned in the chat during the webinar that stuck out to me:
“The relevance of formal vs, informal institutions and their overlay may need more attention. Linking back to [the] interest in how this overlay will be relevant in coherence in a systems’ perspective as well.”
“I’d like to hear more about the coherence of coherence. Coherence with what, in what contexts? I’ve long identified limitations of silo thinking and the myth that any single intervention can have meaningful impacts, in isolation”
“I would also like to hear about the linkage of coherence with sustainability. Also in my opinion, measurement could be key specifically in situation wherein there are different partners and programmes work in tandem”
“’coherence’ isn’t always a ‘good’. […] Its a bit about the argument about institutionalisation as always good!? It depends on coherence for what purpose which I think is what you are implying”
“Decision processes are surely key interfacing formal and informal institutions (or culture)”
“In some humanitarian contexts, “coherence” with national policies and strategies are sometimes not desirable.”
“I would like to support [a] comment on the difference between policy design and evaluation and claiming the need for more inclusion of evaluators in policy design. Not only formative evaluation but the promotion of old fashion ex-ante evaluation which we don’t seem to see much nowadays. Also this relates to how do we cumulate knowledge in program evaluation as discipline… and the needs for evaluators to be less reactive and more proactive (not to that easy of course!)”
“I think it’s interesting that conversations about implementation and evaluation often happen separately – I am not an evaluator but an implementer and we talk about coherence in different language and in different ways. I think involving evaluators in implementation/program design and vice versa would be beneficial to address a lot of this issues with coherence”
“For me, an important part of coherence is the idea of resonance — the idea of adding to what already happening and gaining (non linear) momentum in the process. This links us back to the idea of contributing to trajectories we talked about a couple of weeks ago – trajectories consisting of systems of actors interacting with each other, technology, institutions, etc that deliver coherent sets of outcomes over time. The outcomes can be good or bad, and trajectories may work to resist change.”
“that’s why I was arguing that the choice of unit of analysis is key – I wold [sic] argue it’s the cluster of interacting polices we should be looking at!”
“Isn’t the point that we need to choose what to be coherent with, and being more explicit and transparent about that choice?”
Coherence with what?
When thinking about coherence, many participants in the webinar noted the importance of stating up-front what policies, programs, and values a new program aims to be coherent with. In any given context, there are formal institutions and informal actors present, these different groups might have values that are similar, but also those that are divergent. By stating which aspects of a given context a new program aims to align with, program implementers could move towards building connections with different partners (such as community organizations or policy actors) that are working on similar projects or program areas. These new relationships can then be leveraged to work towards shared goals and ambitions around solving a problem.
Fred Carden, the first speaker in the webinar, discussed different types of coherence – internal, intra-organizational, inter-organizational and international coherence. The idea of coherence can be thought about on many different levels, and perhaps as one aims to increase the scope of organizations and national priorities one wants their program to be coherent with, the difficulty of the task increases. With an increased number of stakeholders, organizations and policies to consider, a larger diversity in views and values are present at the table. This in turn increases the difficulty of designing and implementing a program that is coherent with the views of a broad range of factors.
Is coherence always desirable?
During the webinar, some participants posed questions that sought to understand when coherence is desirable, and when it is not. Are there certain aspects of a program that we hope are coherent with a given context, and other aspects that are aiming to bring a more disruptive energy to a solution space? When there is disruptive change, are appropriate change management tasks undertaken so that individuals are able to understand the change that is being brought, and how their role is shifting as a result? How can disruption and the notion of coherence be reconciled? Although coherence might not be desired in all aspects of a program, it is important to understand how the program fits in the larger socio-cultural, historical, environmental, political and economic contexts. How does the new program build on progress that is already made, and what new inputs does the program bring in? Are the users of the program ready to engage with the new change that is being proposed? Were the users part of the process while developing the new program?
Measuring Coherence
Measuring coherence is also another theme that was brought up multiple times during our discussion. Given that in any given context there are often multiple interventions taking place simultaneously, and that different programs and services synergistically build on each other, how do we measure if a program is coherent in a given context? How do you measure contributions when there are many partners involved? What about the unseen partners who might be playing a key role? Another question that comes to mind is the role of evaluation in looking at problems as a whole, and understanding how multiple programs funded by different organizations are working in tandem towards solutions. Is there value in funding evaluations that aim to understand how an ecology of policies, programs, and stakeholders at multiple levels work together towards solving an issue that is of importance to them?
TECCHI’s webinar, “Contexts, Trajectories, and Assessing Progress Towards Goals”, gave rise to a lot of discussion on the topic of timelines and trajectories for understanding programs and their impacts. Below is a table of some insightful comments from different webinar participants as they engaged with the ideas being presented.
“ex-post we find things that ‘fail’ at final, sometimes succeed later, tho’ most of the time those that succeed (and we ASSUME sustained impacts) ‘fail’ a couple of years later.”
“People seem to think that funders and program designers are foolish to look for simple short term outcomes. I think a case can be made that such behavior is rational and the most effective tactic for addressing important issues.
“the short-term is, in my view, irrelevant to our participants and partners who need truly sustainable development that’s locally driven/ sustainable (listen to Time to listen by CDA/Mary Anderson’s crew)
“I agree the short-term view is problematic. I’m only saying that there are good reasons for it. If we don’t appreciate those reasons, progress will never happen.
“Surely the problem Jon Doe, is the preoccupation by traditional evaluators on projects/programmes because that’s who pays us. If you focus on policy and social change then the whole logic if short term impacts becomes absurd!
“Frankly I think we need Theories of Sustainability designed by those tasked with sustaining the outcomes, not just donors ToC during fixed lifetimes.
“I have been thinking about this. It is possible to know some relevant aspects of context in advance, but impossible to know how context affects outcomes except retrospectively.
“measuring multi-dimensional facets of contexts and trajectories should not force to add and burden the questionnaire/respondents…
“context, people and system matter! great said! How do we ensure the linkages to craft our evaluation more practical in day to day application? The human centered evaluation could lead us further towards reality of contexts?
“I always ask the evaluators in training to apply the concepts that we are learning to a program that the learners are working on or passionate about. Situational and Interpersonal Competencies Domains are so important to evaluators. There is more than the technical, reflective and management competencies domains that evaluators need.
Some of these comments shed light on the tensions that are present when thinking about timelines for impact. What are outcomes that we can expect to see in the short term? What is the nature of these outcomes, and how are they different from the long-term impacts that we hope to see in the future? How do short-term outcomes and long-term sustained impacts cohabitate in the same space of program planning?
When establishing timelines for impact, it is important to think about the assumptions that we are making for an impact to be achieved within a given timeline. It’s also perhaps important to reflect on how we go about creating these timelines in which we hope to observe positive change. Sandra Albert, one of the presenters in the webinar, hinted at how interventions are often planned around naive assumptions about the cultural practices in a given context. Given that contextual realities are often like murky water and there are multiple intersecting factors that play a role in the success or failure of interventions, how do we establish “reasonable” timelines for programs? Michael Woolcock, one of the presenters, spoke about the non-linear nature of trajectories and how evaluating an intervention at different points in time could lead to very different understandings on whether a program is successful or not. He also spoke about doing activities in obscurity before change can be observed. These actions often go unnoticed. Some of the webinar participants also mentioned how funding cycles are oftentimes built around achieving short-term wins. What is the value of these short-term gains? Perhaps, achieving these short-term wins that are less complex in nature could help create momentum and generate excitement about a project, and also bring a sense of hope that change is possible. However, it is important to balance this view with a lens of sustainability and helping communities develop the capacity to sustain impacts even once the official funding runs out. Given that observing change takes a lot of time, how do we go about creating theories of sustainability from the onset of a project?
Moving from sustainable impacts and outcome trajectories, to learning systems, I thought that the comments from the webinar participants on “human centered evaluation” and not “burdening” participants with completing questionnaires, quite intriguing. How do we make the process of giving feedback something that participants want to engage with? In order to feel motivated and empowered to provide feedback, and engage in the process of learning, program participants should feel a sense of ownership of the program, and should feel like the input that they are providing will get fed back into the project, ultimately making their experience interacting with the program more pleasurable. As we engage with these questions, it might be beneficial for program implementers and evaluators to collaborate with individuals across other fields such as design, urban planning, psychology etc. to understand how they engage with processes of continuous learning around the interactions of humans with their environment.
What happens once a program comes to an end? How do program implementers help build a system’s capacity to sustain impacts? How do we measure sustained impacts and how do metrics change over time to respond to the evolving needs of communities? These were some of the questions that were raised by participants during TECCHI’s webinar on Sustainability Evaluation. While the answers to these questions are complex and very dependent on the context of the program, this blogpost aims to reflect on some of the rich discussions that took place during the webinar that might offer some insights to these questions. However, this post might leave you having more questions than answers. Below are a few comments from the webinar participants that stood out to me:
“I just evaluated a complex health intervention here in [Canada] with absolutely stunning results, but everyone told us ‘Once you leave we won’t be able to continue this’”
“If we are such great systems thinkers, why not ponder the question of what the systems reasons for not doing follow-up evaluations?”
“In my work at [an international development organization] I was in the team screening proposals for funding, element of sustainability was weakest. When I visited communities after few months the funding ended, was frustrated to find most of the activities had stopped. When I asked the people, they said it was “NGO project” not ours.”
“Funding timelines and expectations are often sustainability killers even though they are project starters”
“Building ownership or behaviour change takes a long time because its about unlearning . For behaviour change in communities, the outsiders have to change. We have to shift as ‘experts’ with solutions to facilitators.”
“In my work with nonprofits, they don’t really start coming up with concrete sustainability strategies until they’ve done a sort of systems map of all the stakeholders they never thought about who stand to care about/benefit from the work they do, and id new partners as a result”
“Perhaps greater intentionality around exploring the mechanisms that support the achievement of outcomes; and consider how to sustain those mechanisms.”
“You know, we don’t kick our kids out of the house at 14, we slowly make sure they can handle money, handle a job, meet their school deadlines, etc. Maybe we should be planning exit strategies more intentionally in all programs, and in that period we would learn more about potential barriers to sustainability, what’s coming up that needs to be dealt with”
Community Ownership and Sustainable Impacts
The idea of sustainability can be thought about in many ways including sustaining components of the intervention; sustaining impacts after the program ends; and incorporating key ingredients of the intervention into other programs and services that exist within a community. Many participants in the webinar were interested in the idea of building capacity at the community level and facilitating an environment in which communities take ownership of programs, with the goal of sustaining impacts once the funding runs out. One of the participants mentioned that in her work “the team screening proposals for funding, element of sustainability was weakest. When I visited communities after few months the funding ended, was frustrated to find most of the activities had stopped. When I asked the people, they said it was “NGO project” not ours.” It is important to design programs that address a need that the community themselves recognize as something that matters to them. When there is a need that the community themselves recognizes as something that they want to address, they would perhaps be more likely to want to be a part of the design process, and as a result might not view the program as the “funder’s project”, but rather as their own project. The webinar participants also addressed the importance of unlearning our roles in the international development space, “Building ownership or behaviour change takes a long time because it’s about unlearning . For behaviour change in communities, the outsiders have to change. We have to shift as ‘experts’ with solutions to facilitators.” This shift in thinking of funders as facilitators of change rather than leaders of change shifts the dynamics of how programs are typically funded and run, and the power dynamics that are at play between a funder and the community. Often we think about co-designing projects with communities, but what if the area of interest of a funder doesn’t align with the community’s own interests? Who really gets to lead change?
A Timeline for Sustainability and Follow-up Evaluations
Another concept that was brought to light in our discussions was the idea of having a timeline for sustainability. One participant mentioned that often “there’s no sustainability ‘for how long’ plan”, and another noted that there is perhaps a need for “greater intentionality around exploring the mechanisms that support the achievement of outcomes; and consider how to sustain those mechanisms”. When thinking about sustainability, we need to identify what it is that we are we trying to sustain; the programs themselves or the impacts of the programs? What if the mechanisms change, do you still attribute the impact to a particular program? How do we account for the changing needs in a context? How is sustainability being measured? If certain components of a program that were considered to be “critical” still continue in a given context after the funding has run out, do we claim that the program was sustained?
As an example, a few years ago, TECCHI conducted an evaluation for a women’s entrepreneurship program that was funded by Ontario’s Poverty Reduction Strategy. The goal of the program was to help women start and run their own businesses, to help them increase their income. If, hypothetically, a few years after the program was concluded, a woman who had launched a business through the program was successful, and her business continues to grow and she has increased her income as a result, is this the only definition of a “sustained impact”? Suppose another woman realized that her business idea was not working out (for whatever reason – lack of a market for her product/lack of investment from external sources/ difficulty marketing etc.), and as a result she closed her business, and pursued something else. Is this really a failure of the program? In reality, not every business succeeds, and for some individuals having the opportunity to try opening a business, might be a personal success in and of itself (sense of fulfillment that you tried something that you always wanted to do, no matter what the ultimate outcome might be). Would this really be a failure of a sustained impact, just because her income didn’t increase as a result of the program? What about the experience gained from this exercise of opening a business? Wouldn’t it be worse to continue to invest (time, money etc.) into a business that didn’t become profitable? Although a business that is successful and continues to bring in profits after the program has concluded would be the poster child of a sustainable impact for this project, isn’t there also value in the knowledge and skills gained by those individuals whose businesses failed to do as well? What really is a sustainable impact for this program? What would a timeline look like for sustained impacts?
A related concept that was discussed during the webinar was the idea of conducting follow-up evaluations. One participant questioned “If we are such great systems thinkers, why not ponder the question of what the systems reasons for not doing follow-up evaluations?”, while another chimed in saying that “Funding timelines and expectations are often sustainability killers even though they are project starters”. If you are going to conduct an evaluation years after the program has come to an end, you are hoping that remnants of the program you funded still exist in the community at some functional level. If a follow-up evaluation gets conducted, how are we defining a “successful program”, and how would our ideas of “success” change one year post intervention as compared to four years post intervention?
Planning exit strategies
When interventions are being designed, many different considerations are being taken into account – from macro level factors such as the political and social climate in which one is implementing a project, to meso level factors such as the institutional values of the organization running the program, their resource capacities and constraints, as well as micro level factors such as the demographics of the participants who will interact with the program, the character of the program staff they will interact with etc. However, when thinking about the end of a program it is not often that exit strategies are planned out, and it is perhaps important to create exit strategies a priori to work towards ensuring that impacts can be sustained, and progress can be built upon the change that has been made. One of the webinar participants described, “You know, we don’t kick our kids out of the house at 14, we slowly make sure they can handle money, handle a job, meet their school deadlines, etc. Maybe we should be planning exit strategies more intentionally in all programs, and in that period we would learn more about potential barriers to sustainability, what’s coming up that needs to be dealt with”. Intentionally creating an exit strategy with program participants is perhaps helpful in increasing transparency and helping community members seek other resources or mechanisms through which they might be able to sustain impacts, if they believe that the program was useful to them and brought about positive change. Exit strategies should leverage the connections that were built through the program, and this could include community members, volunteers, potential local funders, and program champions who would continue to bring energy to the program once the original funders have left.
While thinking about sustaining impacts, it is important for all parties present to be reflexive and recognize the roles they play, the limitations of their knowledge, and the tensions that might be present given the different interests of the players present. When evaluating programs in the international development field, it is important to view sustainability under a critical lens, and recognize that there might be tensions between funders/program implementers and the community when trying to sustain programs and their impacts once the funding runs out. On one hand, the program implementer’s job depends on a program working well in a given context. Oftentimes, programs only need to work well enough so that positive changes are observed while the funding is present. However, if programs work “too well”, and communities can sustain positive impacts on their own, they break the cycle of dependency on funders. How do funding organizations stay relevant when the communities they are working with have built the capacity to sustain impacts once the program has come to an end?
