Post Webinar Reflections on Sustainability Evaluation

What happens once a program comes to an end? How do program implementers help build a system’s capacity to sustain impacts? How do we measure sustained impacts and how do metrics change over time to respond to the evolving needs of communities? These were some of the questions that were raised by participants during TECCHI’s webinar on Sustainability Evaluation. While the answers to these questions are complex and very dependent on the context of the program, this blogpost aims to reflect on some of the rich discussions that took place during the webinar that might offer some insights to these questions. However, this post might leave you having more questions than answers. Below are a few comments from the webinar participants that stood out to me:

Community Ownership and Sustainable Impacts

The idea of sustainability can be thought about in many ways including sustaining components of the intervention; sustaining impacts after the program ends; and incorporating key ingredients of the intervention into other programs and services that exist within a community. Many participants in the webinar were interested in the idea of building capacity at the community level and facilitating an environment in which communities take ownership of programs, with the goal of sustaining impacts once the funding runs out. One of the participants mentioned that in her work “the team screening proposals for funding, element of sustainability was weakest. When I visited communities after few months the funding ended, was frustrated to find most of the activities had stopped. When I asked the people, they said it was “NGO project” not ours.” It is important to design programs that address a need that the community themselves recognize as something that matters to them. When there is a need that the community themselves recognizes as something that they want to address, they would perhaps be more likely to want to be a part of the design process, and as a result might not view the program as the “funder’s project”, but rather as their own project. The webinar participants also addressed the importance of unlearning our roles in the international development space, “Building ownership or behaviour change takes a long time because it’s about unlearning . For behaviour change in communities, the outsiders have to change. We have to shift as ‘experts’ with solutions to facilitators.” This shift in thinking of funders as facilitators of change rather than leaders of change shifts the dynamics of how programs are typically funded and run, and the power dynamics that are at play between a funder and the community. Often we think about co-designing projects with communities, but what if the area of interest of a funder doesn’t align with the community’s own interests? Who really gets to lead change?

A Timeline for Sustainability and Follow-up Evaluations

Another concept that was brought to light in our discussions was the idea of having a timeline for sustainability. One participant mentioned that often “there’s no sustainability ‘for how long’ plan”, and another noted that there is perhaps a need for “greater intentionality around exploring the mechanisms that support the achievement of outcomes; and consider how to sustain those mechanisms”. When thinking about sustainability, we need to identify what it is that we are we trying to sustain; the programs themselves or the impacts of the programs? What if the mechanisms change, do you still attribute the impact to a particular program? How do we account for the changing needs in a context? How is sustainability being measured? If certain components of a program that were considered to be “critical” still continue in a given context after the funding has run out, do we claim that the program was sustained?

As an example, a few years ago, TECCHI conducted an evaluation for a women’s entrepreneurship program that was funded by Ontario’s Poverty Reduction Strategy. The goal of the program was to help women start and run their own businesses, to help them increase their income. If, hypothetically, a few years after the program was concluded, a woman who had launched a business through the program was successful, and her business continues to grow and she has increased her income as a result, is this the only definition of a “sustained impact”? Suppose another woman realized that her business idea was not working out (for whatever reason – lack of a market for her product/lack of investment from external sources/ difficulty marketing etc.), and as a result she closed her business, and pursued something else. Is this really a failure of the program? In reality, not every business succeeds, and for some individuals having the opportunity to try opening a business, might be a personal success in and of itself (sense of fulfillment that you tried something that you always wanted to do, no matter what the ultimate outcome might be). Would this really be a failure of a sustained impact, just because her income didn’t increase as a result of the program? What about the experience gained from this exercise of opening a business? Wouldn’t it be worse to continue to invest (time, money etc.) into a business that didn’t become profitable? Although a business that is successful and continues to bring in profits after the program has concluded would be the poster child of a sustainable impact for this project, isn’t there also value in the knowledge and skills gained by those individuals whose businesses failed to do as well? What really is a sustainable impact for this program? What would a timeline look like for sustained impacts?

A related concept that was discussed during the webinar was the idea of conducting follow-up evaluations. One participant questioned “If we are such great systems thinkers, why not ponder the question of what the systems reasons for not doing follow-up evaluations?”, while another chimed in saying that “Funding timelines and expectations are often sustainability killers even though they are project starters”. If you are going to conduct an evaluation years after the program has come to an end, you are hoping that remnants of the program you funded still exist in the community at some functional level. If a follow-up evaluation gets conducted, how are we defining a “successful program”, and how would our ideas of “success” change one year post intervention as compared to four years post intervention?

Planning exit strategies

When interventions are being designed, many different considerations are being taken into account – from macro level factors such as the political and social climate in which one is implementing a project, to meso level factors such as the institutional values of the organization running the program, their resource capacities and constraints, as well as micro level factors such as the demographics of the participants who will interact with the program, the character of the program staff they will interact with etc. However, when thinking about the end of a program it is not often that exit strategies are planned out, and it is perhaps important to create exit strategies a priori to work towards ensuring that impacts can be sustained, and progress can be built upon the change that has been made. One of the webinar participants described, “You know, we don’t kick our kids out of the house at 14, we slowly make sure they can handle money, handle a job, meet their school deadlines, etc. Maybe we should be planning exit strategies more intentionally in all programs, and in that period we would learn more about potential barriers to sustainability, what’s coming up that needs to be dealt with”. Intentionally creating an exit strategy with program participants is perhaps helpful in increasing transparency and helping community members seek other resources or mechanisms through which they might be able to sustain impacts, if they believe that the program was useful to them and brought about positive change. Exit strategies should leverage the connections that were built through the program, and this could include community members, volunteers, potential local funders, and program champions who would continue to bring energy to the program once the original funders have left.

While thinking about sustaining impacts, it is important for all parties present to be reflexive and recognize the roles they play, the limitations of their knowledge, and the tensions that might be present given the different interests of the players present. When evaluating programs in the international development field, it is important to view sustainability under a critical lens, and recognize that there might be tensions between funders/program implementers and the community when trying to sustain programs and their impacts once the funding runs out. On one hand, the program implementer’s job depends on a program working well in a given context. Oftentimes, programs only need to work well enough so that positive changes are observed while the funding is present. However, if programs work “too well”, and communities can sustain positive impacts on their own, they break the cycle of dependency on funders. How do funding organizations stay relevant when the communities they are working with have built the capacity to sustain impacts once the program has come to an end?