TECCHI’s third webinar in the Sustainable Development Goals (SDGs) series was centered on the role that evaluations could play in measuring the coherence of interventions. Coherence can be defined as “how well an intervention fits” within a given setting. As a recent graduate, I’ve been surrounded by many classmates who are interested in the notions of disruption, innovation, and bringing new ideas to the solution space. However, too often our courses in university are structured so that students with similar academic backgrounds are in the same classes. Many times the lack of cross-faculty spaces for collaboration have come in the way of engaging with ideas using a lens of “coherence”. How do ideas generated in the field of health interact with those in political science, business and the environment? How do these different fields work together to solve problems in the real-world? The webinar on coherence helped me think more deeply about how ideas and interventions fit within a given context. Below are a few quotes that were mentioned in the chat during the webinar that stuck out to me:

“The relevance of formal vs, informal institutions and their overlay may need more attention.  Linking back to [the] interest in how this overlay will be relevant in coherence in a systems’ perspective as well.”
“I’d like to hear more about the coherence of coherence. Coherence with what, in what contexts? I’ve long identified limitations of silo thinking and the myth that any single intervention can have meaningful impacts, in isolation”
“I would also like to hear about the linkage of coherence with sustainability. Also in my opinion, measurement could be key specifically in situation wherein there are different partners and programmes work in tandem”
“’coherence’ isn’t always a ‘good’. […] Its a bit about the argument about institutionalisation as always good!? It depends on coherence for what purpose which I think is what you are implying”
“Decision processes are surely key interfacing formal and informal institutions (or culture)”
“In some humanitarian contexts, “coherence” with national policies and strategies are sometimes not desirable.”
“I would like to support [a] comment on the difference between policy design and evaluation and claiming the need for more inclusion of evaluators in policy design. Not only formative evaluation but the promotion of old fashion ex-ante evaluation which we don’t seem to see much nowadays. Also this relates to how do we cumulate knowledge in program evaluation as discipline… and the needs for evaluators to be less reactive and more proactive (not to that easy of course!)”
“I think it’s interesting that conversations about implementation and evaluation often happen separately – I am not an evaluator but an implementer and we talk about coherence in different language and in different ways. I think involving evaluators in implementation/program design and vice versa would be beneficial to address a lot of this issues with coherence”
“For me, an important part of coherence is the idea of resonance — the idea of adding to what already happening and gaining (non linear) momentum in the process. This links us back to the idea of contributing to trajectories we talked about a couple of weeks ago – trajectories consisting of systems of actors interacting with each other, technology, institutions, etc that deliver coherent sets of outcomes over time. The outcomes can be good or bad, and trajectories may work to resist change.”
“that’s why I was arguing that the choice of unit of analysis is key – I wold [sic] argue it’s the cluster of interacting polices we should be looking at!”
“Isn’t the point that we need to choose what to be coherent with, and being more explicit and transparent about that choice?”

Coherence with what?

When thinking about coherence, many participants in the webinar noted the importance of stating up-front what policies, programs, and values a new program aims to be coherent with. In any given context, there are formal institutions and informal actors present, these different groups might have values that are similar, but also those that are divergent. By stating which aspects of a given context a new program aims to align with, program implementers could move towards building connections with different partners (such as community organizations or policy actors) that are working on similar projects or program areas. These new relationships can then be leveraged to work towards shared goals and ambitions around solving a problem.

Fred Carden, the first speaker in the webinar, discussed different types of coherence – internal, intra-organizational, inter-organizational and international coherence. The idea of coherence can be thought about on many different levels, and perhaps as one aims to increase the scope of organizations and national priorities one wants their program to be coherent with, the difficulty of the task increases. With an increased number of stakeholders, organizations and policies to consider, a larger diversity in views and values are present at the table. This in turn increases the difficulty of designing and implementing a program that is coherent with the views of a broad range of factors.

Is coherence always desirable?

During the webinar, some participants posed questions that sought to understand when coherence is desirable, and when it is not. Are there certain aspects of a program that we hope are coherent with a given context, and other aspects that are aiming to bring a more disruptive energy to a solution space? When there is disruptive change, are appropriate change management tasks undertaken so that individuals are able to understand the change that is being brought, and how their role is shifting as a result? How can disruption and the notion of coherence be reconciled? Although coherence might not be desired in all aspects of a program, it is important to understand how the program fits in the larger socio-cultural, historical, environmental, political and economic contexts. How does the new program build on progress that is already made, and what new inputs does the program bring in? Are the users of the program ready to engage with the new change that is being proposed? Were the users part of the process while developing the new program?

Measuring Coherence

Measuring coherence is also another theme that was brought up multiple times during our discussion. Given that in any given context there are often multiple interventions taking place simultaneously, and that different programs and services synergistically build on each other, how do we measure if a program is coherent in a given context? How do you measure contributions when there are many partners involved? What about the unseen partners who might be playing a key role? Another question that comes to mind is the role of evaluation in looking at problems as a whole, and understanding how multiple programs funded by different organizations are working in tandem towards solutions. Is there value in funding evaluations that aim to understand how an ecology of policies, programs, and stakeholders at multiple levels work together towards solving an issue that is of importance to them?