How to teach things like judgment, discernment, political navigation, and balancing alternative value sets that might be applied to an evaluation.
There is benefit in adopting an historical lens that has us look at past evaluations and learn from past evaluators, both on the positive and negative side. Importance of evaluation history in who evaluators are.
In addition to simulation, we could think about modelling in our training of evaluators.
Example of Dr. Edmund Gordon and the Head Start educational program in the U.S.
Helped with broadening the view of the kinds of things that ought to be measured as possible outcomes of Head Start – beyond IQ scores to socio-emotional measures like self-esteem.
Follow-up assessments showed the initial benefits of Head Start dissipated over time because they weren’t further supported.
Another program came in – the lessons were lost, the breadth of those measures that needed to be taken in the follow-up program.
Implications of the pandemic for rethinking evaluator competencies
When people’s lives are at risk, evaluation may seem, or even be, less important
Traditional ways of conducting evaluations simply don’t work
Everyone is an Evaluator
An evaluator’s job is to help build on what exists in a setting
The power of people in community
Free Range Evaluation
Commitment to building people’s capacity to do and use evaluation
Funders rarely interested in paying for capacity building; they typically support accountability and Western “science”
Implications for “training” evaluators
(Note: there is a difference between education and training)
Enter every evaluation context with humility and an openness to learning (add to this: curiosity, nimbleness, and empathy)
Be willing NOT to be the expert. Not everyone can do this – unlearning is required. Raises questions about how we select people to become evaluators. Are there some people who have an innate sense and ability?
Foster, support, and believe in evaluation capacity building
Build on the knowledge systems people already have in place (Indigenous knowledge)
How to work in communities
You need the support of positional leaders
Identify, support, collaborate with evaluation advocates in situ (do not call them evaluation champions – this is seen as competitive)
Develop/teach/learn from evaluation liaisons – community members who learn enough about evaluation to engage others, advocate, and support the process
A key requirement: commitment over the long haul
This is not how most contracts are written
Must be able to respond to evolving contexts/crises
Suggests why teaching community members may be a cost-effective way to proceed
“Start where you are. Use what you have. Do what you can.” – Arthur Ashe
We often get caught up in the noise and lose sight of the essential things
A Story that Highlights Getting Caught up in the Noise
In 2014, City of Mosul in Iraq was attacked and taken over by IS. For next 2 years, the 1.4 million people in Mosul lived under the control of IS.
In 2016, a coalition led by the Iraqi army started a battle to take back control of Mosul. It was an extremely vicious battle that ended up destroying all the healthcare facilities within the city.
Coalition forces slowly started taking over one neighbourhood at a time. And while the people in those neighbourhoods were no longer under IS control, they couldn’t leave the city because the government hadn’t set up a system of who did/did not belong to IS.
The situation: absolutely no healthcare in the city; people unable to leave even when under the coalition care; coalition forces had a curfew in the areas that they controlled from 5pm until 7am and would shoot indiscriminately at anyone in the streets during that curfew period because there was a lot of fear.
When the coalition forces controlled about one-quarter of the city, three of us from Doctors Without Borders went into the city to assess needs.
Found 500,000 people that could not leave the city that had no access to healthcare. Heard stories of women dying because they had no access to cesarean sections.
We met a man who wanted to turn an old retirement home into a hospital but could not get any materials. Working together, we provided materials and he provided the human resources and we built a hospital – very low-profile to avoid being attacked.
Within 3 days we had an emergency room, within 1 week a whole in-patient department, and within 3 weeks we were doing cesarean sections.
Some Sort of Magic
This initiative stayed in the heart of everyone involved – there was some sort of magic.
Never ended up doing a formal evaluation because nobody could decide what to evaluate. They would get caught up in either the number of surgeries that were done (which wasn’t the important part) or in the fact that we provided health care in bulletproof jackets and helmets (which also wasn’t the important part).
The magic was that it was needed. It fit the priority of the people. Even though they were so traumatized, they could come together.
Interested in how people are programmed. It takes a long time to program a racist person. It takes a long time to program a neoliberal technocrat.
Kids don’t start out saying: I want to grow up to be a neoliberal technocrat. But that is what they likely would become, despite our best intentions, because it is what the culture of white supremacy does in the U.S.
How do we cut through those things? How do we combat racism? How do we become anti-colonialist? These are the things that are important in any interpretative work we do – whether it is an evaluation of a program or an evaluation of why the Ebola outbreak was so big in the Congo in 2019.
How Colonial and Imperial legacies have led to a structured disposition that leads people to act in the ways they do. How do we get people to understand these structured dispositions? How do we get people to understand the colonial mechanisms that are still built into what we do?
List of 10 books – a mini syllabus
1) The Wretched of the Earth, Frantz Fanon
2) Infections and Inequalities: The Modern Plagues, Paul Farmer
3) Neo-colonialism: The Last Stage of Imperialism, Kwame Nkrumah
4) The Anti-Politics Machine: Development, Depoliticization, and Bureaucratic Power, James Ferguson
5) Bad Samaritans: The Guilty Secrets of Rich Nations & the Threat of Global Prosperity, Ha-Joon Chang
6) The Divide: Global Inequality from Conquest to Free Markets, Jason Hickel
7) The Congo: From Leopold to Kabila: A People’s History, Georges Nzongola-Ntalaja
8) White Plague, Black Labor: Tuberculosis and the Political Economy of Health and Disease in South Africa, Randall M. Packard
Evaluation has become a standing operating procedure.
Evaluation machine – (Peter Dahler-Larsen) a situation where the regime wants evaluation even if it is completely unnecessary.
Evaluation may not be helping problems, but actually creating problems.
Evaluiitis – a new disease spreading feverishly, where everything is being evaluated.
Performance paradox – in public arena, the more you invest in evaluations and systemic evaluations all the time, cannot guarantee that the policies and programs in the organization are the most effective ones.
Incorporating these insights into evaluation training
Learning when these trends take place
Discuss and assess these trends and factors
Think about how to “fight” them
Be resilient
Evaluator’s Resilience
Resilience is not only a cognitive-intellectual thing; it is also an in-depth ethical-behavioral approach/style of evaluators – strength in face of organizational difficulties, stress, (soft) power plays, etc., to fight and bounce back, which helps the profession realize goals such as:
Collaborating with stakeholders/commissioners while also challenging and criticizing them;
Being able to navigate between the demands of evaluation clients and needs of a valid, credible scientific perspective in the evaluation;
Knowing how to deal with fake news, cancel culture, and fake handbags (like “selling” Logframes for ToCs).
Examples of insufficient or limited evaluator’s resilience:
Doing 1000s of implementation studies finding failures, blunders, etc., but continuing to do them because they are requested and paid for;
Claiming to work and test theories of change, while in fact believing them and searching for verification of them;
Accepting a stream of “Say-do-behaviors” by Parliaments, governments, agencies, etc.