Author: admin (Page 1 of 2)
Rethinking Evaluation Criteria At The Policy Level: Implications Of Inequities And Sustainability For Training Policy Evaluators
VIDEO
SLIDES
No slides available.
SPEAKER SUMMARY
Misbehaving Beneficiaries
- In India, a lot of money has been invested in building toilets, but roughly 40% of these newly constructed toilets are not being used because beneficiaries prefer to use outdoor facilities.
- Discussion around how to nudge beneficiaries into better behaviour, including idea of shaming people.
- But the context is not well understood
- Instead of seeing this as “misbehaving beneficiaries”, we might ask: What is it they are thinking about? Where do they come from? What exactly are they living through?
Mistrusting Beneficiaries
- There is no reason why beneficiaries should trust us – we are data collectors who come in from somewhere else. Especially in communities which are more vulnerable or marginalized, evaluators are often seen as being hand-in-glove with the government machinery.
- We need to think about ways to build rapport with the people whom we are trying to talk to and get information from. There may be different ways of doing this. But the idea that we are going into communities to simply extract data is not going to work.
Mistargeted Programs
- There has been a strong feeling that much of the vulnerabilities reside in rural areas; as such, many of the benefits and schemes have had more of a rural-focus than an urban-focus.
- But, when it came to the pandemic, it was the urban areas in India that suffered most. The benefit programs that were designed were in some sense mistargeted and missed out on what was happening during the pandemic in urban areas.
- Inequalities are not stable; they change and they are contextual. While there is a stable element to vulnerabilities, we also need to pay attention to the changing component of inequality.
Mistaken Strategies
- Many programs aim to incorporate people who are most vulnerable as program beneficiaries, but do not necessarily start from the perspective of the beneficiaries themselves.
- e.g., what are the changes that the program recipients would like to see in their own lives?
- Example of Eklavya Model Residential School and the use of the name Eklavya
- Different interpretation of the story of Eklavya:
- Some see it as a great story about reverence towards the educator and teacher; others see it as an example of the inequalities that educational institutions can perpetuate, especially in regards to vulnerable child
An Evaluator’s Journey And Response: Responding To Inequities, Vulnerabilities And Answering Sustainability During COVID 19
VIDEO
SLIDES
SPEAKER SUMMARY
Reflecting on COVID-19 in India – What evaluator competencies are most relevant?
- Empathy
- When we are talking to frontline workers, we need to be appreciative of the work that they are doing, give them time and space, building perspective, showing warmth when we are interviewing, listening.
- Disruptive in a disruptive world
- Using technology as an enabler (e.g., to reach out to communities, to find more convenient times to reach people, such as after office hours).
- Methodology and data multiplicity
- Multiple data points add value
- Relational
- Helps in having everyone on board – convincing that evaluation can add value even in times like these.
- Collaboration
- Helps in pooling resources in the context of a resource-constrained environment; helps in building multiple perspectives.
Who’s Priorities? Reflections from Ebola and Covid Contexts in DRC
VIDEO
SLIDES
Loading…
Reload document 
Open in new tab SPEAKER SUMMARY
- Lessons learned from working in the Ebola context and the COVID context in the Democratic Republic of the Congo – Key considerations when we are evaluating programs to look at sustainability and equity.
- 7 pillars in an MSF outbreak response – the number one step: discussing and engaging with community (we need to make sure that the community is implicated and have all the necessary information before moving on to the next pillars of outbreak response).
- We need to ask: What are the priorities of the people and communities we are designing and setting up interventions for? And then ask: Do you want our help in responding to those priorities? In the case of the Ebola outbreak in the DRC, we learned that Ebola was not the main priority of the communities at the time. The priorities were building wells and having access to clean water. Similarly with COVID, there were different priorities in the community at the time.
- Example of community-driven Ebola isolation centres set up in people’s own communities. This made a huge different because people had ownership in these isolation centres – people actually went to the centres and the outbreaks stopped.
- We will never have sustainability if we don’t have pertinent programming. And pertinent programming only comes from responding to priorities. And we will never know the priorities if we don’t engage with the community.
Epidemic Illusions: On The Coloniality Of Global Public Health
VIDEO
SLIDES
Loading…
SPEAKER SUMMARY
- Epidemic Illusions: On the Coloniality of Global Public Health
- Catalogues how what we think of as value-neutral, objective science actually commits symbolic violence.
- The modern aid complex is built on the idea that descendants of settler colonialists are morally related to people in the global south as potential helpers and saviours. But when you see that we are really beneficiaries and supporters of a global institutional order that systematizes their oppression, then the intervention actually changes from a failed aid development model to one of a reparative justice model.
- Symbolic Violence or Epistemic Injustice
- The capacity to impose the means for comprehending and adapting to the social world by representing the economic and political world in disguised, taken-for-granted forms.
- Words do symbolic violence (e.g., outbreak, superspreader). We need to really interrogate the categories that we use as part of our impact evaluations to see what kinds of ideological work that they are doing.
- Reparative Justice
- Reparations for American descendants of persons enslaved in the US and their potential impact on COVID-19 transmission – shows how eradication of the wealth gap between black and white Americans could have decreased COVID-19 by 30-90%
Rethinking Evaluation Criteria: Implications Of Inequities And Sustainability For Training Evaluators
VIDEO
SLIDES
SPEAKER SUMMARY
Rethinking evaluation criteria in light of issues of inequities and sustainability – additional criteria that are needed:
- Reach
- The share of members of the target group involved in an intervention is often very limited. In reality, we are often only reaching a tiny fraction. This means that the benefits of a project are also very limited.
- Potential for Local/National Replication
- To what extent can the model introduced by the intervention be replicated by national/local institutions? For example: projects provide expensive equipment to locals without sufficient attention to sustainability.
The “Other” Competencies for Effective Practice & Pandemic Response
VIDEO
SLIDES
SPEAKER SUMMARY
- Inside the mind of an evaluation practitioner – the difference between theory and actual practice
- Impact of pandemics, crises, and the rapidly changing context to evaluation practice
- Disruption to normal processes
- Uncertainty about the future
- Discontinuity of core evaluation services
- Inequalities unearthed or exacerbated
- Redundancy as some services become unneeded
- Implications:
- Evaluation practice demands more than conventional set of competencies
- Competencies previously thought of as peripheral become CORE
- Evaluator Training must respond to need for additional skills
- Evaluator Training must build Broad Based Capabilities
- Three broad domains of additional competencies – REFEREE skills:
- Relational Competencies
- Enabling Competencies
- Foundational Competencies
Evaluation Competencies in the Ages of Discontinuity
VIDEO
SLIDES
SPEAKER SUMMARY
Five competencies:
- Engage the broader architecture of evaluating practices
- Need to consider alliances with other applied researchers that are engaged in activities of valuing, judging and recommending.
- Develop capacities to work in transdisciplinary/interdisciplinary teams. Working with a variety of different kinds of knowledge producers, and blurring boundaries between practices to engage integrative knowledge to address complex problems.
- Do collaborative knowledge work
- Participatory approaches. Experiential and the relational understanding of practitioners who are engaged with situations that they themselves are a part of.
- Learning to work in uncertain futures, deal with contingent scenarios, and address issues of the wellbeing of underprivileged people marked by social and political change.
- Expanded notions of collaborative knowledge work include things like: collective social learning, co-production, collaborative adaptive management, and triple loop learning.
- Expand the repertoire of questions evaluators ask
- Focus tends to be on questions of what works, effectiveness, efficiency and impact, and to some extent sustainability.
- Need to ask questions like: What assumptions underlie our understandings of the problem we are addressing and our efforts to address it? Who gains and who loses from what we plan to do or have done? What should we do to address potential exclusion and marginalization of peoples in our activities?
- Develop epistemic fluency
- We work on real world problems and to do so we require different combinations of specialized and context dependent knowledge – different ways of knowing.
- Develop ethical and political fluency
- Developing ethical fluency involves developing moral expertise and capacity for normative analysis. It is the competency to state and clarify moral questions, and provide justified answers to those questions.
- Moral expertise involves conceptualizing and elaborating on the meaning of norms, values and ends that are at stake in a particular intervention.
- Developing political fluency means that evaluators focus on the political dimensions of acting and learning, as well as learning to deal with policy discord and moral disharmony.
Current training in evaluation largely assumes that evaluators are dealing with values in a world of facts. We need far more attention to value dimension of things than the factual dimension.
Rethinking Evaluator Competencies in an Age of Discontinuity
VIDEO
SLIDES
SPEAKER SUMMARY
- Diverse competencies on our teams are extremely important – great for professional development, as we learn from each other on the job
- Our response to the manifestation of discontinuities in our work and the competencies required:
- Change approach mid-project – Developmental Evaluation vs. Implementation and Outcome Evaluation. Understanding different approaches was critical.
- Change role – Facilitator and Problem-solving partner, sometimes Activist vs. Evaluator and Subcontractor. Process competencies become very important.
- Change focus – What is the big picture? What else is happening vs. what is the project trying to achieve? Requires understanding of systems theory + complexity.
- Get comfortable with uncertainty. Requires a resilient disposition comfortable with constant change and chaos.
- Developing evaluation competencies is a systems thing!
- What changed with capacity development?
- VOPE activities moved online – Networking less effective, circles narrowed.
- Academic programs with some contact time moved fully online – Peer learning and networking diminished.
- Staff in organizations worked more offsite and more discontinuous – Mentoring and on-the-job training reduced.
- Building teams tended to be with established relationships – Referrals to “weak ties” reduced.
- Erosion of the settings for capacity building. This disruption is going to continue for a longer time. Our evaluation systems are going to become more fragile. This has implications for equity and diversity.
- Implications for training evaluators in 2021 and beyond?
- Resilience and Adaptability, at individual as well as system level.
COVID, Complexity, Counterfactuals, & Calamitous Conclusions: A Provocation
VIDEO
SLIDES
SPEAKER SUMMARY
- Limits of social suppression interventions exposed in complexity theory – 8 modes:
- Disparate Command and Control Systems
- Interaction and Emergence
- Policy Discord and Moral Disharmony
- Contextual Heterogeneity
- Implementation Heterogeneity
- Ambiguity in Relations and Guidelines
- Temporal Change in Public Attitudes
- Exit and Sustainability Effects
- A Calamitous Conclusion?
- No public policy has ever been subject to more effort, management, investment, and scrutiny than the social interventions to overcome COVID-19
- Yet, only a halting intermittent solution was provided
- Complexity dynamics and the oscillating impact of major policy interventions: Have we seen the merry-go-round before?
- Is there light at the end of the tunnel?
- For policymakers: Remember you are designing complex, adaptive, self-transformative systems. The key task is to try to anticipate the complexity dynamics.
- For evaluators: Remember you are researching complex, adaptive, self-transformative systems. The key task is to try to trace the complexity dynamics.