Impact Evaluation methods and tools

Measuring the change that matters

Large numbers of public and private programs are funded and implemented each year, with the aim of improving socio-economic well-being. Think for example of health, education and infrastructure interventions implemented by governments; advocacy campaigns run by non-governmental organizations; or innovation programs launched by private firms. Once they are completed, it is relatively easy to work out the resources they took up and what outputs they produced. But can we answer what difference the program made and why?

Quantifying and explaining effects are not only recurrent elements in socio-economic development and international aid programs. They are also becoming essential features of private sector initiatives. Measuring social impact is now as important for private companies as it is for non-profit and public institutions. Private companies also need to know the footprint they have on society, beyond accounting for profit and loss.

Knowing what difference a program makes is also a valuable input into evidence-based decision-making, both in developing public policy and in private sector initiatives. Impact evaluation generates robust evidence on what works, what doesn’t work, why, for whom and in which contexts. This is crucial not only to assess the significance of the change brought by your program but also to improve it.

This impact evaluation course will help you develop the theoretical and practical foundations to answer these crucial questions. It will guide you through counterfactual impact evaluation, the approach used to examine whether a program made a difference and how much of this can be attributed to the program. The course will also walk you through theory-based impact evaluation, the approach used to find out why a program worked or did not work.  After completion of this course, you will be equipped with a set of tools that you can confidently use to quantify and explain the effects of a program.

Program structure 

Monday Tuesday Wednesday Thursday Friday
09.00 - 10.30

*Course introduction and presentations

*Overview of the course

*What is impact evaluation?

*The demand for impact evaluation

*The uses of impact evaluation

*The evaluation challenge: attribution and selection bias

*The problem of the counterfactual

*Experimental versus quasi-experimental designs

*Propensity score matching. Worked-through examples

*Methodological challenge of CIE and how to address them

*Realist evaluation. Worked-through examples.

*Case study review

*How to implement and impact evaluation (choosing the right method)

*How to manage and impact evaluation (team, timings, budget)

10.30 Coffee break Coffee break Coffee break Coffee break Coffee break
10.45 - 12.30

*Approaches to causality

*Causal pathways and theory of change

*Worked-through examples

*Counterfacutal Impact Evaluation (CIE)

*Randomized Control Trials (RCT). Worked-through examples

*Case study review

*Group assignment (team work and plenary presentations)

*Qualitative comparative analysis (QCA). Worked-through examples.

*Case study review

*Ethical issues in impact evaluation

*Quality aspects and tips to mitigate common risks

13.00 - 14.00 Lunch break Lunch break Lunch break Lunch break Lunch break
14.00 - 15.30 *Group assignment (team work and plenary presentations)

*Instrumental variables technique. Worked-through examples.

*Regression discontinuity. Worked-through examples

*Theory-based impact evaluation

*Contribution analysis. Worked-through examples.

*Plenary debate (based on readings)

*Group assignment (team work)

*Impact measurement in the private sector

*Evolution and trends

*Methodologies to assess social impact

15.30 Coffee break Coffee break Coffee break Coffee break Coffee break
15.45 - 17.00

*Overview of evaluation designs, methodological approaches and quantitative and qualitative tools for impact evaluation

*Sampling and power calculation

*Difference-in-differences estimation. Worked-through examples

*Case study review

*Process tracing. Worked-through examples

*Case study review

*Group assignment (Plenary presentations)

*Impact measurement frameworks: Worked-through examples

*Closing remarks

Course objectives

  • Develop a robust, evidence-based answer on what difference your program made, and why
  • Understand when and how to conduct impact evaluations
  • Understand the design and implementation of an impact evaluation
  • Use insights obtained from impact evaluations to inform policy decisions

Course topics

  • What is impact evaluation, why are impact evaluations important? Uses and demand for impact evaluation
  • Practical aspects of causality, attribution and the counterfactual
  • Overview of evaluation designs, methodological approaches and tools for impact evaluation (both quantitative and qualitative)
  • Counterfactual impact evaluation methods
  • Theory-based impact evaluation methods
  • How to implement and manage impact evaluations
  • Quality, risks and ethical aspects of impact evaluations
  • Impact measurement in the private sector:  evaluation and impact investing

Teaching methods

The approach will be highly interactive. Lecturer presentations on the topics will be combined with question and answers, debates and plenary discussions. Teaching methods will also feature worked-through examples, case study reviews and group assignments. Assignments will consist on group work followed by team presentations and plenary discussions.

Your profile

The course is designed for public policy makers and policy planners; staff of not-for-profit organizations involved in monitoring and evaluation; and private sector company staff interested in measuring the social impact of their programs. However, it is also open to anyone with interest in unpacking, measuring and defining what difference their program made and why.

Certificate/Diploma

Each participant receives a certificate of attendance on the last day of the course. Only for those who will complete an additional assignment will receive a diploma. In case you would like to have more information you contact ep@msm.nl