So many types of evaluations - where do I start?
Program evaluation
“Evaluation is a process that critically examines a program. It involves collecting and analyzing information about a program's activities, characteristics, and outcomes. Its purpose is to make judgments about a program, to improve its effectiveness, and/or to inform programming decisions.” (Patton, 1987)
Evaluation (well-designed and executed) helps us to make informed decisions. While both research and evaluation involve systematic inquiry, they differ in terms of their purpose, timing, generalizability, stakeholder involvement, and the use of findings.
Different types of evaluation
There are different types of evaluation such as Formative Evaluation, Summative Evaluation, Developmental Evaluation, Economic evaluation, etc. However, two common types of Evaluation frameworks are Formative evaluation and Summative evaluation. This figure is a visualization of formative and summative evaluation.
Formative Evaluation (Process/ Implementation Evaluation)
Formative evaluations can be done during program development and implementation of new programs. Formative evaluation ensures that a program or program activity is feasible, appropriate, and acceptable.
By using a Formative Evaluation, we focus on
Reliability: Were the program activities actually delivered?
Quality: How can activities/processes be improved?
Integrity: Are we doing what we think we’re doing? What are the strengths/weaknesses of daily activities? is the program complete, are missing an activity?
Efficiency: Are the timelines acceptable? Can we improve timelines and processes? Are there any context-related factors affecting performance? (things out of program’s direct control)
Summative (Outcome Evaluation)
Summative evaluations are completed once your programs are well established. It will tell you to what extent, the program is achieving its intended outcomes and will tell you should the Program be continued?
By using a Summative Evaluation, we focus on
Benefit-Cost: Is the program effective? Is it the best use of resources?
Effectiveness: what changes were made? did we meet benchmarks?
Efficiency: Did the program change behaviour as well as expected? If not, why not?
Evaluating Everyday Activities:
An example, the image below illustrates an evaluation of everyday activities, such as baking a birthday cake. This represents a comprehensive program evaluation, incorporating both formative evaluation (assessing input, process, and outcomes) and summative evaluation (focusing on the outcome).
Note: The best practice in program evaluation is to do a comprehensive evaluation and do Summative + Formative evaluation.
Questions: Why do we need both? To evaluate both process and outcomes
Developmental evaluation (DE)
This evaluation framework is based on systems thinking and facilitates innovation by gathering and analyzing real-time data to support informed, continuous decision-making throughout the design, development, and implementation process. This approach is especially useful for innovations where the path to success is uncertain. (Patton, 2010). By examining how a new approach unfolds, DE can help address questions such as:
What is emerging as the innovation takes shape?
What do initial results reveal about expected progress?
What variations in effects are we seeing?
How have different values, perspectives, and relationships influenced innovation and its outcomes?
How is the larger system or environment responding to the innovation?
Economic Evaluation:
This evaluation framework is a valuable tool that enables users to maximize resources, evaluate promising program options, and showcase the advantages of their program (WHO, 2023). Here are some questions to think about:
How do you know you’re making the most of your limited resources?
How do you decide between two promising program options when you can only afford one?
How do you demonstrate to decision-makers that the benefits of your program are worth the costs?
There are different types of Economic Evaluation such as:
Cost-Minimization Analysis (CMA): Compares costs of interventions that have already been proven to have equivalent outcomes.
Cost-Effectiveness Analysis (CEA): Compares costs relative to a single, natural unit of outcome (e.g., life-years gained, cases prevented).
Cost-Utility Analysis (CUA): A special form of CEA that uses quality-adjusted life years (QALYs) or disability-adjusted life years (DALYs) to compare interventions.
Cost-Benefit Analysis (CBA): Converts both costs and benefits into monetary terms to compare net benefits.
Cost-Consequence Analysis (CCA): Lists various costs and outcomes without aggregating them into a single measure, allowing decision-makers to weigh trade-offs.
Reference:
Patton, Michael Quinn. (2010). Developmental Evaluation: Applying complexity concepts to enhance innovation and use (Guilford Press, 2010).
Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). SAGE Publications.
WHO. (2023). Introduction to economic evaluation. Teaching workshop on a national program. Feb 2023.
Drummond, M. F., Sculpher, M. J., Claxton, K., Stoddart, G. L., & Torrance, G. W. (2015). Methods for the Economic Evaluation of Health Care Programmes (4th ed.). Oxford University Press
A little about the author:
I am Mandana Karimi, a Sociology Instructor at Capilano University and an Evaluation Specialist at Fraser Health Authority. I hold a Ph.D. in Political Sociology, an M.A. in Sociology, and a B.A. in Social Planning. My research and professional interests include Political Sociology, Environmental Sociology, Social Health, Policy Analysis, Critical Sociology, Ethnography, and Mixed Methods Research.
