So many types of evaluations - where do I start?

    Program evaluation

“Evaluation is a process that critically examines a program. It involves collecting and analyzing information about a program's activities, characteristics, and outcomes. Its purpose is to make judgments about a program, to improve its effectiveness, and/or to inform programming decisions.” (Patton, 1987)

Evaluation (well-designed and executed) helps us to make informed decisions. While both research and evaluation involve systematic inquiry, they differ in terms of their purpose, timing, generalizability, stakeholder involvement, and the use of findings.  

    Different types of evaluation

There are different types of evaluation such as Formative Evaluation, Summative Evaluation, Developmental Evaluation, Economic evaluation, etc. However, two common types of Evaluation frameworks​ are Formative evaluation and Summative evaluation. This figure is a visualization of formative and summative evaluation.  

Formative Evaluation (Process/ Implementation Evaluation)

Formative evaluations can be done during program development and implementation of new programs. Formative evaluation ensures that a program or program activity is feasible, appropriate, and acceptable.

By using a Formative Evaluation, we focus on

• Reliability: Were the program activities actually delivered? 

• Quality: How can activities/processes be improved? 

• Integrity: Are we doing what we think we’re doing? What are the strengths/weaknesses of daily activities? is the program complete, are missing an activity? 

• Efficiency: Are the timelines acceptable? Can we improve timelines and processes?   Are there any context-related factors affecting performance? (things out of program’s direct control)

Summative (Outcome Evaluation)

Summative evaluations are completed once your programs are well established. It will tell you to what extent, the program is achieving its intended outcomes and will tell you should the Program be continued? 

By using a Summative Evaluation, we focus on

• Benefit-Cost: Is the program effective? Is it the best use of resources?

• Effectiveness: what changes were made? did we meet benchmarks?

• Efficiency: Did the program change behaviour as well as expected? If not, why not?

Evaluating Everyday Activities:

An example, the image below illustrates an evaluation of everyday activities, such as baking a birthday cake. This represents a comprehensive program evaluation, incorporating both formative evaluation (assessing input, process, and outcomes) and summative evaluation (focusing on the outcome).

Note: The best practice in program evaluation is to do a comprehensive evaluation and do Summative + Formative evaluation.

Questions: Why do we need both? To evaluate both process and outcomes

Developmental evaluation (DE)

This evaluation framework is based on systems thinking and facilitates innovation by gathering and analyzing real-time data to support informed, continuous decision-making throughout the design, development, and implementation process. This approach is especially useful for innovations where the path to success is uncertain.  (Patton, 2010). By examining how a new approach unfolds, DE can help address questions such as:

• What is emerging as the innovation takes shape?

• What do initial results reveal about expected progress?

• What variations in effects are we seeing?

• How have different values, perspectives, and relationships influenced innovation and its outcomes?

• How is the larger system or environment responding to the innovation?

Economic Evaluation:

This evaluation framework is a valuable tool that enables users to maximize resources, evaluate promising program options, and showcase the advantages of their program (WHO, 2023). Here are some questions to think about:

• How do you know you’re making the most of your limited resources?

• How do you decide between two promising program options when you can only afford one?

• How do you demonstrate to decision-makers that the benefits of your program are worth the costs?

There are different types of Economic Evaluation such as:

• Cost-Minimization Analysis (CMA): Compares costs of interventions that have already been proven to have equivalent outcomes.

• Cost-Effectiveness Analysis (CEA): Compares costs relative to a single, natural unit of outcome (e.g., life-years gained, cases prevented).

• Cost-Utility Analysis (CUA): A special form of CEA that uses quality-adjusted life years (QALYs) or disability-adjusted life years (DALYs) to compare interventions.

• Cost-Benefit Analysis (CBA): Converts both costs and benefits into monetary terms to compare net benefits.

• Cost-Consequence Analysis (CCA): Lists various costs and outcomes without aggregating them into a single measure, allowing decision-makers to weigh trade-offs.

Reference:

• Patton, Michael Quinn. (2010). Developmental Evaluation: Applying complexity concepts to enhance innovation and use (Guilford Press, 2010).

• Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). SAGE Publications.

• WHO. (2023). Introduction to economic evaluation. Teaching workshop on a national program. Feb 2023.

• Drummond, M. F., Sculpher, M. J., Claxton, K., Stoddart, G. L., & Torrance, G. W. (2015). Methods for the Economic Evaluation of Health Care Programmes (4th ed.). Oxford University Press 

A little about the author:

I am Mandana Karimi, a Sociology Instructor at Capilano University and an Evaluation Specialist at Fraser Health Authority. I hold a Ph.D. in Political Sociology, an M.A. in Sociology, and a B.A. in Social Planning. My research and professional interests include Political Sociology, Environmental Sociology, Social Health, Policy Analysis, Critical Sociology, Ethnography, and Mixed Methods Research.

Introduction to Needs Assessments

“Our program is meeting its stated goals, but our clients are still struggling!”

Have you ever experienced this phenomenon? Your program works as designed – your clients are experiencing the outcomes you’ve decided were important to measure. However, your clients are still struggling.  

Don’t worry. You are not alone!

Needs assessments are critical tools in the toolbelt of any nonprofit or government organization that is responsible for designing programs that serve the needs of a group of people.

A needs assessment is an evaluation tool that helps fill gaps between what is currently happening and your desired outcome. For example, let’s say you manage a nonprofit that serves individuals experiencing mental illness. Your typical program consists of offering free counseling to those who need it. Your clients attend counseling sessions every week and meet their socioemotional outcomes; however, they still cannot maintain a living wage job, so they struggle with maintaining safe and stable housing and obtaining groceries every week.

Currently happening: counseling.

Desired outcome: to live independently.

How do we know what we are missing? It is clear that counseling does not lead your clients to lead independent lives, so what else needs to happen?

Here is where a needs assessment is crucial.

A needs assessment is a type of evaluation that collaborates with the community (your clients or potential clients) to determine their challenges and how to solve them. There are several methodologies to achieve these results.

Surveys, interviews, focus groups, and observations are typical methodologies to glean information for the needs assessments. The key is to gather as much information as possible (from diverse methodologies) about what your population is struggling with and what they think they need to overcome that challenge. Oftentimes, we assume that we are the experts and know what is best – but in fact, the community is the expert in their own experience. Too often, we ask their challenges without asking what solutions they hope to see. Communities often know exactly what their challenges are and exactly what will best support them – they need someone who will co-create those types of programs with them.

When surveying, interviewing, or facilitating focus groups, collect demographic information. Then, compare responses across various demographics. You want to know if all or most people in a certain demographic are answering in certain ways. Are all of your white clients having a different challenge than all of your Black clients? Perhaps a solution that would work in your white American community would not work as well in your immigrant community.

When designing your needs assessment project, consider the following:

1.        Create an ad hoc advisory board. This board will help you draft questions, recruit study participants, and make sense of the results. Having an advisory board (made up of people from the community) ensures you make the right assumptions based on the information you are given. When analyzing data from your perspective, you will likely design a program that looks vastly different than when someone with lived experience analyses the same dataset. Experiences matter and are crucial in the design and analysis process. Your job is not to have the answer but to elevate the voices of those who have the answers.

2.        Triangulate your data by engaging in multiple research methodologies. Use surveys, interviews, focus groups, and observation together. When you have multiple methodologies and the same information is being repeated across each method, you know it is an important feature in your dataset. Stagger your methods so they work with and not against one another. You could conduct a series of interviews or focus groups first to glean key information from important stakeholders, then use that information to design a survey to go out to the masses. Or you could design a survey to go out to the masses to see where people are in general terms, then use those survey responses to design your interview or focus group questions. Either strategy is excellent, depending on the goals.

3.        Consider who you engage in each type of methodology. Understand that you obtain rich, in-depth details about someone’s life with interviews. A focus group gathers details (but not in-depth) on multiple people simultaneously. Focus groups are excellent when gathering data from multiple perspectives; remember that the data may not be as detailed as you need. Sometimes, it is a good idea to follow up a focus group with a few in-depth interviews with people from the focus group. Additionally, the questions you ask will determine your methodology. You don’t want to ask sensitive or potentially embarrassing questions in a focus group. Finally, you want the most detailed and rich information from those most important to the issue. Consider interviewing those you think are key to the issue and not necessarily the most important based on their title. For example, when determining what barriers someone with a mental illness has to employment, you may want to ask someone unemployed rather than asking a CEO who makes no hiring decisions their perspective.

Regardless of the topic, there are some key questions I always like to ask that help set the stage.

1.        What does an average day look like for you now?

2.        What would your dream day look like for you?

3.        What is missing between your average day and your dream day?

4.        What are some of the biggest barriers you face regarding (organizational mission)?

5.        What would that look like if you could design a solution that would alleviate some of your biggest challenges?

Overall, needs assessments provide rich information on gaps between what you are currently doing and what needs to be done based on the perspective of the community you serve. Multiple methodologies and strategies will help you on your journey. The key to remember is community engagement – do not make decisions without community involvement.

If you are interested in conducting a needs assessment but don’t know where to start, contact me at Jodie@ChangeAmplifiers.Com

How Evaluation Supports Program Design and Implementation

As nonprofit leaders, we continuously design or modify programs to better serve our clients. The question becomes: “How do we design programs so they meet the direct needs of our clients?”

The answer to this lies in the program design, implementation, and evaluation cycle. This cycle begins with conducting a “needs assessment.” Based on the “needs assessment,” the program can be designed, and funding can be secured. Once funding is received, program implementation can begin. Finally, at regular intervals, program evaluations occur.

The “needs assessment” is the foundation of this whole process. A “needs assessment” engages potential clients to hear their biggest challenges and potential solutions directly from them. The key to this phase of the cycle is engaging the community. So often, nonprofit organizations skip this step because they think or assume they know the community's needs. The community tends to know what they need – they just need to be empowered to ask for it.

Let’s simplify the example of housing. A nonprofit organization can assume that if there is a housing crisis, people need housing – so they design a program that builds houses. However, after building homes, the organization finds no one buys them. Why?

When asking the community what they need, they say their biggest need is employment to afford housing. Building new houses won’t solve the problem because families do not earn enough to buy a home. However, a program offering employment training services may lead to families having the ability to stabilize their housing situation. This example helps demonstrate that the community is the expert in knowing what they need. So, just ask!

Now that you know what the community needs, it is time to design the program. Since the community knows what they need, create an advisory board of community members to guide the program design, clarify issues that may arise, and build buy-in from the community. Ensure you draft a program design – often in the form of a Theory of Change or Logic Model (more to come on that in March!)

Your advisory board and you have designed a program that could work for your community. Now what?!

Securing funding is a critical component of the cycle. As nonprofit leaders, you fully grasp the importance of funding. It is more difficult to secure funding for a new pilot program than for an established program with results. There are several grant-makers, however, that will support pilot programs. Often, smaller, family-owned foundations will support pilot programs if they are passionate about the program's goal. Another funding avenue is “major gifts” fundraising. Individual philanthropists may feel passionate about your program and will donate substantial financial capital to support it. You may have to apply for multiple grants or seek donations from multiple philanthropists to secure enough funding to implement your program. This is where your program design (your Theory of Change or Logic Model) will come in handy. Grants (and philanthropists, to an extent) want to see that you’ve thought about what your program will do and how it will do it. Use your Theory of Change or Logic Model to work for you.

Finally, a major donor who is particularly passionate about your new program has come forward. Now is the time to implement the program. Implementing programs is what we do every day – we know we need to hire staff, recruit clients, and begin performing services.

After conducting services for a year, you are interested in determining if your program is working. Now is the time to conduct an impact evaluation (more to come in April!) It is best practice to evaluate the program annually to determine if modifications must be made. Additionally, having results is a plus when it comes time to apply for more grants or seek more donations! You can conduct the evaluation internally, using these blogs to offer support and guidance, or hire an external evaluator with expertise in conducting needs assessments and evaluations. Tips and tricks on working with an external evaluator will come in June!

In summary, evaluation is critical to the program design and implementation cycle. It ensures you are designing programs that truly meet the complex needs of the communities you serve; it allows you to see your impact, and, of course, it helps you secure additional funding. Use this blog series over the course of this year to develop your evaluation program! Reach out to me at Jodie@ChangeAmplifiers.Com for questions about all things evaluation!