Welcome to the e-learning lesson on developing a plan for outcome measurement. The world of evaluation uses countless words and varying terminology to describe the process of evaluating outcomes. For the purpose of this e-learning, we will be discussing outcome measurement as a means of exploring the impacts or results of a program or initiative. An outcome is a measurable and observable change in individuals, groups, organizations, systems, or communities. Outcome measurement is a systematic way to assess the extent to which a program has achieved its intended results. The main questions addressed in outcome measurement are: What has changed in the lives of individuals, families, organizations, or the community as a result of this program? Has this program made a difference? How are the lives of program participants better as a result of the program?
By the end of this lesson, you will be able to: understand how to identify measurable outcomes and outcome chains, develop a logic model, and identify performance indicators and performance targets to support the outcome measurement process.
Outcomes communicate value.
Outcomes are the reason nonprofit organizations strive to deliver programs and build their capacity. Most importantly, however, proper evaluation techniques provide your organization with proof of their value to existing funders, potential funders, and the larger community. Whether this value is communicated in dollars or the number of individuals served, quantifiable performance measures are becoming important in the increasingly competitive social service industry.
Successful outcome measurement hinges on a cohesive plan.
Preparation and planning is a key part of the outcome measurement process. The better the planning, the more impact the outcome measurement will have on your organization and your organization's bottom line. By the conclusion of this lesson you will be able to:
- Understand how to identify measurable outcomes and outcome chains
- Develop a logic model
- Identify performance indicators and performance targets to support the outcome measurement process
CHAPTER 1: Getting Started
Your outcome measurement efforts will benefit greatly from some planning at the front end. Before you jump head first into the outcome measurement process, convene stakeholders and organizational leadership to discuss the important questions. Where will you focus your efforts? What are you hoping to accomplish through the process? Who will be involved in the process? How will you engage members of your organization in the process? What is your timeline? What resources are at your disposal? Once you have answers to these questions, you can begin to look more closely at the specific outcomes and performance indicators that will drive the outcome measurement process.
Narrow your scope and clarify your goals.
In starting out, it's important not to try to measure too much. Start slowly and learn from your experience. Don't try to perform outcome measurement at the same time for everything you do. Pick one program or service as a starting point. Questions that will help you brainstorm where to begin include:
- Is a funder, board member, or staff person requesting that you look at a particular program?
- Do you have a new program with an unproven track record?
- Do you have an existing service with shaky performance?
- Do you have an effective program that you want to document as being so?
Once you have decided what program to evaluate, explore the purpose for the outcome measurement processes. There are a variety of reasons that could motivate outcome measurement, including:
- To support new program design and identify success in a new program area
- To support program redesign by determining whether the program's underlying theories and assumptions are correct
- To support funding requests and describe how you will measure a program or service's impacts
- To support staff or board planning processes and report on a program's impact
- To support a funder's request for evidence of the program's effectiveness
- To support internal reviews and determine whether to continue to allocate funding to a program
- To support organization-wide quality improvement and refine program delivery
Assemble a well-rounded team.
The outcome measurement team will require a well-rounded group with varying skill sets and a devoted leader. As you begin to assemble your outcome measurement team, consider who on your staff possesses the following skills:
- Project coordination, including laying out tasks in a sequence, informing other staff of their roles and assignments, providing assistance to the team as they complete their parts of the evaluation process, and ensuring that the work is being done
- Service or program knowledge, including the ability to identify the relationship between the activities being provided and the intended impacts and an understanding of the types of outcomes your program could achieve
- Computer skills, including expertise in formatting surveys and other data collection instruments, creating spreadsheets or databases, and entering data
The outcome measurement process can often feel intimidating or threatening to program staff, as outcome measurement is sometimes a response to poor performance or part of a corrective action plan. To allay your staff's concerns about outcome measurement, involve them in the process whenever possible.
Outcome measurement is an investment.
Developing an outcome measurement plan will require an investment of time and resources. For organizations first learning outcome measurement, it may take as long as three years to develop a comprehensive outcome measurement system. Timing is also important to funders. Make sure to consider when your funders' reporting cycles are so that you are producing outcome measurement results at a time that aligns with their requests for information about your programs' impacts. Keep in mind that there are times when conducting outcome measurement may not be a good idea. If your organization is in crisis or severe financial trouble, outcome measurement cannot be a priority.
In addition to investing time, outcome measurement may require an investment of other resources, including:
- Staff labor
- Communication (e.g., postage and telephone)
- Supplies and equipment
CHAPTER 2: Understanding Outcomes
Prior to beginning the process of data collection, you'll need to define the outcomes you want to accomplish. As a nonprofit organization, your intended outcomes focus on the impacts or changes that your clients experience as a result of your programs and services. When crafting outcomes, consider the who, what, and how of the initiative. Who will be impacted by the initiative? What will change as a result of the initiative, and how will it change? When identifying outcomes, it can be helpful to organize your thoughts in the form of an outcome chain that logically links your services to client learning, behavior, and, eventually, results.
Outcome statements capture the "who," "what," and "how."
Formulaic, straightforward outcome statements can help you develop your outcomes. When crafting an outcome statement, consider the details of the initiative you are providing, the recipient of that service or program, and the intended impact of that service on your target recipients. For instance, a youth-serving nonprofit might identify the following outcome statements in relation to their afterschool programming:
- Increased school engagement for participating students
- Reduced high-risk behavior for participating students
- Increased high school graduation rates for participating students
Outcome chains document correlative relationships.
Outcome chains can help organize your thinking about what you hope to achieve. They require you to put your program theory to work and articulate how your activities will bring about the impacts in the organizations with whom you are working. Outcome chains create a logical progression of the short-term, intermediate, and long-term outcomes that lead to your goals. Consider the example of a large nonprofit that provides training to smaller nonprofit organizations. The provision of training can create the following chain, linking reactions, learning, behavior, and results:
Not all organizations can and will measure all the different outcomes noted in an outcome chain. Consider the example of the outcome chain above. Your organization may not have the tools and resources to evaluate outcomes as they relate to reaction, learning, behavior, and results. If your circumstances do not allow you to evaluate all areas, focus on the earlier outcomes noted in the outcome chain. There is no point in measuring for results if you cannot point to the series of outcomes that impacted those results.
Develop realistic, informed outcomes.
When crafting outcomes, consider the following:
- Are the outcomes related to the “core business” of your organization or program?
- Is it within your control to influence these outcomes?
- Are your outcomes realistic and attainable? Are your outcomes achievable within funding and reporting periods?
- Are your outcomes written as change statements—will things increase, decrease, or stay the same?
- Have you moved beyond client satisfaction in your outcomes?
- Is there a logical sequence among your short-term, intermediate, and long-term outcomes?
- Are there any big “leaps” in your outcomes, i.e., gaps in the progression of impacts?
CHAPTER 3: Logic Models and Outcome Measurement
An organization should have a well-developed logic model in place before they finalize a comprehensive outcome measurement plan. A logic model maps out an overview of an organization's tools and resources, the services they provide, and the intended impacts of these services. A basic logic model documents inputs or resources, activities, outputs, and short-term, intermediate, and long-term outcomes. Inputs or resources are the assets that an organization is prepared to invest to support or implement a program, including things like money, staff, and equipment. Activities capture the methodologies an organization plans to use in order to implement a project, while outputs describe activities in more finite, numerical terms such as the units of service provided. Lastly, outcomes capture the changes, benefits, and overall impact that the program or initiative has had on an organization's client population. Once a well-developed logic model is in place, an organization can begin to identify performance indicators that will help to measure the organization's progress towards its intended outcomes.
Ask the right questions.
When looking to identify the elements of your organization's logic model, consider the following:
- Inputs/Resources: What inputs or ingredients do you need to operate your program? How many staff? How large a budget? How many clients do you have/need?
- Activities: What will you do? What methods will you use to deliver your program or activities? What content areas will you cover? What will you provide?
- Outputs: What will be the tangible products or units of your program or activity? How many clients will you serve?
- Outcomes: What impact will your program or activities have on your clients? What is reasonable to expect in the way of change?
Logic models capture underlying assumptions and program theory.
This connection between the activities you provide and the outcomes you hope to accomplish is known as the program theory; it articulates the assumptions about the ability of certain activities to drive particular changes. Many nonprofit providers use logic models as a tool to reflect the program theory underlying their programs. One contribution that outcome measurement can make in your own organization is to demonstrate whether or not the program theory underlying your organization's activities and initiatives is valid.
Logic models document relationships.
While not all logic models look the same, they all serve the same purpose: to graphically capture the assumptions and cause and effect relationships that drive your organization's work on a project.
Download a sample logic model template and test your understanding of the different elements of a logic model using the activity on the right.
CHAPTER 4: Developing Performance Indicators
The next step in putting together an effective outcome measurement plan focuses on deciding how you are going to make your intended outcomes measurable by defining a set of performance measures or indicators.
Because outcomes can be broad in nature and somewhat vague, performance indicators serve as a bridge, connecting intended outcomes and the actual data collection process. Identifying performance indicators and creating targets for performance help your organization to determine whether you have, in fact, had a measurable impact on your clients and reached your programmatic goals.
Indicators are specific and observable.
In order to serve effectively as a bridge to data collection, indicators must be specific items of information that describe observable and measurable characteristics or changes in corresponding outcomes. Indicators must be measures that can be seen, heard, counted, reported, or enumerated using some type of data collection method.
Performance indicators answer questions like, "How you will know when changes have occurred?" and "How you will know when you have achieved the outcomes?" Thinking ahead to possible data collection methods will tell you if your indicators are specific enough. Ask questions like these to determine whether your indicators will work:
- How can I see the change? (Through what kind of observation?)
- How can I hear the change? (Through interviews? Focus groups?)
- How can I read the change? (Through surveys? In records?)
Be thorough, yet realistic when identifying indicators.
Each outcome you identify will generally have one to three supporting indicators depending on the complexity of the outcome. When selecting indicators, keep things simple and be sure to collect indicators that are useful, meaningful, and informative. Keep in mind that you will be responsible for measuring these indicators through your data collection process. Consider the following checklist:
- Do your indicators make sense in relation to the outcomes they are intended to measure?
- Are your indicators directly related to the outcome? Do they define the outcome?
- Are your indicators specific?
- Are your indicators measurable or observable? Can they be seen (i.e., observed behavior), heard (i.e., participant interview), or read (i.e., client records)?
- Is it reasonable that you can collect data on the indicators?
- Is it within your resources to collect data?
Targets, comparative standards, and baselines give meaning to indicators.
When developing indicators, it is often helpful to identify targets, baselines, and comparative standards.
- Targets are the specific levels of achievement that you hope to achieve for an indicator or outcome. Targets are the quantifiable goals you hope to achieve.
- Comparative standards are the data points that are used as a comparison or standard of achievement for a particular indicator or outcome. Comparative standards help to provide insight into how your targets relate to or improve on past performance.
- Baselines are the data points that are gathered to provide a comparison for assessing program changes or impact. A baseline will allow you to set reasonable targets and measure the progress against these targets.
Outcome measurement is a systematic way to assess the extent to which a program has achieved its intended results. Preparation and planning is a key part of the outcome measurement process. The better the planning, the more impact the outcome measurement will have on your organization and your organization's bottom line. You should now understand how identifying outcomes and outcome chains, developing a logic model, and identifying performance indicators and performance targets are integral parts of planning for the process of outcome measurement. Thank you for taking the time to learn about developing a plan for outcome measurement.
Outcome measurement proves correlation, not causation.
Keep in mind that outcome measurement explores what your program provides, what its intended impacts are, and whether or not it achieves them. It will not prove that the changes that take place are a result of your program. In order to prove direct causation, an organization will need to take part in experimental research and a controlled study to link your specific programs and activities to results. Naturally, this requires considerable time, effort, and resources.
Learn more about outcome measurement at the links below.
- Center for Program Evaluation and Performance Measurement
This website from the Office of Justice Programs' Bureau of Justice Assistance provides basic information about initiating and planning evaluation processes and planning and links to other government resources on the topic, as well.
- Outcome Measurement Resource Network
This website, maintained by the United Way of America, includes key definitions and concepts around outcome measurement, resource lists, sample outcome measurement initiatives, and a full-text resource library.
- Demystifying Outcome Measurement in Community Development
Developed for Neighbor Works America by the Joint Center for Housing Studies of Harvard University, this 2007 report by Renu Madan provides valuable information on how and why community development organizations measure outcomes.