Creating and Implementing a Data Collection Plan

Here are descriptions for five approaches or designs you are likely to use for your data collection.  You may want to employ more than one type of design.

Design 1:  Post-only Measures

• Data are collected once: at the end of the program, service, or activity
• Example: Level of participant knowledge on a survey after a training workshop

Design 2: Pre/Post Measures    

• Data are collected twice: at the beginning to establish a baseline and at the end of the program
• Example: Comparison of an organization’s documented fundraising success before and after receiving technical assistance

Design 3: Time Series    

• Data are collected a number of times: during an ongoing program and in follow-up
• Example: Monthly observations of an organization’s collaboration meetings to track changes in partnership development and communication

Design 4: Measures with a Comparison Group

• Data are collected from two groups: one group that receives the intervention and one that doesn’t       
• Example: Comparison of data on skill development from individuals who participated in training and those who have not yet taken your workshop
• Note: Comparison groups can be very useful in demonstrating the success of your intervention.  The main question is, can you find a group of people or organizations that is just like the group with whom you are working?  In order to provide a valid comparison, the two groups must have the same general characteristics.  A similar group may be difficult to find.  However, if you are working with different groups at different times, and the groups are similar, this approach may work for you. 

Design 5: Measures with a Comparative Standard

• Data are collected once: at the end of the program, service, or activity, and are compared with a standard
• Example: Comparison of this year’s data on organizations’ success in fundraising versus last year’s data
• Note: Comparative standards are standards against which you can measure yourself. There are standards of success in some fields (e.g., health mortality and morbidity rates, student achievement scores, teen birth rates). For intermediaries, however, there are unlikely to be many regarding your program outcomes or indicators. You can, however, compare your results for one time period to an earlier one, as shown in the example above. You collect data for the first time period as your baseline and use it as your standard in the future.

It will be vital to find reliable and trustworthy people to collect and manage the data. Consider who will collect the data and how you will recruit these people. What steps will they need to take to collect the data? How will you train them? Finally, who will be responsible for monitoring the data collection process to ensure you are getting what you need? It’s important to answer each of these questions during your planning. You don’t want to be surprised halfway through the process to discover your three-month follow-up surveys were not mailed out because you didn’t identify who would do so!

Communicate with the organizations you serve or the program’s staff to inform them of this step in the evaluation process. Make sure they know that you will be collecting data, either at the time of service or in follow-up. Clarify why it is important to you and how you intend to use the data. Organizations often have outcome reporting requirements themselves, so they usually are responsive if they have been alerted to your needs ahead of time. Advising them in advance about your data collection plans will help increase their willingness to participate during implementation.

Anonymous and confidential do not mean the same thing. “Anonymous” means you do not know who provided the responses. “Confidential” means you know or can find out who provided the responses, but you are committed to keeping the information to yourself.

You must ensure that you protect the confidentiality of any individual’s data or comment. It is easy to make your surveys anonymous, but if you want to track people over time, you’ll likely need to attach ID numbers to each person, keeping a list of the names and numbers in a locked file.

It is important to inform people that you are measuring your program’s outcomes and may use data they provide in some way. You must let them know that their participation is voluntary and explain how you will maintain the confidentiality of their data.