Creating and Implementing a Data Collection Plan

The primary advantage of surveys is their cost in relation to the amount of data you can collect. Surveying generally is considered efficient because you can include large numbers of people at a relatively low cost. There are two key disadvantages: First, if the survey is conducted by mail, response rates can be very low, jeopardizing the validity of the data collected. There are mechanisms to increase response rates, but they will add to the cost of the survey. We will discuss tips for boosting response rates later in this lesson. Written surveys also don’t allow respondents to clarify a confusing question. Thorough survey pre-testing can reduce the likelihood that problems will arise.

Reliability can also be thought of as the extent to which data are reproducible. Do items or questions on a survey, for example, repeatedly produce the same response regardless of when the survey is administered or whether the respondents are men or women? Bias in the data collection instrument is a primary threat to reliability and can be reduced by repeated testing and revision of the instrument.

You cannot have a valid instrument if it is not reliable. However, you can have a reliable instrument that is not valid. Think of shooting arrows at a target. Reliability is getting the arrows to land in about the same place each time you shoot. You can do this without hitting the bull’s-eye. Validity is getting the arrow to land on the bull’s-eye. Lots of arrows landing in the bull’s-eye means you have both reliability and validity.