Analyzing Data and Communicating Results

The goal of this capacity builder’s training was to equip faith-based and community organizations with the knowledge, skills, and tools to successfully manage Federal grants. The capacity builder measured the outcome in two ways: first, in a post-training survey, participants were asked to agree or disagree with the statement, “I gained new knowledge about managing Federal grants.” These outcomes were self-reported, and if the capacity builder wanted to get more objective assessments about whether the organization did, in fact, learn something about managing Federal grants, they might have administered a brief survey or quiz prior to and after the workshop.

Let us suppose that 54 percent of recipients reported that they gained new knowledge on the topic. The analysis would ask, “Is this good? What does this tell us about the training, and possibly about the participants?” On one hand, this tells us that nearly half of the participants did not learn anything new from the training. You might conclude that your training participants have all the knowledge they need of grants management, and schedule other training topics. But it is possible that your staff observes something very different—perhaps that most of the participating organizations are not tracking employee time in accordance with Federal rules and regulations, or retaining records. Now you have a different situation where your training contained very important information that the participants simply did not receive. Your next step, rather than to move onto other topics, might be to find other ways to deliver the necessary information.

One capacity builder used index scores for a capacity assessment and tracked the progress of each organization as well as the cohort across several categories of assessment. Here is a sample of the data, which analyzes the average and median index scores of all organizations in a cohort for each category. The assessment asks whether the organization has key practices in place for each of the categories. The results are tabulated into scores as the percentage of “yes” answers. The index scores are tracked across time by category and in aggregate for the organization and the cohort.

As expected by the program, results from the analysis indicated that index scores were increasing across the cohort. However, in the area of financial management, the scores actually dropped. According to this capacity builder, this was because after the initial assessment, the participating organizations increased their knowledge about the topic and then answered the questions on the assessment differently, realizing that they did not have as many effective financial management practices in place as they initially thought. What could have been interpreted as a “negative” outcome is instead explained as a possible “positive” outcome.
The capacity builder might also be interested in what capacity dimensions saw the most frequent improvements overall. This might help them define future capacity building. The program manager might start by identifying the single biggest area of improvement for each organization:

Let’s suppose that the individual capacity index scores for the previous example are compared with the hours of technical assistance each organization received, and were plotted. Why would this be of interest? In many programs, a central theory of capacity building is that more technical assistance yields better results for capacity building. If an organization is measuring capacity through the index scores and measuring the amount of technical assistance in hours, it makes sense to explore the relationship between those sets of data to see what information is available to managers. Contracted evaluators may perform rigorous statistical analysis, including correlation and regression analysis as discussed in the previous section, however, you will see in the chart below that even non-statistically savvy managers can examine the relationships between two variables.

Use the activity on the right to explore what this data set looks like when plotted on a two-variable chart.