Creating and Implementing a Data Collection Plan

The primary advantage of surveys is their cost in relation to the amount of data you can collect. Surveying generally is considered efficient because you can include large numbers of people at a relatively low cost. There are two key disadvantages: First, if the survey is conducted by mail, response rates can be very low, jeopardizing the validity of the data collected. There are mechanisms to increase response rates, but they will add to the cost of the survey. We will discuss tips for boosting response rates later in this lesson. Written surveys also don’t allow respondents to clarify a confusing question. Thorough survey pre-testing can reduce the likelihood that problems will arise.

Here are some examples of ways to use surveys:

• Track grassroots organizations’ use of and satisfaction with technical assistance services you provide.
• Survey all organizations receiving technical assistance to learn about changes in their fundraising tactics and the results of their efforts to raise more money.

Click here to download the “Technical Assistance Survey Template.” You can adapt this template for use in your program evaluation.

Interviews use standardized instruments but are conducted either in person or over the telephone. In fact, an interview may use the same instrument created for a written survey, although interviewing generally offers the chance to explore questions more deeply. You can ask more complex questions in an interview since you have the opportunity to clarify any confusion. You also can ask the respondents to elaborate on their answers, eliciting more in-depth information than a survey provides. The primary disadvantage of interviews is their cost. It takes considerably more time (and therefore costs more money) to conduct telephone and in-person interviews. Often, this means you collect information from fewer people. Interview reliability also can be problematic if interviewers are not well-trained. They may ask questions in different ways or otherwise bias the responses.

Here are some examples of ways to use interviews:

• Talk to different grassroots organizations to learn about the way in which they are applying new knowledge of partnership development.
• Interview individuals within an organization to explore their perceptions of changes in capacity and ability to deliver services.

While interviews with individuals are meant to solicit data without any influence or bias from the interviewer or other individuals, focus groups are designed to allow participants to discuss the questions and share their opinions. This means people can influence one another in the process, stimulating memory or debate on an issue. The advantage of focus groups lies in the richness of the information generated. The disadvantage is that you can rarely generalize or apply the findings to your entire population of participants or clients. Focus groups often are used prior to creating a survey to test concepts and wording of questions. Following a written survey, they are used to explore specific questions or issues more thoroughly.

Here are some examples of ways to use focus groups:

• Hold a structured meeting with staff in a community-based organization to learn more about their grants management practices, what worked during the year, and what did not.
• Conduct a discussion with staff from several organizations to explore their use of computer technology for tracking financial data.

Observations require well-trained observers who follow detailed guidelines about whom or what to observe, when and for how long, and by what method of recording. The primary advantage of observation is its validity. When done well, observation is considered a strong data collection method because it generates firsthand, unbiased information by individuals who have been trained on what to look for and how to record it. Observation does require time (for development of the observation tool, training of the observers, and data collection), making it one of the costlier methods.

Here are some examples of ways to use observations:

• Observe individuals participating in training to track the development of their skill in the topic.
• Observe community meetings sponsored by grassroots organizations to learn about their partnership-building techniques and collaborative behavior.

Internal records available to a capacity builder might include financial documents, monthly reports, activity logs, purchase orders, etc. The advantage of using records from your organization is the ease of data collection. The data already exists and no additional effort needs to be made to collect it (assuming the specific data you need is actually available and up-to-date).

If the data is available and timely, record review is a very economical and efficient data collection method. If not, it is likely well worth the time to make improvements to your data management system so you can rely on internal record review for future outcome measurement work. Just a few changes to an existing form can turn it into a useful data collection tool. A small amount of staff training can increase the validity and reliability of internally generated data.

Here are some examples of documents or records from which you can gather data:

• Sign-in logs from a series of workshops to track attendance in training, measuring consistency of attendance as an indicator of organizational commitment to learning.
• Feedback forms completed by workshop participants to learn about satisfaction with training provided.

Official records can include Federal, state, or local government sources such as the U.S. Census Bureau, health departments, law enforcement, school records, assessor data, etc. If the data is relevant and accessible, then official record review is very low-cost.