Data Collection

5.5 Conduct data collection

(Tips on focus groups, observations, interviews, questionnaires and review of written sources)

Focus groups

• a gathering of people who share a common interest or experience and engage in a discussion about it guided by a trained or experienced facilitator
• focus groups are less structured than interviews
• telephone conference calls can be used if a meeting is not possible
• participants should be assured of confidentiality at the outset

Facilitator’s role

Facilitator has a general script with a few specific open-ended questions

• follows up interesting side topics that may provide useful information
• training needed because individuals can affect group dynamics, swaying people by the force of their personality to agree, disagree or keep their opinions quiet
• facilitators do not express their own opinions or engage in debate
• facilitator tries to ensure a balanced, comprehensive discussion
• introduces topics
• asks follow-up questions
• gauges the degree of agreement and dissent
• probes the strength of people’s opinions
• limits discussion where necessary
• tries to bring everyone into the discussion to air all viewpoints
• brings the focus group to a close at allotted time
• summarizes main points
• asks about agreement or clarification of those points and any other issues not raised

Timing

Since it can be difficult to find a convenient time and venue for a focus group, some community programs have successfully combined focus group sessions with special events like luncheons, potluck dinners or game days

• usually 1-2 hours in length
• refreshments commonly provided
• childcare for siblings is helpful
• allows parents to talk freely without sense of urgency to leave
  
• focus group of parents is sometimes held while children are participating in a program session elsewhere

Recording discussion

• facilitator does not take notes
• programs preferably have one or more staff people take notes in detail and then compare notes afterwards
• no names should be attached to comments in notes
• confidentiality observed in follow-up reports as well
  
• focus group discussion can also be videotaped or recorded, with the consent of participants
• videotapes of discussions kept in-house
• analyzed without attaching names or other identification to comments

Observations

All observers should have a shared understanding of programming and context before conducting observations.

Levels of observation

Community programs use various types of observation for evaluation.

• external observers use qualitative techniques
• view program activities on a single occasion or periodically over the course of the session
• record subjective highlights
• choose a case study
• develop a narrative of activities, or
• monitor agreed-upon indicators through observation
  
• internal observers: program leaders, staff or program volunteers observe activity sessions on a more regular basis. They focus their observations on particular aspects, e.g.,
• consistency of program operations, accordance with planning
• participant characteristics, participation, interaction and success of curriculum activities,
• varying skill levels, development and observable effects over time

Advantages

• observations can prompt and inform questions to be posed in subsequent interviews or questionnaires, and
• identify aspects and patterns that are unrecognized or unappreciated by staff, or need further exploration

Observer effects

Periodic, low-key and/or unobtrusive observations provide a more representative view of a program than single observations and allow observers to see changes that may be overlooked on a daily basis.

The act of observing affects what is being observed. Effects vary depending on who or what is being observed and how the observation is carried out. Effects may be short-lived, particularly when those being observed are children who are busy and interested in something other than the observer. Repeated observation becomes routine and so unremarkable.

Familiarity and objectivity

Internal observers are less likely to disrupt programming. They can differentiate between unusual situations and normal behaviour more easily than external observers and may bring an additional layer of knowledge to observations of individual behaviour and relationships. However, their greater familiarity with personnel and participants may make objectivity difficult.

Observers may want to talk to volunteers or staff after the observation period to discuss their ideas, ask follow up questions and learn about the representativeness of that particular session. When volunteers or program leaders are observers, timely feedback and objective discussion with peers in the program about their observations is important.

Clear guidelines for observations will help direct feedback away from personal discussion.

Consistency

To ensure consistency of observation techniques and perceptions among different observers, programs can hold trial-runs with role-play scenarios, then have them compare and discuss their own observations. To improve the reliability of observations, different observers should try to reach consensus, minimizing differences in what they report and their interpretation of events.

Recording observations

Expand notes and fill in details as soon as possible after the observation.

Clearly identify the time, date, location, session, name of observer and names of sources.

Observers may not want to make detailed notes during the observation, preferring to watch unobtrusively and be fully engaged with what is going on during the session. An abbreviated checklist of what should be observed can be helpful to:

• focus on the evaluation’s areas of concern
• allow quick note-taking either during or immediately after the session
• provide a consistent format for a volunteer log

Interviews

Interviews can be conducted in person or by telephone. An interview may have the feel of a conversation but should follow a prepared script or list of questions.

Advantages

Software for monitoring telephone interviewing and coding data (CATI Computer-assisted Telephone Interviewing) is available for large projects but probably neither necessary nor cost-effective for most community programs.

• interviews provide more flexibility than questionnaires for wide-ranging discussion
• can explore opinions, changes in attitudes, feelings about experiences and explanations about behaviour
• interviewers can ask follow-up questions or prompt informants to provide more details or clarification
• interviewers have the advantage of spontaneity in following important unanticipated comments; however,
• should resist adding new questions or varying wording

Inconsistent data from one interview to another cannot be easily compared or analyzed

Factors to consider

• language
• literacy level
• culture
• gender of interviewer vis-à-vis informant
• setting for the interview
• timing
• need for childcare and
• expected length of the interview
• attitudes of the interviewer

The above factors, which relate to either interviewer or informant, can affect the success of an interview. It calls for careful planning to get the most information from the interview process.

Role of the interviewer

Introductions

• interviewer politely introduces herself/himself,
• explains the general purpose of the evaluation
• how the interview fits into the process,
• the time it will take, and any other circumstances such as inducements to participate ( e.g., payment, reimbursement for transportation costs, gifts or extras like entries for a prize draw)
• reiterates the confidential nature of the interview, (that names will not be used with any comments or quotations without prior permission)

Recording interviews

• ask for permission before tape recording a conversation
• review promptly to create a full transcript of replies
• issues of security and confidentiality are as applicable to recordings of interviews as to other data formats

Taking notes

If an informant prefers not to be recorded, trained interviewers:

• write down responses verbatim, word for word
• write neatly
• carefully link responses to questions so the responses can be read and correctly analyzed
• review notes during pauses and immediately after the interview
• look for incomplete or illegible notes, while the informant is still present and able to clear up problems

Ensure there is sufficient space on interview forms for complete responses or additional work pages.

Guiding the interview

When informants give a very brief answer to open-ended questions, interviewers can probe by saying, e.g., “Is there anything else?” or “Could you explain that a bit more?” being careful to use the same wording for each informant.

Interviewers need to guide responses to open-ended questions (questions without a set of optional responses)

• to give each interviewee an equal opportunity to say what is important to them without interruption
• clarify ambiguity
• resolve contradictions
• probe for thoughtful not ill-considered answers
• ensure consistency in probing questions

Voice and body language

Interview questions are ideally simple, unambiguous, and unbiased (not morelikely to lead to one particular response than to another.)

However, an interviewer can affect the neutrality of questions by

• tone of voice,
• inflections or
• mannerisms

Questionnaires

Questionnaires share with other data collection methods the need for an appropriate introduction, on the form and preferably repeated orally to explain the place of the questionnaire in evaluation, the confidentiality policy and the value to the program of full, honest responses. When carefully written and pretested, questionnaires are simple and easy methods of obtaining high-quality data. Questionnaires can be completed by the informant (self-complete) or completed with assistance at the program, completed off-site and returned, mailed, or completed online.

For pre- and post- test data collection

• arrange consistent administration of questionnaires to all participants
• absent participants should have pre-test as close to the original time as possible
• for post-tests, anticipate timing and venue to encourage completion of questionnaires
• make arrangements to obtain questionnaires not completed or returned

Completion rates

Completion rates vary depending on the method of administering questionnaires and, depending on circumstances, also the length and complexity of the questionnaire.

• have clear, consistent procedures for tracking questionnaire completion status, e.g., a tracking chart for individual questionnaires and various stages of follow-up

It is important to try to get as high a response rate as possible. If a large percentage of informants do not return questionnaires, the question arises whether respondents and non-respondents differ in some appreciable ways, which would bias the survey results.

Factors

Particularly for pre- and post-tests or questionnaires, it is important to obtain a high completion rate for matched pre- and post-tests to be able to measure changes and possible program effects over time.

• individuals who know the project and people involved are more likely to return questionnaires
• self-complete questionnaires handed out during ‘class time’ or similar gatherings to be handed back immediately, have a high rate of return
• questionnaires handed out as people are leaving a class or event have a lower completion rate

Mailed questionnaires

• relatively inexpensive way of surveying a large number of informants or sample a population such as community residents or youth
• lowest rate of return initially
• require additional time to obtain correct names and addresses
• may require extensive follow-up to increase numbers

Phone surveys

Unless prearranged, phone surveys using standard questions are problematic. They are hindered by the widespread use of answering machines, necessitating time-consuming call-backs and follow up.

Review of other written sources

Other written sources may provide evaluation or comparison data related to indicators of program effects. These could include community data, e.g., crime statistics, public health data, or records (at a more micro level) such as fire calls in a tenant housing area, school attendance figures and library usage. They may also include demographic data and statistics from other surveys such as the National Longitudinal Survey of Children and Youth or provincial child health surveys.