Developing New Data Collection Tools

4.5 Guidelines for writing new questions:

Question types

Open-ended questions

• allows the informant to answer freely
• useful when you cannot predict all possible responses and want to know the range of available options
• more time-consuming for the respondent, who may then choose to skip them
• handwritten responses more time-consuming to record and analyse
• may be difficult to decipher or ambiguous

Closed-ended questions

• gives a set number of responses to choose from
• useful when the question writer knows what range of responses is available
• adding an “other” category with a “please specify”, takes care of unforeseen options but is more time-consuming to code.

Response options

Checklists

• include instructions to “check ONLY one”, or “check all that apply”.
• if the latter, any box left blank by accident would be coded as a “NO” response
• avoid that problem by providing options for every item, e.g, yes, no; never, sometimes, often, always

Scales

Since devising good scales is more difficult than it seems, try to find scales that have been used somewhere else previously.

• providing a scale for a response e.g., from 1-5 or from poor to excellent, is common
• options need to be mutually exclusive and not confusing
• options cover the full range of likely replies and are evenly spaced
• generally include an odd number of options, (often 5 or 7 points) to allow for a neutral mid-point
• omitting a neutral midpoint forces respondents to state an opinion by not giving them the “I don’t know” option
• a Likert Scale asks for an opinion about a statement, e.g, “Please read the statements below and then circle the one number on the right that best describes how much you agree or disagree with each statement.”
• keep response scales in the same order from question to question (1-5) not (5-1), e.g., “I feel comfortable with my volunteer tutor.” (1-5 from total disagreement to total agreement) and “I am afraid of being bullied by older children in the program.” (1-5 from total disagreement to total agreement) (When coding the latter question, keep the scoring in the same positive direction as the first question by reversing the numbers. See Step 6)
• positive statements are usually preferable unless a negative will give more relevant data

Provide sufficient information to informants

• provide definitions of standard terms (e.g., wages, household income) on the questionnaire or
• have staff give definitions while distributing questionnaire
• avoid defining words related to opinions or feelings
• informants may feel inhibited or restricted to a particular range of responses
  
• provide examples instead of definitions to suggest types of feelings or opinions

Assumptions

• review questions to look for hidden assumptions
• discuss the meaning with others to suggest other interpretations
• informants can misunderstand and misconstrue questions that writers think are obvious
• e.g., Does the question specify a clear time frame or frequency, e.g., “in the past 12 months”, “this school year?”

Neutral questions

Avoid questions that anticipate one option more than another or do not provide a full range of options. For example, asking a child, “How much did you like being in the program, a lot or a little?” provides no possible response for someone who didn’t like it at all. Giving more options, with a broader range, provides more precise, therefore more meaningful data.

One concept per question

• avoid multiple ideas in a question (often containing “and”), where a respondent can answer “yes” to one part of the question and “no” to another part
• divide such questions into two separate questions that can be coded and compared separately

Follow-up questions

• consider following up questions about attitudes or opinions by asking about the intensity with which they are held.
• link opinion to behaviour to help evaluate the strength of people’s responses

Verifying responses

It is helpful to verify the consistency or accuracy of responses by asking more than one question about a certain attitude or behaviour in a different way, where space allows.

Encourage precision

For example, it would likely be more helpful for data analysis to ask an open-ended question, “How many times did you attend the program this session?” instead of giving a range of options such as “<4, 5-9, > 10.”

• avoid grouping data or collapsing categories for essential questions since you may need more precise data later on.
• there is a trade-off between missing opportunities for analysis and asking for so much precision that informants can’t or won’t want to provide it
• pretesting can help determine the best balance for each questionnaire.

Personal questions

One major exception to precision concerns certain personal data. Respondents may skip questions they find too personal and possibly not complete the questionnaire. There may be less intrusive ways of finding out the information, e.g., on the subject of income, providing a checklist of options with a fairly wide range (>$5,000.) will improve response rate.

• Other information can stand in for income, such as educational level and home ownership, which are good indicators of income and less likely to bother respondents.

The ‘not applicable’ category

A full range of responses may include “not applicable.” Questions that cannot be answered because they do not apply to respondents can be irritating and possibly influence their attitude towards completing the questionnaire.

Skipped responses (non-responses) may have been skipped on purpose or by accident, creating ambiguity for analysis Think about all possible interpretations when writing questions and as needed add, “Please explain.” or ask a filter question.

Filter questions

• a preliminary question to filter out respondents who lack some knowledge or experience necessary to answer a question
• e.g., “Do you like playing checkers?” implies that the respondent knows what checkers is and has played it. Ask respondents first, “Have you ever played checkers?” and have those who answer “NO” skip to another question.

Skip patterns

• instruction to skip a response after a question filters out some respondents
• can be confusing and distracting
• consider using arrows if there are several skips
• best placement is directly after the response, to prevent the informant from overlooking the instruction
• phrase the skip as a positive statement, e.g.,

Make forms easy to complete

• try for consistency in typeface, layout of similar questions
• more white space on forms, especially between questions, increases readability and is more inviting for respondents
• reduce the amount of writing respondents need to do, e.g., provide check boxes, circles or ( )
• provide sufficient space for written answers
• create logical order and flow to questions
• provide instructions for each question or each different type of question
• make instructions more noticeable with a different font or boldface
• keep instructions and questions together, not on separate pages
• start each sentence on a new line for easy readability

Provide information about confidentiality (also See Step 5)

Explain the purpose for collecting data on the form or in a verbal introduction (preferably both). Tell informants about confidentiality: that their responses will not be reported individually or with their names. It may be necessary to reassure participants that the program welcomes both positive and critical comments in order to identify gaps in service and strengthen programming. If personal questions are being asked, the need for confidentiality, and confidence that it will be maintained, are especially important. Some projects may require more safeguards around confidentiality than others. Since programs that want to follow individuals over time within a program or over various sessions, need to be able to identify the participants who provided data, it is important to have some identifier, either name or another unique identifier like gender coupled with date of birth. Check that the identifier is unique. Data may be kept anonymous to all but one person who tags each questionnaire with an identification code instead of a name, records it on a master list, then links it to other data records by the same code. The master list should be kept secure and accessible to only one person on the evaluation team.

Even if names are collected, information provided in reports should not be traceable to an individual either by name or by specific identifying information. A personal statement such as a testimonial or quotation could be published with permission from the individual (and/or parents). However, families and programs may not anticipate all the possible negative consequences of releasing personal information that is distributed publicly.