 Designing
the Data Collection Process
3.1 Decide on
evaluation methods
There are several alternative methods for evaluating
the effectiveness of a particular program or program
element. Determine which are most suitable for the
program situation.
Experimental studies
the ideal for social science research
With rigorous standards that include:
- randomized
selection of people to receive a
program,
- matched
group characteristics of
participants and non-participants,
- a
degree of blindness
so that participants do not know what elements
are being tested and
researchers do not know whether data come from participants
or
non-participants.
Encouraged for research purposes
- not
yet common for evaluating community
prevention and intervention programs
- usually
involves the assistance of
academic institutions or research departments of
large agencies
Need an example?
- In cases such as agencies with sufficiently
high case loads or schools, it has been possible
to randomly select people to participate, or not, in a program.
Ensuring that evaluators are blind to whether
data apply to participants
or non-participants is still a concern.
Sampled studies
- when
large numbers are involved, as in
national, multi-site programs, it may be necessary to
sample participants, i.e., select
a part of the population to represent the whole population.
(See Glossary
for Statistical samples)
Comparing participants
to non-participants is problematic because of possible selection
bias: whatever factors made participants decide to join a program
in the first place may make them different from the comparison
group of non-participants.
Quasi-experimental studies
-
more common for community groups—compares a group
of participants to another group of people with
similar characteristics outside the program.
Group comparisons
- appropriate
within a multi-site program if program
curriculum, delivery and characteristics of participant
groups are the same.
Programs may compare different elements of a program,
e.g., when a program introduces a bullying awareness
presentation for one session but not to a comparable
group in the same program.
Quantitative and qualitative research
Quantitative research
- uses
measurable, quantifiable data and
mathematical formulas or statistics
- Community
groups most commonly use
basic statistics, e.g., percentages and averages
that can be
turned into
graphs and charts to easily show patterns
of characteristics and behaviour.
- basic
tools are data collection forms such
as questionnaires, standardized tests and scales
- requires
careful data collection, analysis
and statistical methods
- findings
can be used to predict similar outcomes
in similar circumstances
Qualitative research
- uses
descriptive explanations and narrative
to describe phenomena that cannot be measured
easily
- provides
context of a particular place and
time
- may
not be comparable or generalized
to other or larger groups of
people
- basic
tools: interviews,
open-ended questions, focus groups and observation
notes
- gives
respondents more freedom to bring
up points or issues in their own words
which are important
to them but may not have been considered by the program
developer or evaluators
- especially
helpful for development or pilot-testing
of programs, processes or questionnaires
Content analysis
Quantitative and qualitative research can both be used
to good effect in community program evaluation
- technique
for systematically pulling meaningful
data from qualitative data sources,
-
e.g., volunteer observations, interview notes, focus
group discussions, personal narratives and participants’ comments.
- readily
available computer software packages
systematically search out meaning beyond the easily recognizable
anecdote
- manual
analysis possible with smaller
amounts of data
Methods:
- look for topics and patterns
to emerge from the material itself
OR
- develop a list of topics or
themes first, then code and sort answers
accordingly, adjusting topics to
suit the information
and level of detail wanted.
Ideally, two or more people discuss the probable topics then analyze data independently
and compare results to overcome individual bias about importance and priorities
Time-dependent data collection
Some program effects may be short
lived, some more long lasting, others may be delayed
or perhaps even increase over time.
- techniques
for measuring the extent of change in
a participant
- assesses
where a participant rates in
terms of objectives at the end of a program
- commonly
a pre- and post-test method, where
programmers obtain the same type of data about participants
at
the start of a program and again after it
ends
- may
include additional data collection,
periodically during long programs and/or additional
follow-up,
- may
also assess further characteristics
of participants which could also
have shown change
- data
collected with a standard set of
questions in an interview, a custom questionnaire,
standardized scale
or
a combination of all
three
Formative and summative methods
- complement
to pre-post-test methods
- formative
methods compile data over
time to show development
- summative
evaluation is a one-time
test of the final outcome
Follow up evaluation
- evaluations
that follow participants or gather
data about them over a longer time frame after the end
of the program, or over several
sessions
- likely
to increase evaluation knowledge
- can
be difficult and costly to
trace and re-establish contact with participants
- try
to establish a reliable method for
contacting people at the outset
Think carefully about the type
and timing of effects which you can expect from your program,
and this will guide the type of evaluation methodology which
will work best for you.
Some evidence suggests that effects of programs are
generally short lived unless they are somehow reinforced
by booster programs or other interventions.
|
