 Developing
New Data Collection Tools
7.5 Closing the evaluation circle
Evaluation as process
The results of an evaluation process lead back to
its original purpose. If those results satisfy the
original questions, program managers and staff have
a direction charted for future program operations.
Evaluation can be extremely satisfying. It can show
that basic assumptions are sound and can confirm a
theory of change. Sufficient documentation of procedures
allows consistent duplication of the program in another
setting or with another population. However, a promising
evaluation is not the final word on a community program’s
effectiveness.
Community programs contend with change in every session:
different participants, often different personnel,
which means that the same level of effectiveness
is not guaranteed another time.
Programmers need
to repeat the same questions and pose new ones.
They need to be ready to adjust programs—look
for new solutions to unforeseen challenges—capitalize
on unexpected bonuses—learn from lessons— and
expand elements that seem most promising.
Rethinking evaluation
Results may not satisfy the questions you originally
posed. Results occasionally find that programs are
counter-productive or ineffective in meeting their
original objectives.
Factors can relate to the program itself, the theory
behind it or the quality of the evaluation.
Results, for instance, can be influenced by an indicator
that is unsuited to capturing the anticipated change
because the measurement timeframe is inappropriate,
i.e. either program effects build slowly over a longer
period after the program ends or were short-lived
and are no longer evident.
- Changes might be too subtle to be readily observed
or respondents might not have adequate knowledge
of either the child or circumstances where some change
is noticeable.
- Learning happens in small increments and may
not be evident in the early stages or when there
are small
numbers of participants and informants.
Results may raise other questions about longer-term
or delayed effects that demand some follow-up or refinement
of the evaluation methods— more data from another
session, more children, different sources, closer observation,
different questions. The evaluation process is iterative,
leading back on itself, engendering more questions
and the need for deeper understanding.
Learning from others’ evaluations
Programmers can learn from others’ evaluation
results as well as from their own. More community groups
are engaging in some form of systematic evaluation and
posting results on websites and in databases of research
associations. Programmers can compare their methods and
evaluation results. Following these or similar guidelines—asking
appropriate questions, carefully collecting and analyzing
data, and filtering it through experience— will
enable community programs to build up a bank of knowledge,
with realistic solutions to help both programs and children
succeed.
|
