DESIGNING AND CONDUCTING FORMATIVE EVALUATIONS

The following are notes from chapter 10 of The Systematic Design Instruction, sixth edition by Dick, Carey & Carey.

Formative evaluation is the process designers use to obtain data that can be used to revise their instruction or the materials they use to make them more efficient and effective. It is different from summative evaluation, which refers to the collection of data used to determine the effectiveness of the final version of the instruction.

There are three basic phases of formative evaluation. The first phase is a one-to-one evaluation, the second one is a small group evaluation, and the third stage is usually a field trial where the performance evaluation occurs in the performance context.

Here are some samples of questions to ask when performing formative evaluations: Are the materials appropriate for the type of learning outcome? Do the materials include adequate instructions on the subordinate skills, and are these skills sequenced and clustered logically? Are the materials clear and readily understood by representative members of the target group? What is the motivational value of the material? Can the materials be managed efficiently in the manner they are mediated?

The types of date you may want to collect include test data (entry behavior, pre, post and performance tests), comments by the learners, the learners' overall reactions to the instruction and their perceptions of the difficulties, the time required for the learners to complete various components of the instruction, and the reaction of the manager or supervisor who has observed the learners using the skills in the performance context.

The purpose of the one-to-one stage is to identify and remove the most obvious errors in the instruction, and to obtain the learners' initial performance indications and reactions to the content. This is accomplished through direct interaction between the designer and the individual learners who are representative of the target population. The learners not only go through the instructional materials, but also take the tests provided with the materials. Don't be surprised if some test items that may appear clear to you will be totally misinterpreted by the learner. The designer will be evaluating the clarity, the impact and the feasibility of the instruction. Only a very rough estimate of the learning time can be obtained in this phase of the evaluation. Attention should also be paid to not overgeneralize the data gathered from only one individual.

Small-group evaluation has the purpose of determining the effectiveness of changes made following the one-to-one evaluation and to identify any remaining learning problems the learners may have. Another purpose of this phase is to determine whether learners can use the instruction without interacting with the instructor. During this phase the instructor can determine the time required for the learners to complete both the instruction and the required performance measures. The small group should consist of eight to twenty learners. If the group is not homogeneous, it should consist of low, average and high-achieving students, learners with various native languages, learners who are familiar with a particular procedure, and young inexperienced learners as well as mature ones. The instructor should intervene as little as possible in the process.

Questions to ask during the small group phase could be: Was the instruction interesting? Did you understand what you were supposed to learn? Were the materials directly related to the objectives? Were sufficient practice exercises included? Were the practice exercises relevant? Did the tests really measure your knowledge of the objectives?

During the third stage of evaluation, the designer should determine whether the skills that have been taught are retained and used in the performance context, and whether these skills have the desired effect on the organization. The designer should get suggestions from the learners and those they work with of how to improve the instruction. In framing questions for this phase, the designer should be specific about which skills are of interest in the evaluation.

One concern in any evaluation of the materials is to ensure that any technical equipment is operating effectively. Also make sure you work with learners in a quiet setting. And be prepared to obtain information that indicate that your materials are not as effective as you thought they would be, even after going through an extensive instructional design process.