Group Members:

Keena Douglas keenad@nova.edu

Jonathan Lamey lameyj@nova.edu

Monica Shakespeare monshake@nova.edu

Nadine Salmon nsalmon@nova.edu

Appendixes

REPORT ON FORMATIVE EVALUATION

OF INSTRUCTOR MATERIALS

Introduction to the Formative Evaluation Process

Formative evaluation involves the activities that assist in determining if the materials are effective in order to decide how to improve it. In normal situations, a pre - test and subsequent posttest are administered as a part of the delivery of instruction. In the case under consideration the pre – test is not necessary, since the target group consists of teachers in a college setting, and the many years of experience both in teaching and in evaluating (using previous versions of the JBTE Assessment Form) ie quite extensive. We therefore went straight to the rest of the process.

The Instructional Design process would not be complete without going through a process of evaluation. For this process, certain guidelines laid out primarily in the text by Dick, Carey and Carey (2000) are utilized. In the developmental process, various other sources were consulted, but the primary rubrics were from the named text.

The One-to-one Evaluation

In the one-to-one evaluation of instructor materials, the process involved review by a subject matter expert, and two (2) members of the target population of learners. The results of these processes as summarized, are presented herein

Evaluation by Subject Matter Expert

Rough copies of the following were presented to the expert:

1. The instructional analysis

2. The Goal statement and the behavioral objectives

3. The Instructor Materials

4. The new JBTE Teaching Practice Assessment Form

5. A copy of a blank Assessment Form

A single subject matter expert was used due to the following:

1. The individual is an evaluation specialist

2. The individual has some 30 years experience performing such tasks

3. The individual has been selected as an international reviewer of Teacher College Examinations for the Caribbean region including Belize on the Central American Mainland, and Guyana in South America.

4. The individual has acted as an external Examiner for the internationally bases JBTE in the past.

5. The number and nature of the learning tasks did not merit further consultation, since it involved a specific situation where teacher trainers were required to learn only the specifics of using the new assessment form.

What was Expected of the Expert

The author, in the meantime, looked for the following in the comments by the expert:

1 Statement of the objectives

2 Analysis of the instruction

3 Accuracy and currency of the content

4 Appropriateness of the instructional materials in vocabulary, interest value, sequence, and learner participation activities.

5 Clarity and appropriateness of evaluation criteria

6 Relative position of this activity in consideration of other activities

The scoring of the above will be reflected in attached table(s).

Review of Materials by Participants (Teacher Trainers)

Two (2) participants were randomly chosen for this exercise. A total time of fifty (50) minutes was spent with each person. There was a fifteen (15) minute break between each session .

Under normal circumstances, a minimum of three evaluations are done per category of learners or "users". These categories/classes would be as follows:

(a) Above average performer(s)

(b) Average performer(s)

As with the irrelevance of the pre- and posttests, the administering to the above three categories was found to be irrelevant, because teacher trainers (the target group of learners) are all high achievers who have many years of experience using evaluation forms and need no validation/baseline test.

Suggested changes "Content"

There were a few inconsistencies in the design document. These include double words in the text (based on typographical errors), construction errors in terms of tense, and congruence with the class of users such as "teacher trainers" as against "student teachers", or other workers. These were pointed out by the subject matter expert.

The subject matter expert also suggested changing the designated "student" (of the form's use), to bona fide "learners". Not only is this a more globally accepted designation, but "student learner" could be confused with "student teacher" who is a trainee teacher

The "working" definition of "planning" was preferred to the technical definition according to the expert..

Suggested changes - "Process"

The most obvious change recommended is timing. The evaluation should take place when there is sufficient time for all participants to be less anxious about the process.

It is further suggested that a category labeled "other factors" be included for the future. This will provide a forum for non-represented comments by participants. At times, these can guide format and lead to improvement in the modus operandi of evaluation schema.

Constraints Faced During Evaluation

The project was conducted within a purely academic environment. Teachers and students face major final examinations for the Spring Semester. Constraints are thus divided into two (2) categories. These are as follows:

1. Learner constraints

(a) Culminating activities such as term papers, action research reports, and the sitting of examinations were imminent, and participants felt pressured to participate in the activities, since the mental "load" was heavy in the work environment.

(b) In the one-on-one evaluation, learners (teacher trainers) felt that group interventions were better than single opinion contributions to the design process. This is debatable, since our textbook recommends single learner inputs.. They felt that their singular decisions and suggestions were too pivotal in determining the directions for the future, and that this may negatively influence the fate of significant others.

2. Examiner Constraints

(a) Chief External Examiner had current system-wide external assessments to complete, and was very busy. Timing was thus a problem.

(b) The Examiner was trained in Great Britain, and as such, had a differing standard of spelling for some items. These were corrected accordingly, after dialog.

Recommendations for Future Actions

1. The 'bottom up' approach in designing the assessment form (teachers in the

system gave their input before policy was made) was appreciated by all, but some teacher trainers suggested that the materials should have been tested in all territories before being used system-wide.

2. It was suggested that different videotapes should be used for each parameter to be

evaluated. The constant replay of the same tape accounted for a certain degree of boredom on the part of the learners. It is to be noted that scores for :sufficiency of variation of content/examples, and appropriateness of demonstrations were scored low by both the external examiner and the target learners. This may have been as a direct result of the repetition of the videotape for each parameter in turn.

Table 1: Results of Subject Matter Expert Evaluation of Instructor Materials

(The Subject Matter Expert may represent a typical instructor who would deliver the instruction)

To indicate how appropriate you find the instructor materials insert a tick in the appropriate slot. A score of "U" will indicate that you have not given a grade, "1" indicates the lowest grade and 4 indicates the highest.

Components	Criteria – Clarity of Instruction	U	1	2	3	4
Messages	M1 Vocabulary is appropriate					X
“	M2 Sentences are concise					X
“	M3 Messages are precise					X
“	M5 Introductory statements are appropriate					X
“	M6 Explanations are adequate and clear				X
“	M7 Concluding statements are appropriate					X
“	M8 Transitions are smooth,				X
Links	L1 Instructions are set in the proper context					X
“	L2 Examples are appropriate					X
“	L3 Analogies are relevant					X
“	L4 Illustrations are appropriate					X
“	L5 Demonstrations are appropriate			X
“	L6 Reviews are appropriate					X
“	L7 Summaries are appropriate					X
Procedures	P1 Sequence is logical					X
“	P2 Segment size is appropriate – not too large or too small				X
“	P3 Transitions are smooth - not too sudden					X
“	P4 Pace is timely – not too fast or too slow					X
“	P5 Presentation is sufficiently varied			X
	Criteria - Impact on the Learner
Attitudes	At1 Information and skills are relevant					X
“	At2 Information and skills can be learnt with reasonable effort					X
“	At3 Satisfied that skills are adequate				X
Achievements	A1 Overall appropriateness for this situation					X
“	A2 Appropriateness for future use					X
	Criteria – Feasibility
Learners	Lr1 Pitched for experience and appropriate level of maturity					X
“	Lr2 Experience appropriate level of independence					X
“	Lr3 Experience high level of motivation					X
Resources	R1 The time is adequate for instructions					X
“	R2 Equipment is appropriate and adequate for the learners				X
“	R3 Environment is appropriate and user - friendly					X

Please write additional comments (including suggested modifications) in the spaces below.

Table 2: One-to-One Learner Evaluation Form For Instructional Materials

(Learners are Teacher Trainers in the College)

Show how meaningful and appropriate (helpful) you found the instructional materials by inserting a tick in the appropriate slot. A score of "U" indicates that you have not given a grade, "1" indicates the lowest grade and 4 indicates the highest grade assigned.

Components	Criteria – Clarity of Instruction	U	1	2	3	4
Messages	M1 Vocabulary level is appropriate					X
“	M2 Sentences are concise				X
“	M3 Messages are precise				X
“	M5 Introductory statements are appropriate					X
“	M6 Explanations are adequate and clear					X
“	M7 Concluding statements are appropriate					X
“	M8 Transitions are smooth,				X
Links	L1 Instructions are set in the proper context					X
“	L2 Examples are appropriate					X
“	L3 Analogies are relevant				X
“	L4 Illustrations are appropriate				X
“	L5 Demonstrations are appropriate			X
“	L6 Reviews are appropriate					X
“	L7 Summaries are and appropriate					X
Procedures	P1 Sequence is logical					X
“	P2 Segment size appropriate – not too large or too small					X
“	P3 Transitions are smooth - not too sudden					X
“	P4 Pace is timely – not too fast or too slow					X
“	P5 Presentation, sufficiently varied			X
	Criteria - Impact on the Learner
Attitudes	At1 Information and skills are relevant					X
“	At2 Information and skills can be learnt with reasonable effort					X
“	At3 Satisfied that skills are adequate					X
Achievements	A1 Has overall appropriateness for this situation					X
“	A2 Appropriate for future use					X
	Criteria – Feasibility
Learners	Lr1 Experience appropriate level of maturity				X
“	Lr2 Experience appropriate level of independence				X
“	Lr3 Experience high level of motivation			X
Resources	R1 The time was adequate for instructions					X
“	R2 Equipment was appropriate and adequate for the learners					X
“	R3 Environment is appropriate and user - friendly					X

Please write additional comments (including suggested modifications) in the spaces below.

Evaluation of Student Materials

Learning Specialist

The Learning Specialist took the form and the materials and went through them alone without the Instructors Manual.

Using the Evaluation sheet as an instrument to judge the Learner Materials the Learning Specialist suggested the following changes for the Learners’ Package:

1. Direction are needed for the user of the package

2. Self Instructional Print Materials need to be integrated with Information from Instructors Manual because M4, M5, M6, M7, L7, P4, A1, A2, & R1 could not be observed (e.g. material was written to be self- explanatory within a seminar so participants were directed to listen when CEE gave motivational speech).

Comments for improvement of the form are:

1. There is an error at L7, the word “and” needs to be removed from between

“are” and “appropriate”.

2. the change of the word “components” at the top left hand column of the

form. The term decided on was “

3. There is need for a posttest slot on the form, under the heading,

“Achievement”.

These changes were withheld for the one - on – one evaluation since the material is being designed for the learners and the designers felt that it was wise to hear from the learners before making any changes.

One–on-one Evaluation

The One – on – one evaluation was carried out on the Learner package. This process was done with 3 different Subject Matter Experts (SME) from the target group. The designer sat and read the Instructors’ Manual, to each participant, going through the process, as if at a seminar. Each of these persons read the Self Instructional Print Material and carried out the process as instructed. The results were:

· Each person asked for directions to use the package

Comments for improvement of the form are:

· Suggested changes at R1, and R2 for the verbs to be changed to reflect the

present tense for consistency, “was” changed to “is”.

After this process, the designer will go through the suggestions and decide which aspects of the package will be changed. The necessary changes are then implemented.

Table 3: Results of Subject Matter Expert Evaluation of Student Materials

Table 3: Results of Learning Specialists Evaluation of Student Materials

Indicate how helpful you find the student material by inserting a mark in the appropriate slot. U indicates that you have not graded the material, 1 indicates the lowest grade and 4 indicates the highest.

Components	Criteria – Clarity of Instruction	U	1	2	3	4
Messages	M1 Vocabulary is appropriate					X
“	M2 Sentences are concise					X
“	M3 Messages are precise					X
“	M4 Introductory statements are appropriate	X
“	M5 Explanations are adequate and clear	X
“	M6 Concluding statements are appropriate	X
“	M7 Transitions are smooth	X
Links	L1 Instructions are set in the proper context					X
“	L2 Examples are appropriate					X
“	L3 Analogies are relevant					X
“	L4 Illustrations are appropriate					X
“	L5 Demonstrations are appropriate					X
“	L6 Reviews are appropriate					X
“	L7 Summaries are and appropriate	X
Procedures	P1 Sequence is logical					X
“	P2 Segment size appropriate – not too large or too small					X
“	P3 Transitions are smooth, not too sudden					X
“	P4 Pace is timely – not too fast or too slow	X
“	P5 Presentation, sufficiently varied					X
	Criteria - Impact on the Learner
Attitudes	At1 Information and skills are relevant					X
“	At2 Information and skills can be learnt with reasonable effort					X
“	At3 Satisfied that skills are adequate					X
Achievements	A1 Overall appropriateness for this situation	X
“	A2 Overall appropriateness for future use	X
	Criteria – Feasibility
Learner	Lr1 Experiences appropriate level of maturity					X
“	Lr2 Experiences appropriate level of independence					X
“	Lr3 Experiences high level of motivation
Resources	R1 The time was adequate for instructions	X
“	R2 Equipment was appropriate adequate for the learners					X
“	R3 Environment is appropriate and user - friendly					X

Table 4: Results of One–to –One Student Evaluation of Student Package

(3 members of the target group)

Indicate how helpful you find the student materials by inserting a mark in the appropriate slot. U indicates that you have not graded the material, 1 indicates the lowest grade and 4 indicates the highest.

Components	Criteria – Clarity of Instruction	U	1	2	3	4
Messages	M1 Vocabulary is appropriate					xxx
“	M2 Sentences are concise					xxx
“	M3 Messages are precise					xxx
“	M4 Introductory statements are appropriate					xxx
“	M5 Explanations are adequate and clear					xxx
“	M6 Concluding statements are appropriate					xxx
“	M7 Transitions are smooth				x	xx
Links	L1 Instructions are set in the proper context					xxx
“	L2 Examples are appropriate					xxx
“	L3 Analogies are relevant					xxx
“	L4 Illustrations are appropriate					xxx
“	L5 Demonstrations are appropriate				x	xx
“	L6 Reviews are appropriate					xxx
“	L7 Summaries are and appropriate					xxx
Procedures	P1 Sequence is logical					xxx
“	P2 Segment size appropriate – not too large or too small				x	xx
“	P3 Transitions are smooth, not too sudden					xxx
“	P4 Pace is timely – not too fast or too slow				x	xx
“	P5 Presentation, sufficiently varied				x	xx
	Criteria - Impact on the Learner
Attitudes	At1 Information and skills are relevant					xxx
“	At2 Information and skills can be learnt with reasonable effort					xxx
“	At3 Satisfied that skills are adequate					xxx
Achievements	A1 Overall appropriateness for this situation					xxx
“	A2 Overall appropriateness for future use					xxx
	Criteria – Feasibility
Learner	Lr1 Experiences appropriate level of maturity					xxx
“	Lr2 Experiences appropriate level of independence					xxx
“	Lr3 Experiences high level of motivation					xxx
Resources	R1 The time was adequate for instructions				x	xx
“	R2 Equipment was appropriate adequate for the learners					xxx

EXPERT EVALUATION OF ASSESSMENT INSTRUMENTS

A Specialist in testing and measurement was asked to appraise the checklists that are to be used for the ultimate assessment of the learners’ performance. Table 5 shows the results of this appraisal.

Table 5: Evaluation Form for Appraising Assessment Instruments

Directions: Kindly peruse the attached assessment instruments and rate the elements by placing a tick (Ö) in the appropriate column. Items should be rated from 1-4 where 4 is the highest and 1 the lowest. “U” indicates that you have insufficient grounds on which to score.

Please feel free to make comments.

Observations	U	1	2	3	4
1. Instruments are congruent with the objectives					Ö
2. Instructions are clear.					Ö
3. There is sufficient time for completing the items.	Ö
4. The number of items is reasonable.					Ö
5. Each item is clearly stated					Ö
6. Items are properly sequenced					Ö
7. Vocabulary is easily understood					Ö
8.The format is appropriate					Ö
9. The scoring procedures are easily understood					Ö
10. Feedback to learners is indicated					Ö

COMMENTS: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

The expert rated nine of the items at ‘4’ the highest score. She could not rate item four since the designer had not indicated the time in which the instrument should be completed. This will be corrected for the one –to-one evaluation.

ONE-TO-ONE EVALUATION OF ASSESSMENT INSTRUMENTS

Table 6 shows the results of the one-to-one appraisal of the assessment instruments.

Table 6: Evaluation Form for Appraising Assessment Instruments

Please feel free to make comments.

Observations	U	1	2	3	4
1. Instruments are congruent with the objectives				Ö	ÖÖ
2. Instructions are clear.				Ö	ÖÖ
3. There is sufficient time for completing the items.				ÖÖ	Ö
4. The number of items is reasonable.				Ö	ÖÖ
5. Each item is clearly stated			Ö	Ö	Ö
6. Items are properly sequenced				Ö	ÖÖ
7. Vocabulary is easily understood				Ö	ÖÖ
8.The format is appropriate		Ö	Ö	Ö
9. The scoring procedures are easily understood					ÖÖÖ
10. Feedback to learners is indicated					ÖÖÖ

Most items were given fairly high to high scores. However, item #8 was given consistently low scores. It would seem as if the would-be learners are not familiar with an assessment form on which the learner assesses himself and the instructor does so too. The designers have decided to retain the format for the small group and field trial phases of the evaluation process.

APPENDIX : EVALUATION REPORT FORM - FORATIVE EVALUATION

Blank Evaluation Form

To indicate how helpful you found the student materials inserting a mark in the appropriate slot. U indicates that you have not given a grade, 1 indicates the lowest grade and 4 indicates the highest.

Components	Criteria – Clarity of Instruction	U	1	2	3	4
Messages	M1 Vocabulary is appropriate
“	M2 Sentences are concise
“	M3 Messages are precise
“	M5 Introductory statements are appropriate
“	M6 Explanations are adequate and clear
“	M7 Concluding statements are appropriate
“	M8 Transitions are smooth,
Links	L1 Instructions are set in the proper context
“	L2 Examples are appropriate
“	L3 Analogies are relevant
“	L4 Illustrations are appropriate
“	L5 Demonstrations are appropriate
“	L6 Reviews are appropriate
“	L7 Summaries are appropriate
Procedures	P1 Sequence is logical
“	P2 Segment size appropriate – not too large or too small
“	P3 Transitions are smooth - not too sudden
“	P4 Pace is timely – not too fast or too slow
“	P5 Presentation, sufficiently varied
	Criteria - Impact on the Learner
Attitudes	At1 Information and skills are relevant
“	At2 Information and skills can be learnt with reasonable effort
“	At3 Satisfied that skills are adequate
Achievements	A1 Overall appropriateness for this situation
“	A2 Appropriateness for future use
	Criteria – Feasibility
Learners	Lr1 Experience appropriate level of maturity
“	Lr2 Experience appropriate level of independence
“	Lr3 Experience high level of motivation
Resources	R1 The time is adequate for instructions
“	R2 Equipment is appropriate and adequate for the learners
“	R3 Environment is appropriate and user - friendly

Please write additional comments (including suggested modifications) in the spaces below.

________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

EVALUATION FORM FOR APPRAISING ASSESSMENT INSTRUMENTS

Please feel free to make comments.

Observations	U	1	2	3	4
1. Instruments are congruent with the objectives
2. Instructions are clear.
3. There is sufficient time for completing the items.
4. The number of items is reasonable.
5. Each item is clearly stated
6. Items are properly sequenced
7. Vocabulary is easily understood
8.The format is appropriate
9. The scoring procedures are easily understood
10. Feedback to learners is indicated

Back to Top

Back to Home Page

Group Members: Keena Douglas keenad@nova.edu Jonathan Lamey lameyj@nova.edu Monica Shakespeare monshake@nova.edu Nadine Salmon nsalmon@nova.edu Appendixes

Group Members:

REPORT ON FORMATIVE EVALUATION

OF INSTRUCTOR MATERIALS

Table 4: Results of One–to –One Student Evaluation of Student Package

Group Members:

Keena Douglas keenad@nova.edu

Jonathan Lamey lameyj@nova.edu

Monica Shakespeare monshake@nova.edu

Nadine Salmon nsalmon@nova.edu

Appendixes