This book was developed by the Evaluation Training Consortium (ETC) project at the Evaluation Center, Western Michigan University.

The purpose of this book is to assist personal responsible for use in training, teacher education and other professional development programs in private and public agencies, public school, colleges and universities. By using this book, the user be expected to become more systematic in their program development activities and at the same time, help them to recognize that their tasks can never be totally organized or at times even remotely logical or orderly. Rather, program planning and evaluation is a dynamic process, set within the confines of ever changing organizational needs, priorities and values. This book can serve as practical guide for educators in order to improve their program evaluation quality.

The book is organized into two major areas. The first area is the Sourcebook , which contains chapters of guidelines, resources and references for each of the seven evaluation functions. It provides optional approaches, guidelines, and procedures for designing and conducting evaluation to meet training needs. It looks at the decisions that evaluators needs to make most often and provides the user with just those aspects of program evaluation that are more critical for success. The Sourcebook lists seven major evaluation functions, identifies thirty-five critical questions and tasks (p. 2-4), and shows alternative procedures for accomplishing the evaluation task at hand. Each question addressed in the sourcebook proposes some options for the evaluator in dealing with a question or issue, provides alternative procedures available to accomplish the evaluation tasks related to the issue. Each question also provides guidelines and criteria that help the evaluator to determine whether the job has been done adequately.

The seven major evaluation functions are:

a. focusing an evaluation and clarifying its purpose,

b. designing an evaluation,

c. collecting information,

d. analyzing information

e. reporting: interpreting and using evaluation findings,

f. managing evaluation activities,

g. evaluating evaluation efforts.

Each function is defined, then the several key decisions needed to complete the function are explained. The sourcebook contains examples, guidelines, criteria and checklists such that user can use to do more effective evaluation. Each function elaborated in a number of questions that helps plan evaluator. For example, in function: focusing an evaluation and clarifying its purpose (a), the question is "what is the purpose for evaluating?" The general purpose for evaluating should be clear. To determine the most appropriate evaluator and evaluation strategy, it’s important to know why the evaluation taking place. In the casebook there is an example of evaluation purposes (Case L1, p. 227). It also includes references to other books and resources that can be useful in evaluating training programs.

The second area is Casebook, which shows how others have designed conducted and used program evaluation procedures as a regular part of their work. These absorbing case studies illustrate the problems of people who are trying to perform evaluation productively while facing real-life obstacles. Discussion questions provided with each case study get user involved in asking the tough questions user want to be prepared to answer when it comes time to be evaluated in his own work. The casebook, which contains collection of twelve stories about evaluation applied to real-life projects and programs in different settings. These show people planning, conducting and using evaluation. Each of case-examples is a story about evaluation within a particular training program. The case examples, designed to portray evaluation applications of different types in different settings, were contributed by field practitioners and written in conjunction with ETC staff. They are fictional accounts but based on actual programs and used of evaluation. Each case-example is annotated to highlight the seven major evaluation functions as set forth in the sourcebook. This is done to show how these functions differ according to particular program needs and settings. Following each case is a set of review and discussion questions to help extend the lessons available in the case-example.

Design manual, which separated part of sourcebook and casebook, contains worksheets, directions and guidelines for designing an evaluation. Its organization is similar to the sourcebook, as it helps us produce different parts of an overall evaluation design. Each section presents an example of the design product needed; give us worksheets, directions and aids for producing that document and provides a checklist for assessing our work.

For using this book, there is no one particular order in which these materials are meant to be used. We could begin in any of the three parts, using them alone or in combination. Where we begin and how we use the materials depends on what we want to use them for.

• The structure of the book easily can be understood. The book provides a hands-on, simple, succinct, "how to" guide to program evaluation. The book contains examples (example of an object description, p. 14-15; example of evaluation design, p. 49-58), diagram (diagram of ways to describe objects, p. 9), criteria and checklist (checklist for evaluation design, p. 47-49) we can use to more affective evaluation.
• It provides many alternative ways to think about and to plan evaluation. While every evaluation design is essentially made of similar elements, there are many ways to put elements together. The three major decisions that determine the general approach to evaluation are: (a). fixed vs. emergence evaluation design; (b). formative vs. summative evaluation; and (c). experiment/quasi-experiment design vs. unobtrusive inquiry.
• There are many tables and figures have been included to efficiently summarize information.
• Case scenarios and practical examples are given to demonstrate how each component can be applied in the work setting.
• It provides many alternative procedures available to accomplish the evaluation tasks related to the issue (p. x-xi). For examples, to learn more about evaluation applications in our setting: we should read the case-examples for our setting(local school, college), to become more knowledgeable and proficient in evaluation, read the sourcebook, and then try some more evaluation work.
• It provides guidelines and criteria that help the evaluator determine whether the job has been done adequately.
• There are suggestions for using the materials of this book, depending what the purpose of our program evaluation, for example: to learn about how evaluation could be used to help with particular problems; to learn more about evaluation application in our setting; to learn more about evaluation in general and how it fits in with training and to train others.
• The marginal notes (in the left side each page of casebook) are intended to help us understand what is happening "evaluation-wise" in the cases. For example, in page 227 there are focus questions of each paragraph. In the first paragraph, the focus is on context of evaluation: "what is the setting", in the second paragraph, the focus is object: "what should be evaluated" and so on.

The weakness:

• There is not glossary, that is a list of technical or special words, in the context of program evaluation, especially those occurring in a particular text that can help reader to get the meaning or explanation of the technical words.
• There are many technical terms from statistic that are not adequate explained or footnote, author had better refer technical term from statistics to others application statistical book, for example: Krathwohl, 1998; Grimm, 1993 Minium, 1993, or Guilford, 1978.
• There are many technical terms from evaluation (validity, reliability) with brief explanation, author had better suggest reader by footnote: "more information about this technical term you can get explanation in book……".
• It would have been better if the author to give many cautions and warnings (common errors) that might be done by researcher or evaluator. Evaluator can make common errors in every stage of program evaluation. For example:

• In focusing the evaluation, especially identify audience or stakeholders, the evaluator fail to distinguish between clients and other stakeholders, overidentifying stakeholders making it impossible to proceed (Sanders 1994). Occasionally, evaluators overlook the needs and rights of stakeholders because of their gender, ethnicity or language background. Once in a while, evaluators focus on program goals and objectives identified only by program management and staff; evaluators fail to include the perspectives of various stakeholders.
• In designing evaluation, the evaluators fail to establish the evaluator’s credibility in both content and methodological areas. Usually, evaluators assume that evaluator’s approach to evaluation is acceptable to the client. The evaluators design the data collection and analysis procedures without considering what criteria, such as performance by a comparison group or performance in term of predetermined standard, will be needed to interpret the finding.
• In collecting information, the evaluators fail to give voice to multiple stakeholder groups in the process of selecting priority evaluation questions, or the evaluators fail to consider the trade-off between comprehensiveness and selectivity at every stage of the evaluation (Sanders 1994). Some times, evaluator choose a data collection and analysis plan from a research methods textbook or other general guide without considering whether the plan can be carried out in the giving setting. It also occurs that evaluators collect information because it is convenient, for instance, because instruments already exist, rather than because it is necessary. Often, evaluators fail to consider characteristics of respondents that may affect their ability to understand or respond to a procedure as implemented, for example physical handicaps, language proficiency and reading ability.
• In analyzing and interpreting evaluation, the evaluators fail to check the accuracy of the obtained description of the program through direct examination or observation or by confirmation by program personnel (Sanders 1994). Failing to determine what value perspectives for instance: educational, social, economic and scientific of the client and stakeholders perceive to be important in interpreting the finding of the evaluation. In analyzing quantitative information, evaluator often assume that significant statistical results are necessarily of practical significance; and interpret non significant statistical results because they happen to have practical significance, unless there is evidence of replicability. In analyzing qualitative information, evaluators fail to distinguish among different sources of qualitative information on such basis as credibility, degree of expertise and degree of involvement. Some times evaluators fail to consider alternative interpretations of reality and the multiple value perspectives that may exist in an evaluation situation.
• In reporting evaluation, the evaluators direct the report to the client or sponsor while ignoring other intended users. Reporting a judgment about or interpreting finding as either a strength or weakness without considering alternative perspectives that might change that conclusion. Evaluators often determine stakeholders for the evaluation reports on the basis of convenience or economy rather than on the basis of ethical and legal considerations; and provide selected information to some members of a group affected by the evaluation and not to other members of groups.
• In managing evaluation, the evaluators believe that calling the attention of the client to a real or potential conflict of interest will be sufficient to correct the problem, especially when public disclosure or corrective action may reflect adversely on the client. Occasionally, evaluators assume that result from test scoring machines and computers are accurate just because they have been produced by machine (Sanders 1994). Evaluators often assume that people who collect information follow administration instructions correctly and that respondent read, understand and follow all the directions they are given.
• In evaluating evaluation (meta evaluation), evaluators fail to answer meta evaluation question as the following: Were the findings of the program evaluation clear? Were the design and implementation of the design appropriate for the study? Were the conclusions justified by data? Were the results adequately disseminated? And so on. Some times, evaluators assume that every program evaluation study requires a formal meta evaluation study and assign internal meta evaluation responsibility to a junior staff member.

• There are many inaccurate statistical concepts, for example:

• Brinkerhoff’s statement about sample sizes (p. 90) that the larger sample, the more precise the generalization. It is not absolutely true, because there are others factors that can influence the precision of the generalization. The factors are:

1. the more precise we desire to be about the exact nature of the target

population

2. the more the units in the sample vary from one another on the

characteristic of interest-being heterogeneous rather than homogeneous.

3. the smaller effect to be sensed relative to the normal variation among the

sampling unit. (Krathwohl, 1998). In addition, we can use sequential

until some criterion of adequacy is met.

• In analyzing and interpreting evaluation information, Brinkerhoff gave a chart which portray what is the typical score which represents the middle of the group (p. 131). In ordinal data, he suggests using mode or median; and for interval/ratio data he suggest to use mode, median or mean. But he didn’t give criterion for which distribution we use mode? and for which distribution we use mean or media? More explanation about that one, we can refer to Grimm(1993) or Minium (1993).
• In correlational analysis, he ignores (didn’t emphasize) many factors that can create misleading correlation coefficients, for example: effect of outliers, effect of range and effect of nonlinearity (Krathwohl 1998, Grimm 1993).

• About standards or characteristic of effective program evaluation, there are four major characteristics to effective program evaluation: utility standards, feasibility standards, propriety standards, and accuracy standard (page xix). However, Brainard (1996) add relevance standards. Relevance means that the focus of a program evaluation and the information-gathering methods need to meet the objectives of the evaluation. Relevancy also is determined by the extent to which a program evaluation provides results that are helpful in developing or improving the program.

The main aim of the study is to develop some chapters of a mathematics textbook based on realistic mathematics in senior secondary school. These exemplary materials (chapters) should provide a model for mathematics textbook in secondary education. This study may start with a front-end analysis in order to obtain a clear picture of the starting situation and the main purpose of the study. In this step, prototype of exemplary materials will be developed. Based on reflections of developers on the prototype and formative evaluation results, prototype will be continually refined and evolve towards a final deliverable. Results of the formative evaluation may lead to revision of the prototype. In this way, each prototyping cycle represents the evolution of the system. Based on that process, it is needed to interview and to distribute questionnaire to target group (student), experts, teacher and other interested groups. In order to improve present performance of teacher in the classroom, which will teach with new approach (realistic mathematics), a training program for teacher will be developed. This training program fosters three kinds of learning outcomes: acquisition of a new knowledge, skill development and attitude change. Furthermore, in order to know student learning progress, the test will be conducted. Therefore, in focusing the evaluation, we should consider many elements in the setting that probably will influence the evaluation. In order to handle this problem, we can use key issues: "what elements in the setting are likely to influence the evaluation" in page 23-26. In designing evaluation we need to assess the quality of the design evaluation. For this aims, key issues: "how do you recognize a good design" page 64-71 might be help us. Additionally, we should use this book together with Tessmer’s book, as Tessmer was mention that the focus of his book is on the formative evaluation of instructional products such as texts, lectures and multimedia instruction (Tessmer, 1993). In the step collecting data by interview, questionnaire or test, information in the book about "collecting information" page 77-115 apparently important for us. With the information in this part of book, we can:

• Determine the information sources we will use,
• Decide how we will collect data
• Decide whether we need to sample and, if so, how
• Determine how precise our information must be and design a means to collect it
• Establish procedures to maximize validity and reliability
• Plan the logistics for an economical information collection procedure.

In the step analyze the information and summarize result, we must begin the important work of pulling together the information we have gathered. Most information will be either qualitative (from interview) or quantitative (from questionnaire and test). Information (data) must be properly handled and stored in order to prepare it for analysis. This includes coding data, aggregating and organizing it, and storing it for safekeeping and ready access. The main idea in ways that facilitate its use, and keep it from getting lost or forgotten. For support this aim, page 119-122 (aggregate and code data if necessary), page 123-126 (verify completeness and quality of raw data), page 127-144 (select & run defensible analysis) and page 145-147 (interpret the data using prespecified and alternative sets of criteria), can help us. However, in analyzing quantitative data, due to inaccurate many statistical concepts in this book, we should consult standard statistics reference books, for example Krathwohl, (1998); Grimm, (1993) Minium, (1993), or Guilford, (1978).

Education and training programs are evaluated in order to determine their quality and gain direction for improving them. It is needed clear definition of what constitutes a reasonable evaluation of educational program. There is other book, Sanders (1994), which provide standards for the practice of educational program evaluation. This book contains a set of standards that speaks to the responsible conduct and use of educational program, project and materials evaluations. The standards provide a guide for evaluating educational and training programs. It is better if this book (Brinkerhoff’s book) commonly use with Sander’s book.

To acquire that book, we can found in Toegepaste Onderwijskunde (TO) library with catalog number: TO 371.2 p 089. If we want to buy that book, we can contact to distributors:

1. Kluwer Academic Publisher Group, Distribution Centre, PO Box 322

3300 AH Dordrecht, The Netherlands.

2. Kluwer Nijhoff Publishing, Kluwer Boston Inc.

190 Old Derby Street Hingham, Massachusetts 02043, USA.

Brainard, E.A. (1996). A Hands-on Guide to School Program Evaluation,

Bloomington: Phi Delta Kappa Educational Foundation

Grimm, L.G. (1993) Statistical Applications for the Behavioral Sciences. New

York: John Wiley & Sons.

Guilford, J.P & Fruchter B, (1978). Fundamental Statistics in Psychology and

Education, (6^th Edition), New York: McGraw Hill.

Krathwohl, D.R. (1998). Methods of Educational & Social Science Research 2^nd

Edition. New York: Longman.

Minium, E.W. ; King, B.M.; Bear,G. (1993) Statistical Reasoning in Psychology

and Education. New York: John Wiley & Sons.

Sanders, J.R. (1994). The Program Evaluation Standards, 2^nd Edition, London: Sage

Publications.

Tessmer, M. (1993). Planning and Conducting Formative Evaluations, London:

Kogan Page