Research On Instructional Development

Development research in the CASCADE study(chapter 3 , N. Nieveen)

The aim of the CASCADE study was to develop a computer support system which assist curriculum developers in optimizing the effectiveness of their formative curriculum evaluation efforts and its contributions to development methodologies for lesson materials as well as computer support system.

Requirements for a high-quality supports system(three quality aspects):

Validity, Practicality, and Effectiveness.

Three key components of a software product(CASCADE System):

Content, support, and user interface
Therefore, the three quality aspects are considered to be requirements for each key

Meeting the aim and requirements

Activities in Development Research:

Front end analysis(problem and context analysis)
Prototype development:
Analysis of ‘the start-of-the-art knowledge’ which is made explicit in a growing conceptual framework and included in the prototypes;
Design prototypes;
Formative evaluation of prototypes;

Summative evaluation of the final product when it is implemented in practice
Reflection on the development methodology

Front end Analysis

1) The first idea/background of CASCADE that a computer support system for formative curriculum development may be worthwhile for curriculum developers of the SLO

2)The preliminary study was carried out in order to further explore these first ideas

3) Theoretical framework was gathered in order to gain deep understanding of computer-supported curriculum development and to link this notion to the SLO context

Prototype Development( Two stages)

The Prototyping Stage- concentrate on the development of CASCADE

Focus on : validity and practicality of the content, support and user interface

5 prototypes of CASCADE:

preliminary computer-based prototype; - throw-away prototype
paper-based prototype;
first computer-based prototype;
second computer-based prototype;
final version –built in Delphi Programming environment

Computer-based Prototype are designed using Apple Macintosh-based prototyping tool called ‘HyperCard’ and its programming language called ‘HyperTalk’

Population and Sample: Two main type of respondents: users and experts

Three major formative evaluation approaches were chosen:

cooperative evaluation, b)walkthrough c)expert appraisal 2) The Assessment Stage

Focus on: practicality and effectiveness of CASCADE for a larger target group

Research Question:

What is the practicality and effectiveness of the final version of CASCADE in the SLO practice?During the preparation and performance of formative evaluation activities

The Methodology of Developmental Research

(Richey, Nelson; 1996)

Defining the Research Problem

Focus on particular aspect of the design, development, or evaluation process.

Establish research parameters and identify the scope of study: development

Evaluation, revision & testing

Goals of DR guided by research question instead of hypothesis

The Review of Related Literature/The Conceptual Research

General Research , Literature Review focus on dependent/independent variable
Development Study, directly to relevant study

The Research Procedure

Longer than other types of research: due to detailed research procedure & timelines
Procedures:
Population and Sample

designers, developers, evaluators, instructors, program facilitators, and learners

Research design

Overview of common used research methods in DR studies

---------------------------------------------------------------------------------------------------

Type of DR Documentation of DR Primary Research Methods Used

---------------------------------------------------------------------------------------------------

1 Yes case study, evaluation

2 sometime survey, experimental, evaluation, qualitative

---------------------------------------------------------------------------------------------------

Data Collection and Analysis

Result and Conclusions

Result: - dissertation/ reports

- articles(in professional journals)

Conclusions:

all project attempt to make additions to the field's knowledge of instrctional design, development, and evaluation
done by constructing new procedural research, or describing 'lesson learned' in a particular project.

Educational Design and development: An overview of paradigms

(Visscher, I.V.& Gustafson, K. & Plomp, Tj, 1998)

Four Perspectives on how the educational design and development processes might be conducted.

educational development is depicted as an instrumental process that start with the formulation of specific goals and outcomes for the product to be developed.

Example of this instrumental paradigm: planning-by-objectives approach(vignette1 )

As a communicative process in which product is developed in close interaction and communication between the development team and others involved.

Example of communicative paradigm: consensus approach is in vignette 2

As a pragmatic process in which developers create their product by quickly building, testing, and revising several prototypes of the product. See vignette 3; (pragmatic paradigm:interactive and repeated tryout and revision).

Developers act like an artist(artistic procedure), who, based on their unique expertise and experience, create their products in response to the specific situation in which they work. See vignette 4(artistic paradigm: creation of products based on connoisseurship approach).

Summary the four paradigm & their vignettes is in table 1. Refer to good design on each

Paradigm	A design product is good…	The design process is good….
Instrument	If it meets a pre-specific good, standard; If there is a clear relationship between goals, learning situations, and process, and outcomes	If it starts with a thorough needs and problems analysis, in a specification of the solution If it occurs rationally, logically, and systematically
Communication	If it meets the standard as discussed and agreed upon by the team and other stakeholders If the standard is shared by all those involved	If it contains activities aimed at, at resulting in, reaching clarity and consensus among team members about what to make and how to make it
Pragmatic	If it has been proven useful and effective with and for users	If design and evaluation activities are intertwined If prototypes are regularly tested with users for their usefulness and effectiveness
Artistic	If it meets the developer's own subjective quality criteria	If the developer relied on his/her own judgment and has used that judgment well

Differences in functions of formative evaluation, analysis & prototype are in table 2.

--------------------------------------------------------------------------------------------------------------------------

Paradigm Function of analysis Function of evaluation Function of prototype

--------------------------------------------------------------------------------------------------------------------------

pragmatic Most analysis activities are To test part of a product Global outlines of a product can

conducted by means of in practice to get insights function as a prototype. They are

evaluating prototypes about its quality and to evaluated to identify requirements

specify further design for design and to determine usefulness

requirements and effectiveness of product

--------------------------------------------------------------------------------------------------------------------------

The differences between role of the developer and relation with client in table 3.

--------------------------------------------------------------------------------------------------------------------------

paradigm Role of the developer Relation with client

--------------------------------------------------------------------------------------------------------------------------

pragmatic Developers work out of prototype, with or Users have substantial influence on

without the client, and test them to users the shape of product

--------------------------------------------------------------------------------------------------------------------------

Conclusions:

Better if developers should be able to combine elements form each of the paradigms into a unique model for a specific project.
All paradigms and their accompanying perspectives, tools, and techniques can and do play useful roles in designing effective, efficient, relevant, and engaging educational experiences.

Evaluation Studies and Action Research (chapter 23. Krathwohl, R.D)

Evaluation study has some characteristics:

It is decision driven instead of hypothesis
A Consensus limited to evaluator and clients
Utilization is a prime criterion
Process evaluation as important as product
Users involved in process evaluation and as a controller

(for this point, in action research the primary audience is the researcher themselves)

Some evaluation studies used to improve a product/program- formative evaluation

And others for establish the project determining its value - summative evaluation

Formative evaluation is used in the development stage to guide an evolutionary process

Alternative Evaluation Approach

Connoisseurial or expert-judgment evaluation(Eisner 1976, 1981)

- evaluations employ experts in the field to provide a judgment

the oldest form of evaluation
to minimize charge of bias, structure the evaluations in many ways all agree are fair, state criteria in written publicly form, involves multiple judges, issue a written report with opportunity for response, and provide for a review of the report and procedure

Objectives- and Goal-based evaluation

focus on the intended goals of a program
a major question is, whose version of intended goals is used, that of sponsor, administrator, staff, clients, or other stakeholders

Goal-free evaluation

Evaluators determines what occurred and compares that with intended changes

Management-oriented evaluation

It design to assist managers in the administration of programs. These approaches are often concerned with efficiency and effectiveness

Adversarial evaluation

Evaluators prepare cases to represent two or more alternative points of view on a problem and present their cases as in a court of law

Consumer-oriented evaluation

Evaluators are intended to help consumers decide what program to use, what product to buy and so forth.

Response naturalistic evaluation

Evaluations uses qualitative methods to describe what occurred and to seek to be responsive to the stakeholder audience

Participation-oriented evaluation

The stakeholders are involved in determining the evaluation process

Action Research

The evaluation is planned, carried out, and interpreted by those wanting the results for their own decision making. They may prepare a report for colleagues and others like them. Professional evaluators may be used as consultants