1. What are the key characteristics of experiments? This part of the process looks at procedures and experimental validity issues. The researcher should discuss weaknesses and other limitations in the methodology procedures. Some of the key characteristics of experiments are: (1) that they should express what you are striving for, (2) that they should be able to control variables of interest, (3) that they will be able to manipulate variables, and (4) that they need to be able to observe the dependent variable.

2. What are the general aims of experiments? The general aims of experiments are: (1) that they demonstrate a relationship between variables, (2) that they try to establish general laws about the relationship, and (3) that they realize that threats exist which can hamper the researcher’s ability to draw the proper conclusions.

3. What are the four types of experimental validity? The four types of validities are: (1) construct validity, (2) statistical conclusion validity, (3) internal validity, and (4) external validity. Construct validity is similar to instruments. It assumes the nature of the variable, is protected by a strong literature review and strong instruments, and it includes defining and measuring. The idea is to get a well-defined construct to avoid overlap (ways to think about this are things like self-esteem and self-concept which are not really the same thing). Statistical conclusion validity answers whether or not two variables co-vary. This experiment is protected by strong procedures, good sampling, and a strong data analysis section. Internal validity determines whether or not one variable affects another. It is protected by a strong experimental design. External validity determines whether or not the results of the study are generalizable. It is also protected by a strong experimental design.

4. What research question is associated with each type of validity? With statistical conclusion validity, the question is, Do two variables co-vary? With internal validity, the question is, Does this variable affect another variable? With external validity, the question is, Are the results of this study generalizable?

5. What are the names and definitions of the threats to construct validity? The threats to construct validity are: (1) inadequate preoperational explication of constructs (did a poor job of defining construct before experiment (missed or added pieces, probably a weak literature review), (2) mono-operational bias (only used one instrument to measure construct, so must show that that instrument is established and accepted), (3) mono method (measure variable with only one type of instrument), (4) hypothesis testing (subjects in study try to guess purpose of the experiment, and this affects some of their answers), (5) evaluation apprehension (test anxiety will affect the study), (6) experimenter’s expectancies (subjects in the study pick up on what the experimenter expects, and this can force results and hurt neutrality), (7) compounding constructs and levels of constructs (a treatment is effective at one level, but increasing it will lead to its being more ineffective (diminishing returns) è shock treatment is like this because a little is beneficial but too much can be dangerous with extreme discomfort), (8) interaction of different treatments (must let one treatment wear off before administering the next), (9) interaction of testing and treatment (given a pretest, the subject realizes what the researcher is measuring, and that can affect later results), and (10) restricted constructs generalized across other constructs (multiple constructs can interact just like multiple treatments).

6. What are the names and definitions of the threats to statistical conclusion validity? The threats to statistical conclusion validity are: (1) low statistical power (type I or II error è instrument or experimental design is too weak and cannot detect the type of error), (2) violated assumptions of the statistical tests (assumptions stated in the statistical portion of the report on problems in the Analysis of Data section), (3) fishing and error rate problem (associated with ANOVA (differences detected among different groups) è the test will only tell you where there is a difference but not which group is different, but there are posttests that will solve this problem (Sheffany or Potucky)), (4) reliability of measures (the instruments are not reliable so hard to draw desired conclusions), (5) reliability of treatment implementation (inconsistency in administering either treatment or instrument), (6) random irrelevancies in experimental setting (random, not systematic (noise, windows, and other distractions)), and (7) random heterogeneity of respondents (trying to divide and compare two groups, but in one group, the researcher somehow has two groups (or subgroups) where one subgroup does well but the other does not).

				Tom of Bethany

Back To TLEE's Home Page

Index to Selected Essays And Book Reviews

Lesson 19. Experimental Validity (Part II)

Send email to: tlee6040@aol.com