[Note for bibliographic reference: Melberg, Hans O. (1997), Three statistical
examples: Boeing, sex and education, A follow up on How many examples do you need to
make generalizations http://www.oocities.org/hmelberg/papers/970127.htm]
Three statistical examples: Boeing, sex and education
A follow up on last week's How many examples do you need to make
generalizations
by Hans O. Melberg
Last week I argued that our background beliefs about how things are related are important
when we decide how much weight to put on one example. This week I shall give three more
examples.
1. Boeing's generalization from two incidents
The Economist reported on 25. January 1997 (p. 25) that the Boeing was going to
make some modifications to the rudder of all 2700 Boing 737 planes. The reason for this
was two unexplained crashes with this aircraft in the USA. Now, one may ask whether two
unexplained crashes is enough to be even remotely certain that the rudder was causally
involved. As with the bees I suspect that two examples are enough (but less so in this
examples than with the bees). The background beliefs about how planes are designed, tested
and how crashes are examined make me relatively certain that they have found a weakness.
Moreover, one could also argue that even a relatively weak belief should be acted upon
when the consequences of not doing so could be fatal.
2. Christians and Sex
The newspaper Vart Land recently wrote about an interview survey of 22 randomly
picked girls (from three religious organizations) which supposedly showed that they
attempt to convey Christian norms about sex was a "total failure." (Tone M.
Soedal, "Sex-forkynnelse treffer ikke de unge", 20. January, 1997, p.7.) I
simply do not think that a survey of 22 girls is enough to prove this. Moreover, in an
interview situation the respondents are likely to give the answers that the researcher
wants AND the researcher is likely to put emphasis on what she wants to hear. Finally the
data is far to weak to be sub-divided into smaller categories to draw conclusions based on
this sub-sample. For example, there are not enough examples to argue that the girls from
one organization has a different attitude and behaviour than girls from another
organization.
3. Educational achievement and the number of books your parents have
Another newspaper-article (by Berit Kolberg in Dagbladet) writes about a survey of
10 000 youth aged 13 - 18 in which the researcher tried to find which factors determined
the educational achievements of the youth (measured by their grades). Clearly, 10 000
examples should be enough to draw reliable conclusions, but one may question whether the
effort was worth the rather unsurprising results. For example, I was not surprised to
learn that the number of books in the family is a reliable predictor of educational
achievement. Nor was I surprised to learn that children from broken homes or children with
parents on social welfare, received lower grades than average. I was, however, mildly
surprised to learn that immigrants tended to be more diligent than other students.
Overall, here we have enough examples, but few interesting results.
A more serious problem was the implied message in the article that there was a causal
connection - not only a correlation - between the number of books in a family and
educational achievement. I do not believe the researchers made this connection, but the
article may be misunderstood to mean that the correlation implies causation.
Overall
Boing, sex and education - Where does this leave us? I believe all three examples
illustrate the importance of always considering the context - the background of causal
beliefs - before we examine one particular issue using statistics. Sometimes this
background leads us to believe that a few examples are enough to draw conclusions and act
on these (bees, planes); Sometimes 22 examples are not enough (since I believe there is
great variation in the behaviour of Christian girls unlike the behaviour of bees); And,
finally, sometimes 10 000 examples are enough but not really interesting since it only
confirms what we strongly suspected (maybe even reliably so i.e. that it was not a mere
prejudice). Moreover even strong correlation in a sample of 10 000 does nor prove that
there is a causal relationship. (See my comments on proofs by correlation as opposed to
proofs by inference in "The Cultural Approach to Russian Politics"). Overall
conclusion? Beware of statistical evidence!
Note: If you want to read more about the misunderstood concept of statistical
significance, there is a good article by Deirdre McCloskey and Stephen T. Ziliak
("The Standard Error of Regressions") in Journal of Economic Literature
34 (1996), pp. 97-114).
[Note for bibliographic reference: Melberg, Hans O. (1997), Three statistical
examples: Boing, sex and education, A follow up on How many examples do you need to
make generalizations http://www.oocities.org/hmelberg/papers/970127.htm]