[Note for bibliographic reference: Melberg, Hans O. (1997), The criteria for good
academic work - 1996: My aim and the year in review http://www.oocities.org/hmelberg/papers/970303.htm]
The criteria for good academic work
1996: My aim and the year in review
by Hans O. Melberg
By what criteria should we judge academic works? In the following I shall suggest and
expand on five criteria. A good piece of work should be surprising, convincing, important,
honest and well written. In expanding on these labels, I shall also try to comment on some
of my own works in order to see where I can improve.
1. Surprising
When I read an article or a book I want to learn something I did not know before. This, in
short, is what I mean when I say that a work should be surprising. Maybe surprising is the
wrong word since it is possible to learn something new which is not very surprising. It
might, therefore, be more correct to say that a good work should contain information -
ideas, analyses - which I have not thought about, or considered to be true, before I read
the paper: It should be non-obvious.
Now, the reader might ask, the demand that a work should be surprising is itself not a
very surprising claim. Maybe not. However, it is also true that I have wasted a lot of my
time reading obvious conclusions and analyses. To mention two examples. In an otherwise
decent book, Stephen White mentions that 1.1% of the respondents in a survey "stated
that they would listen with more attention to a lecturer who simply read out the text in
front of him" (p. 118). He also wrote that "the better educated [representatives
of the nationalities] appear ... more likely to know Russian" (p. 150, Political
Culture and Soviet Politics). There are many other examples of obvious conclusions and
studies, such as serious research concluding that children with parents who have many
books tend to get better grades than children of parents with few books. Hence, while the
demand may not be surprising, actual practise demonstrates the necessity of stressing the
criterion that we should aim at non-obvious knowledge.
Jon Elster once wrote that surprising results are not the only results worth pursuing
since we may use research to transform prejudices into well founded beliefs. (A chapter on
Basic Research in the book Vitenskap of Politikk). I partially agree. However,
imagine the choice between two activities - one which has a high probability of leading me
to revise my beliefs in toward a more correct position and one which might lead me not to
revise my beliefs though the research may make the beliefs better founded. I would choose
the first activity: It is better to be right for the wrong reasons than to be wrong! (or?)
At least in a pragmatic sense since you base your actions on the beliefs. Though, the
problem might be that we do not know which activity has the highest probability of leading
to revision of beliefs. Although we do not have detailed probability estimates, I do think
I am more likely to revise my beliefs when I read a book by authors with opinions that
(initially) differ from mine. This is why I read Jon Roemer's book about socialism, since
I tend to be a conservative/liberal and he is not. Thus, reading him may lead me to change
my beliefs - at least the probability for this is higher than if I read a book by a
conservative with whom I broadly agree.
Personally, my own ideas may not be too surprising. The main reason being that I am
still in the process of learning what other writers have written. However, I try to write
about topics which are debated. In order to make sure do this I must always quote an
author who has an opinion different from me, as I tried to do extensively in the long
paperThe Cultural Approach to Russian History and in general by always starting
from concrete examples.
2. Convincing
By convincing, I mean that the conclusion should be backed by solid evidence: There should
be a high probability that the conclusion is true. Now, this is certainly not a surprising
requirement, but it is possible to elaborate a bit by pointing to how I think it should be
done. In the previous year I have given quite a lot of thought to this and in my paper The
Cultural Approach to Russian Politics I outline the two routes to knowledge: through
statistics (correlation) or through deduction from a few axioms. (see also the paper Tre
årsaker til urettferdigheter). Once again this might not be too surprising, but it is
important to go into the details of the matter: What are the strength and weaknesses of
each approach. This is what I tried to find out in my paper Against Correlation and
some of my other papers on statistical hypothesis-testing. So far my suspicion is that the
deductive/axiomatic approach often gives the best returns, but this is a very context
dependent conclusion (i.e. depending on what you want to investigate).
3. Important
What makes some topics or conclusions important? Ultimately this is a value-judgement, but
I think an insights is more important than others when it has the potential significantly
increase the welfare of many people (i.e. quantity times quality). Of course, we disagree
on what constitutes a better situation, but I am unsure about the extent of this
disagreement. Surely, basic welfare, liberty, opportunities to use your abilities
(self-realisation) (and many other values) all make up a part of what we think is a good
life. Not all research has the same potential for improving the lives of people and I
would like to concentrate on the research that has the largest potential.
With these grand words it is time to provide some concrete examples. Consider Jon
Elster's words that "The basic postulate from which I start is that the goal of the
social sciences is the liberation of man." (Jon Elster, Logic and Society
(1978), p. 158).Once again, the proof of the pudding is in the details. For example, the
idea of precommitment as a way to improve your situation (and the situation for a
collective community) and research to become aware of the unconscious mechanisms that
shape our desires and beliefs in order to avoid the mistakes resulting from these beliefs
and desires. Elster's work is full of these very practical and important ideas (since they
have the potential to improve the existing situation) and this is one of the reasons why I
take such a keen interest in his work.
I shall use one more example to illustrate the difference between more and less
important ideas. Consider an idea which will improve the efficiency of a factory. For
example, a new way of organising the workforce. Now, consider another idea - for example
new legal rules which sharpens the competitive environment for all firms. Assume that the
consequence of this last idea is to increase the growth rate of the economy from 2 to
2.5%. The difference between the first and the second idea is that the first produces a
one-time improvement, while the other idea gives a permanent higher growth rate. It seems
to me that the second kind of ideas are most important since the consequences over time
are so much greater. This, of course, does not mean that we should exclusively
concentrate on research into the second kind of ideas. One-shot gains are important as
well, but it is a question of degrees of importance. Consider the following: What activity
gives the highest payoff - to read a book or attend a course which will enable you to read
faster/learn better. Clearly, we should not spend all out time on only one activity (what
is the point of being able to learn fast if you never spend any time on learning
something!) Yet, the payoff from the second activity (learning how to learn) has, until a
limit, the highest payoff.
4. Honest
A work is honest if the author openly admits its weaknesses and his doubts. For example,
John Roemer openly admits in his book A Future for Socialism that "we do not
have a definite solution to this problem [the problem of who is going to monitor the
monitors of firms] " (p. 76). Jon Elster is also very good at admitting when he is
wrong or on weak grounds. Why is this important? The most important reason is that we live
in a second best world in which we have to trust each other to some extent. It is simply
impossible to be an expert on everything. Nor is it possible to know the history of all
the authors you read. Hence, when Jon Elster wrote an article on Constitutionalism in
Czechoslovakia, he started by admitting that he did not know the language, nor was he an
expert on the history of Czechoslovakia. This information is important for a person
reading the article because it affects the way he should read the article. For example, my
review of Richard Dawkins' book River out of Eden: A Darwinian View of Life is less
authoritative than a review written by a person knowledgeable about biology. Thus, to
avoid that people take my review more seriously than it is intended, I have to admit my
limitations.
There is one problem with the discussion above: that the history of ideas should not be
allowed to interfere with their evaluation. Simply knowing that I do not know much
biology, does not entitle you to dismiss my review of Dawkins. The only proper way to
attack an argument is by pointing to facts which are directly relevant to the
truth-value of the argument. Thus, the arguments made by Elster in his article on
Czechoslovakia cannot be dismissed by saying "He is no expert on the history of
Czechoslovakia." Instead the reviewer has to point out exactly what is wrong with his
arguments. I have read many arguments in which these two ways of arguing is confused. For
example, Aasmund Egge argued that my review of his book was inspired by extreme-right wing
American historians - a fact which is not relevant to the truth of my arguments. Another
example is Brian Barry's review of G. A. Cohen's book in TLS. In this review, Barry argues
that Cohen's individualistic approach is inspired by his membership in the September Group
[a group of methodologically conservative but politically radical intellectuals] and that
Cohen is inclined to neglect alternative meanings of self-ownership since he is
"reluctant to have his fox [Nozick] shot" when it has "given him such good
sport." This may or may not be true, but in a review I believe it is better to focus
on the quality of the arguments, not their origins or the (here: subconscious!) motives of
the author. The two are obviously distinct: The quality of an argument should assessed on
its own and NOT (not even implicitly as I think Barry does with Cohen) be dismissed by
pointing to its sources.
Well, if the history of ideas are irrelevant to their evaluation, why do we need to
know the limitations of the author? To be honest, I need to think more about this, but I
have a beginning of an answer. The stated limitations may point to where we should search
in order to find mistakes and weaknesses. It may turn out that the limitations do not
affect the quality of the arguments, but in a time-scarce world it is good if the authors
are honest enough to guide the reader as to where they should search. Moreover, there may
be some limitations which creates a relationship between the history of ideas and their
reliability. In short, in an ideal world the two should be kept distinct, but given
limited information and time I am willing to trust an author who I know more than an
author which I do not know. Maybe the books of David Irving contains a few interesting
insights, but the fact that he is obviously wrong about many things makes me more cautious
when I read him than when I read Elster. Is this correct? Not in an ideal world, but given
the limitations on the time available to me it might be a good strategy (defined in terms
of which strategy would give me the most correct beliefs over a lifetime). Or?
In any case, in a review I am not allowed to use the source of ideas to judge its
truth-value. I might use the source as a guide to its weaknesses and to how I should
approach the book. However, in the final review only the arguments count, not its sources.
5. Well written
This criteria is rather obvious and does not deserve much elaboration. In this category I
include structure, clarity and beauty of language. I also have to confess that I probably
need to improve on this account. Although my writings are reasonably clear, there is much
room for improvement in terms of grammar, punctuation and language. To give three
examples: I should use more understatements (use words like weakness instead of flaw - see
my apology to Aa. Egge); I should learn the English comma rules (!); and I should end my
schizophrenic switching between American and British English. This is partly a question of
spending enough time on the papers, and partly a question of reading more good literature.
Not to be included?
There might be other criteria which should be included on the list above. For example, I
often read, and sometimes I write this myself, that a book is good because it is
stimulating or fruitful in generating further research. However, I am more and more
convinced that these labels should be used less as a mark of quality. Anything can be
fruitful and stimulating. A book which is stimulating for me can be utterly boring for
another and there is no neutral criteria to distinguish the two. If one by fruitful means
that the ideas in the book have many important consequences, then this is already covered
by my criterion labelled "Important." Moreover, I do not want to make a
trade-off between fruitfulness and reliability. It is easy to conceive of some fantastic
idea which might be very stimulating, but the idea and the thoughts it generates may be
totally unreliable. No, I agree with Jon Elster who writes that what we need is not
exciting conclusions, but reliable conclusion. If it is surprising this is an additional
bonus.
Lastly, I am unsure about two things. First, whether there might still be some criteria
which I have forgotten (and which does not fit into my five categories). Second, whether
the criteria allow for trade-offs or if there is some lexicographic internal hierarchy.
For example, imagine the a choice between two books - one which is highly reliable,
moderately surprising but not very important and another which is moderately reliable,
highly surprising and very important. It seems to me that the second book would be worth
reading before (or instead of) the first. However, I am also deeply reluctant to trade
reliability (truth) with some of the other criteria. Maybe there should be some kind of
complicated weighting system in which reliability was given a large but not exclusive
role.
Overall
Where does this leave me? In order to better fulfil the above criteria, I shall - for a
trial period - write two versions of my papers. The first week shall be a draft, the
second week I shall try to improve the draft. In this way I shall force myself to think
consciously about the above criteria. It will reduce the quantity of papers, but I hope it
will improve the quality.
Addition 16. April, 1997:
A confused and abstract addition!
I read a couple of ariticles in Apollon (A magazine distributed by the Univerity of Oslo)
which convinced me that I am partially wrong when I insist that a work should have
important implications (criterion 3) in order to be judged worthy of publication. The
argument goes as follows: There is - and must be because of scarcity of time - a certain
division of labour within science (social science as well as natural science). Thus, for
the group it might be worthwhile to make one person to spend much time on what looks like
a "small" or unimportant question. If we all behaved like isolated individuals -
only focusing on the questions with "big" implications - this might produce a
collectively worse result than if some people are directed to the small question. (Also:
Some small questions may turn out to have important implications, a fact we can only know
after research has been conducted - research none will do if they individually only choose
the big questions). So, practical limitations means that it is worthwile for some to
concentrate on apparently small questions with few important implications.
The second point I want to add, is that I am still uncertain about the rejection of
fruitfulness as a criteria. I admit that I find some models more fruitful than others (in
terms of raising questions for further research). However, I also remember an article by
Jon Elster with some relevance to this question. Some people might find that taking a bath
gives them good academic ideas. Others might find that a certain frame/model/structure is
fruitful in generating ideas. Maybe it is valuable to create such frames, but, as the
example with taking a bath shows, there is no limit to what can be considered
"fruitful." The source of creativity is personal. Thus, I am still reluctant to
include fruitfulness as a criteria, but I am uncertain. Maybe people are quite similar so
that we are inspired by the same things? In this case it would be valuable to presents
frames/models which opens up the creativity of many people.
Sources:
1. An issue of Appollon (late 1996 or early 1997) with several articles on good/bad
science. I was particularely inspired (!) by an interview with Dagfinn Føllesdal)
2. Jon Elster, Høyre eller Venstre, Nytt Norsk Tidsskrift (a critique of an
article by Hanne Haavind)
[Note for bibliographic reference: Melberg, Hans O. (1997), The criteria for good
academic work - 1996: My aim and the year in review http://www.oocities.org/hmelberg/papers/970303.htm]