[Note for bibliographic reference: Melberg, Hans O. (1997), Using popularity as a
judge of quality - The year in review 1997, www.oocities.org/hmelberg/papers/971222.htm]
Using popularity as a judge of quality
The year in review, 1997
by Hans O. Melberg
Introduction
It may, perhaps, be naive to believe that we can learn lessons from history, but on a
personal level I nevertheless feel it is valuable to reflect a little on the past academic
year. By looking at past mistakes, I hope to improve my future papers. The problem, of
course, is that I can hardly be a neutral judge of my own papers. To solve this problem
one might try to use popularity as a "neutral" judge of quality.
Popularity as a judge
The page averages about 450 hits every week. Of these, by far the most popular paper, is Lessons from History: The collapse of Communism and the end of the Cold
War. This immediately makes me suspicious about using popularity as a judge of quality
since I personally do not consider "Lessons from History" to be among the best
papers. A look at the complete "list of top hits" confirmed my fears: Some of my
worst papers received many hits but some of the decent papers received few hits. The list
for 1997 is as follows:
The most popular papers
1. Lessons from History: The collapse of Communism and the end of the
Cold War (1200)
2. Evolution or Creationism: Does science and religion compete in the
same arena? (480)
3. A review of reviews: Making Sense of Marx (460)
4. Three arguments about rational choice theory in sociology
(340)
5. The pros and cons of state intervention (320)
6. Inflation: An overview of theories and solutions (290)
7. Four distinctive aspects of Soviet and Russian Military Thinking
(280)
8. Pushkin or Baumol - Poets, Economists and Contestable Markets
(270)
9. The Cultural Approach to Russian History - How reliable? (250)
10. Economic sense and non-sense: A review of Krugman's Peddling
Prosperity (240)
11. Three statistical examples: Boeing, sex and education (230)
12. Why don't people believe that free trade is good? (210)
The list of most popular papers contains several essays that in my opinion are far from
my best. "Evolution and creationism" is, maybe, not bad, nor is it particularly
good. The fourth most popular paper, "Three arguments about rational choice theory in
sociology", almost qualifies as bad. Contrary to what I wrote then, I now think
emotional behaviour and rational behaviour often conflict (My anger may make me hit you,
and this action is usually not "rational"). Also, I am much more unsure about
the problem of including psychic benefits. In one sense they should be included (all gains
should be counted), but including these benefits also makes the theory close to worthless
(or at least much more difficult to test). In the same category - close to bad - comes
some of my other old essays (before 1996), such as "Inflation: An overview of
theories and solutions" and "Four distinctive aspects of Soviet Military
Thinking." Of these, the first is incomplete and too informal. The second is better,
but not very original, and plagued with some stylistic problems (for instance the use of
the current tense of the verb). In sum, the most popular papers are not the same that I
think are my best papers.
What are my best papers, and why is there a divergence between my judgment and that of
my readers. The first question can be dealt with by making an alternative list of
partially acceptable papers:
My list of potentially good papers (1997 only)
How (not) to explain puzzling Russian behaviour - A Review of R.
Hingley's The Russian Mind
This, I believe, is good enough to be published. Structured, concrete, balanced.
Why don't people believe that free trade is good?
This needs to be rewritten before it can be published - for instance, the afternote should
come first (a slightly better version was published in Norwegian in Observator no.
3, 1997). It is, hopefully, a little original - at least I have not seen the question
discussed in this way elsewhere.
The assumption of rational selfishness - How can it be justified?
When should it be rejected?
Combined with the essay "What is economics", extended (maybe include something
about "economic functionalism" and its dangers) and rewritten (discussing a few
more concrete issues), this could be a good article.
Why (dis)believe rational expectations? A review of Sheffrin's
Rational Expectations
S. Sheffrin was kind enough to reply to the review, and he though the review raised
important issues that all economists should think about. Of course, I do not claim that my
discussion is original - but I do think it is quite complete and informative. (But, the
last part - about "an interesting idea" and artificial equilibriums - should be
cut).
The criteria for good academic work
If one eliminates the comments about my own papers, and introduces more/better examples,
this could become a decent paper.
What is economics? A method or a topic?
Presents a frame and some arguments which could be developed into an interesting paper.
And the list of not so good papers
Boeing, sex and education
It does not really add anything to the more interesting Bees, marbles
and generalizations based on one example
How bureaucracies waste resources - And how it could be avoided
Not so much wrong, as just obvious, not very reflective and systematic rambling on a topic
which I have already written a slightly more better observation (see Externalities,
bureaucracies and tradable budgets)
Non-utopian utopians? John Roemer's Coupon System
Way too long, not focused, some unfair criticism (the parallel with "scientific
socialism"), though parts of it is still valuable (and the criticism about nostalgia
for the Soviet system is not unfair).
Justifying the welfare state: Biased but useful (review of Barr)
Although some of the arguments are OK, the essay is stylistically horrible and needs to be
rewritten. Especially the section about the dangers of state intervention is weak (the
argument can be strengthened by pointing to different arguments about the dangers of state
power than the somewhat feeble argument that the state can abuse the information it has in
its registers.)
Nationalism: Is it definable, important and worthy of attention?
The problem with this review, is that I did not go "deep" enough. For instance,
whether some variables are "resistant" to rational investigation calls for a
much closer thought than I gave it.
Culture - An explanatory variable or an interpretive approach?
This paper, believe it or not, is not too bad, but it became far too long and should be
rewritten focusing on the issue that became clear only towards the end of writing the
essay (how culture can be used to explain actions when we define culture as a set of
meanings)
How does one explain the second issue - the divergence between my judgement and that of
my visitors. Some of the possible reasons can be discussed under the headings:
accessibility, interest, marketing, headlines.
My most popular paper, "Lessons from history", was originally a lecture for
high school students. The written version is a little more advanced, but it is still quite
accessible (I think). The same cannot be said about all of my other essays. Although
intelligible, I doubt that high school students have the necessary background to
understand, say, my review of Maddala's book "Econometrics."
But, accessibility - I think - is not the main reason. A more important factor is
probably interests. A paper that is good need not attract a lot of visitors if it is about
a topic which few find interesting. My more methodological work about culture may fall
into this category, as well as some of the speculative thoughts about statistics.
Marketing is maybe the most important cause of the different number of hits. I know
that "Evolution vs. Creationism" is linked from another page, and many of my
hits probably come from this page. The same is true of "Three arguments about
rational choice theory in sociology" which is both linked from two other pages, and
(undeservingly) put on the reading list at an American university. In any case, I will not
spend too much time marketing the page; if the page is good the number of visitors will
grow without my intervention.
The headlines are probably also important in attracting hits. I suspect the title of
"Boeing, sex and education" attracted some visitors - some of whom must have
been greatly disappointed. In any case, people are much better off reading "Bees,
marbles and generalizations based on one example" than the mentioned article. Another
article which has a uninteresting title, is Counterproductive
which does not deserve to be among the papers with the lowest number of hits (32 in a
year).
Sub-conclusion
For a number of reasons discussed, I do not think popularity is currently a good guide for
me if I want to discover which papers are good and bad. In the long run - and I mean the really
long run - sustained popularity may be a better guide to quality. But we do not live in
the long run so this source of information is unavailable.
One possible, and feasible, source of quality analysis is comments from
"experts." In the next year I will actively try to seek such comments, for
instance by sending my reviews to the authors and asking for criticism. The replies will
be published on my page (See Comments) and linked from the
papers.
I have discussed several possible ways of judging the quality of a paper; by its
popularity, by comments from experts, and subjective opinion (adjusted for the possible
biases due to interests, marketing etc.). I do not dismiss any, even short run popularity
is sometimes right. After all, "Why don't people believe that free trade is
good" and "The cultural approach to Russian history" are on the top 12
list. I do, however, feel that more expert comments would be valuable.
The lessons and what's next?
This year I wrote 37 papers for this page (this excludes several works in Norwegian, some
of which were published in Observator - a Norwegian journal run by students at the
Department of Economics, the University of Oslo). 20 of these were
"Observations", 17 were book reviews and none deserve to be called articles.
Looking back, I now realize that I should have written at least one article - for instance
the observations "What is economics" and "The assumption of rational
selfishness" could be combined to make a decent article. The same can be said of the
many articles about culture, which at one point will be integrated into a more coherent
whole. This must one goal for the year ahead: To write at least two articles.
Looking back I wrote about three main topics: Economics (mainly state intervention),
Culture (and how it is used as an explanatory variable), and Statistics (the concept of
significance). I would like to expand a little, to read more about political theory,
mathematics, the natural sciences and philosophy. However, having already promised to
write two articles, I cannot also promise to expand my choice of topics.
I do want, however, to read more classics, more advanced books, and less
"popular" books. For instance, I did not have time to read "Hume and the
problem of causation" but I did read rather "light" books like Skitovsky
"After Communism", Barro's "Getting it Right", Krugman's
"Peddling Prosperity" and Landsburg's "The Armchair Economics". Surely
something must have gone wrong with my priorities when I found the time to read these
"light" books and not the time to read the high quality "heavy" books.
This is a lesson well worth taking into the next year. To make this more than mere words,
I have already bought a number of classics that I am going to read: Weber, Tocqueville,
Adam Smith, and Keynes to mention a few.
A word about style
I am aware that my less than perfect command of English, as well as lack of time and
attention, sometimes makes my English appear clumsy. For this I can only apologize, and
try to improve. However, I promised this last year too, yet the number of typing errors
has only increased. In the future I will try to eliminate the errors from the last week's
paper before I publish a new paper.
Conclusion
Whatever its use for other people, this page is useful for me as a motivator, an archive
and as a method for forcing me to think and write. This alone is reason enough for me to
continue to work with the page and the papers.
[Note for bibliographic reference: Melberg, Hans O. (1997), Using popularity as a
judge of quality - The year in review 1997, www.oocities.org/hmelberg/papers/971222.htm]