[Contents Page] [Previous section] [Next section] Back to my homepage
| This page [2a]: | 2.1 Use |
| 2.2 Citation frequency | |
| 2.3 Citation analysis | |
| 2.4 Local applications | |
| 2.5 Other evaluation methods | |
| Following page [2b]: | 2.6 The age of citations |
| 2.7 Other limitations of citation counts | |
| 2.8 Irrelevant applications |
The literature freely employs or measures "use" and "usage," without definition.[1]
The principle of usefulness says simply that libraries should collect what patrons use. An obvious problem is that there is no clear definition of what comprises "use" ...[2]
This project must also apply "usage" and similarly fuzzy terms, but with the context hopefully conveying meaning. In general, "using" a source means referring to it for its contained information --- sufficiently so as to require, or encourage, a citation.[3] Quotations from the literature may very well imply or support other meanings.
Even if use is explicitly defined, the question still arises "Are usages equal?"
A journal "used" thirty times may produce as much "value" as one used sixty times. One session with the American Historical Review may occupy more time (and perhaps be worth more) than several uses of Time or even the National Lampoon.[4]
In research, citing a source merely implies that it was used by the author or probably has utility for the reader:
... a frequently cited publication is a measure of the utility of that particular publication, not a judgment of its importance or impact.[5]
Even an author who cites sources without reading them is assuming that they are actually useful, or is relying on their presumed, apparent utility.
However, not even all citation-related usages are equal.[6] Furthermore not all users exhibit the same quality of use.
Some citations may be 'sloppy' in the sense that the citing author may not have carried out an exhaustive literature search, or may simply have misunderstood the arguments in the cited article.[7]
Nor is convenience or ease of access to a source necessarily a primary factor in determining how often that source is cited. This was evident in a parallel study of undergraduate psychology theses at two different, but analogous, U.S. universities where "[local] availability of journals does not necessarily influence their citation frequency."[8] Furthermore, the papers of any one author or particular journal may be cited erratically --- some popular, others ignored.[9] For researchers, this effect may compound reiteratively:
The two phenomena, library use and citations, are products of basic scholarly activities: searching for material, reading it, and writing about it ... which includes the citation-leads-to-use-leads-to-citation cycle.[10]
Nonetheless, analyzing citations, by frequency, can indicate both the "core" works or journals and the comparatively poorer performers;[11] i.e., targets for acquisition and deselection.
Citation studies have emphasized the differences between patterns of materials used, obsolescence, authorship, and institutional affiliation in social science and humanities research. These important categories ... when applied to the universe of published literature, ... have special implications for collection development in research libraries. ... the findings of such studies can prove invaluable when making decisions concerning "deacquisition," collection evaluation, and collection building efforts.[12]
We may artificially distinguish:
A reference study gives artificial equality to all items in the bibliographies,[14] whereas a citation study reflects all the actual uses (which more accurately indicates the comparative research value of each source). Reference studies also overemphasize the importance of "ornamental" quotes or citations, especially when implicit references are counted.[15] Conversely, citation counts may only overemphasize reference works which are crucial to the analyzed work(s). It is, perhaps, debatable whether this truly distorts the citation count; but I shall follow Budd's opinion that
With primary materials, especially in literature, the counting of subsequent references actually can be misleading. ... It is more useful to identify primary work as the central focus of study of the source item ...[16]
The goal in collection management should not be to build large comprehensive collections, but rather to build collections suited to the needs of our primary users. Local citation studies can provide information useful in developing collections to fulfill this objective.[17]
Some authorities deny this moderate view;[18] or add the proviso "if it is used in conjunction with other evaluative measures."[19] Yet "quantitative methods, other than citation analysis, are time-consuming, complicated, and expensive to conduct."[20]
Others authorities mistakenly interpret citation frequencies as linear parameters of value; a common assumption of articles using citation analysis results to allocate various Serials:Monographs ratios for the acquisitions budget.[21] Incidentally these sources disagree as to whether or not, in citation counts, these ratios are consistent over time.[22]
Although "the study of only locally produced citations may skew the study towards materials the library already owns;"[23] perhaps the converse is more important --- locally significant resources may not feature well in national or international citation databases.[24] Indeed local, rather than national, citation data has been shown to be a better predictor of local future usage.[25]
Furthermore, theses make excellent specimens for the evaluation of such local resources:
Librarians should study the products of research undertaken, or presumed to have been undertaken, in the library under study. Theses and dissertations serve as a convenient source of such in-house research.[26]
Finally, building and maintaining a local collection involves more than merely acquiring popular items:
... in a large collection, ... journal rankings from citation studies are useful for determining which journals should be monitored for theft, vandalism, and wear, acquired in multiple copy for branch libraries, and obtained in microform for preservation.[27]
Other methods include journal reshelving [alias "usage"] studies which may correlate well with citation studies.[28] However citation counts measure only journal "use" by published researchers,[29] relevant to the purpose of a research library, whereas shelving studies measure journal "usage" by any and all of the library's clientele.[30] Clearly these two groups are different and unequal:
... for many research libraries a journal that provide relevant material for faculty and students in their scholarly pursuits is preferable to one that people merely thumb through. It is to the former that citation studies relate.[31]
Furthermore, as Griscom asks, "How can one be certain that an item pulled off the shelf was actually used by the patron?"[32] Similarly, Swigger and Wilkes question the validity of reshelving studies,
... because the meaning of "use" in this context is most unclear, and at best the uses are unequal. All that is really counted is the number of times a journal was left unshelved. ... Because the behavior is unobserved, one is not sure what one is counting.[33]
Finally, citation counting is both a repeatable and an unobtrusive data collection procedure.[34] Therefore citation counts automatically have partial guarantees of reliability and validity that are unavailable to reshelving studies (see §7.4 below).
Some authorities criticize citation counts because they fails to reflect the journal preferences of patrons as deduced from inter-library loan [I.L.L.] statistics.[35] Others counter-argue that there actually is significant correlation between the results from analyzing inter-library loans and from citation counts.[36]
1 E.g., John Laurence Kelland and Arthur P. Young, "Citation as a Form of Library Use," Collection Management 19, no. 1/2 (1994), 82:
The frequency of usage of materials in different parts of a library's collection has been measured in various ways: external circulation, measuring in-house use, and citation frequency of material in a particular collection. Compared to the first two approaches, the latter does not directly measure library use.
2. Keith Swigger and Adeline Wilkes, "The Use of Citation Data to Evaluate Serials Subscriptions in an Academic Library," Serials Review (Summer 1991), 42.
4. In the opinion of Robert N. Broadus, "A Proposed Method for Eliminating Titles from Periodical Subscription Lists," College & Research Libraries 46 (Jan. 1985), 30. See also Barbara A. Rice, "Science Periodicals Use Study," Serials Librarian 4 (Fall 1979), 36.
5. E.g., James K. Bracken and John Mark Tucker, "Characteristics of the Journal Literature of Bibliographic Instruction," College & Research Libraries 50 (Nov. 1989), 667. Similarly Per O. Seglen, "The Skewness of Science," Journal of the American Society for Information Science 43 (Oct. 1992), 636 and Pam Royle, "A Citation Analysis of Australian Science and Social Science Journals," Australian Academic & Research Libraries 25 (Sep. 1994), 162.
6. Mengxiong Liu, "Progress in Documentation. The Complexities of Citation Practice: A Review of Citation Studies," Journal of Documentation 49 (Dec. 1993), 390. See also Bluma C. Peritz, "On the Objectives of Citation Analysis: Problems of Theory and Method," Journal of the American Society for Information Science 43 (Jul. 1992), 449:
... in many fields, the community of producers of research results is different from the community of their users. ... Moreover, citation of a study because of its connection with the subject matter of the citing paper may be qualitatively different from a citation indicating its use or application. In other words, citation of a paper because of some feature of its contents is indicative of a different kind of value than a citation which indicates the use of its results.
7. Martin Cave et al., The Use of Performance Indicators in Higher Education: The Challenge of the Quality Movement, 3rd ed., Higher Education Policy Series, no. 34 (London: Jessica Kingsley, 1997), 182.
8. Larry Hardesty and Gail Oltmanns, "How Many Psychology Journals Are Enough? A Study of the Use of Psychology Journals by Undergraduates," Serials Librarian 16, no. 1/2 (1989), 150. Their data indicated that:
DePauw owned several journals heavily cited by Indiana students. DePauw students, however, did not cite these journals.
9. Eugene Garfield, "What Citations tell us about Canadian Research," Canadian Journal of Information & Library Science 18 (Dec. 1993), 27.
10. Kelland and Young, "Citation," 93-4. This is related to the "Matthew Effect" --- summarized in Stephen J. Bensman, "Journal Collection Management as a Cumulative Advantage Process," College & Research Libraries 46 (Jan. 1985), 18.
11. Cf. Michael E. Koenig, "Citation Analysis for the Arts and Humanities as a Collection Management Tool," Collection Management 2 (Fall 1978), 247-9; Robert J. Greene, "Computer Analysis of Local Citation Information in Collection Management," Collection Management 17 (1993), 18; Wanda V. Dole and Sherry S. Chang, "Survey and Analysis of Demand for Journals at the State University of New York at Stony Brook," Library Acquisitions: Practice & Theory 20 (Spring 1996), 31-2; etc.
12. Anne L. Buchanan and Jean-Pierre V. M. Herubel, "Comparing Materials Used in Philosophy and Political Science Dissertations: A Technical Note," Behavioral & Social Sciences Librarian 12 (1993), 64. Similarly Anne L. Buchanan and Jean-Pierre V. M. Herubel, "Profiling PhD Dissertation Bibliographies: Serials and Collection Development in Political Science," Behavioral & Social Sciences Librarian 13 (1994), 9-10; John Cullars, "Characteristics of the Monographic Literature of British and American Literary Studies," College & Research Libraries 46 (Nov. 1985), 512; John Budd, "A Citation Study of American Literature: Implications for Collection Management," Collection Management 8, no. 2 (Summer 1986), 50; Marifran Bustion and Jane Treadwell, "Reported Relative Value of Journals versus Use: A Comparison," College & Research Libraries 51 (Mar. 1990), 143. Arguments for citation analysis to determine academic journal management are summarized and advocated in John C. Calhoun, "Serials Citation and Holdings Correlation," Library Resources & Technical Services 39 (1995), 53-5.
13. E.g., Cullars, "Characteristics... British & American Literary Studies"; also Buchanan and Herubel, "Profiling PhD Dissertation Bibliographies." See also Jean-Pierre V. M. Herubel, "Philosophy Dissertation Bibliographies and Citations in Serials Evaluation," Serials Librarian 20, no. 2/3 (1991), 65-73; Joy Thomas, "Graduate Student Use of Journals: A Bibliometric Study of Psychology Theses," Behavioral & Social Sciences Librarian 12, no. 1 (1993), 1-7.
14. Cf. John Cullars, "Characteristics of the Monographic Scholarship of Foreign Literary Studies by Native Speakers of English," College & Research Libraries 49 (Mar. 1988), 158-9.
15. Cf., Cullars, "Characteristics... British & American Literary Studies," 513.
16. Budd, "Citation Study," 60.
17. Greene, "Computer Analysis," 19-20; Likewise Koenig, "Citation Analysis," 247ff; Broadus, "Proposed Method," 33; Theresa Dombrowski, "Journal Evaluation Using Journal Citation Reports as a Collection Development Tool," Collection Management 10, no. 3/4 (1988), 178. Local studies are also encouraged by Herubel, "Philosophy Dissertation Bibliographies," 68; Thomas, "Graduate Student Use," 4; Buchanan and Herubel, "Profiling PhD Dissertation Bibliographies," 10; etc.
18. Maurice B. Line, "Use of Citation Data for Periodicals Control in Libraries: A Response to Broadus," College & Research Libraries 46 (Jan. 1985), 36-7. See also the negative literature reviewed in Calhoun, "Serials Citation," 54.
19. Dombrowski, "Journal Evaluation," 175. Similarly Bensman, "Journal Collection Management," 24-5; also Sylvia and Marcella Lesher, "What Journals Do Psychology Graduate Students Need? A Citation Analysis of Thesis References," College & Research Libraries 56 (Jul. 1995), 317.
20. Stephen E. Wiberley, Jr., "Journal Rankings from Citation Studies: A Comparison of National and Local Data from Social Work," Library Quarterly 52, no. 4 (1982), 358.
21. Robin D. Devin, "Who's Using What?" Library Acquisitions: Practice & Theory 13, no. 2 (1989), 167-70; Robin D. Devin and Martha Kellogg, "The Serial/Monograph Ratio in Research Libraries: Budgeting in the Light of Citation Studies," College & Research Libraries 51 (Jan. 1990), 46-54; Michael Bowman, "Format Citation Patterns and Their Implications for Collection Development in Research Libraries," Collection Building 11, no. 1 (1990), 2-8, etc.
22. Cf. Devin and Kellogg, "Serial/Monograph Ratio," 52 versus Bowman, "Format Citation Patterns," 3, 6.
23. Bowman, "Format Citation Patterns," 7. Similarly, Sylvia and Lesher, "What Journals," 314.
24. Dombrowski, "Journal Evaluation," 179.
25. As shown by Wiberley, "Journal Rankings," 355:
When one examines comparisons of the 30 most-cited titles, a clear, although often small, difference appears between the success in prediction by national and local sources. The 30 journals most cited locally, 1971-74, are, with one exception, more successful in predicting the 30 journals most cited locally than are national sources.
26. Richard Griscom, "Periodical Use in a University Music Library: A Citation Study of Theses and Dissertations Submitted to the Indiana University School of Music from 1975-1980," Serials Librarian 7 (Spring 1983), 38. Similarly Thomas, "Graduate Student Use," 3; also Sylvia and Lesher, "What Journals," 314.
27. Wiberley, "Journal Rankings," 358.
28. Dole and Chang, "Survey," 33.
29. Broadus, "Proposed Method," 32-3; Bowman, "Format Citation Patterns," 3, 7.
30. Swigger and Wilkes, "Use of Citation Data," 52. See also the next quote, by Broadus.
31. Broadus, "Proposed Method," 33.
32. Griscom, "Periodical Use," 36. Again, note that fuzzy concept of "use." Fortunately, he later observes (p. 37):
The primary fault of use studies lies in disagreement among researchers over what exactly constitutes library use. Is use circulation? Is use pulling books off shelves?
For other criticism of reshelving studies see Broadus, "Proposed Method," 31; Robert N. Broadus, "On Citations, Uses, and Informed Guesswork: A Response to Line." College & Research Libraries 46 (Jan. 1985), 38-9; Bustion and Treadwell, "Reported Relative Value," 150; etc.
33. Swigger and Wilkes, "Use of Citation Data," 42.
34. Wiberley, "Journal Rankings," 348. See also John M. Budd, "The Literature of Academic Libraries: An Analysis," College & Research Libraries 52 (May 1991), 290.
35. See the comments of Line summarized in Bensman, "Journal Collection Management," 23, and questioned by Kelland and Young, "Citation," 85.
36. Summarized in Bensman, "Journal Collection Management," 23-4.