[Contents Page] [Previous section] [Next section] Back to my homepage
Sections
7.1 Tally sheets
7.2 Tabulation : explanation and key
7.3 Working list of journals
7.4 Reliability and validity
For each thesis the bibliography was photocopied. The copy was labeled and used as a tally sheet for counting the citations. Possibly troublesome items were highlighted &/or annotated; e.g. ---
Uncatalogued, unpublished, primary sources or other "raw" data were not counted or tabulated. Conversely, cited works not listed in the bibliography were added to the tally sheet, when encountered, and their citations also counted.
The results were tabulated and the following details of each cited work noted.
This section is a discussion of the tabulated results, column by column. In any case of conflict between a thesis bibliography and a record in the University's library system, NOTIS, the bibliography is considered to be wrong because all non-temporary NOTIS records have been verified (at some stage). All the tabulated categories below are primarily determined by the NOTIS record, if it exists; whenever, on the balance of probability, the tally sheet appeared erroneous, it was corrected.
Column Codes & Explanation
Th Thesis number --- e.g., Ed01.
Ref Reference number. There are two sequences:
n number of times the item was cited. As discussed above (§2.3) a primary source is artificially given a count of "1." Often it was not obvious which reference in the bibliography (if any) the author intended; in such cases, I chose the most probable reference.
Date Preferably the actual date of the item. E.g., the date of the reprint, not the original copyright date - however, for microform reproductions of documents first published after 1945, the original date was used; microform reprints of older documents have no date assigned (the field was left blank). If no date could be reliably ascertained, the field was also left blank. Items which were putatively "in press" or "in publication" were assigned a date of 1996.
Format
C. Conceptual format. Where there was no NOTIS record, the reference was taken at face value provided it was sufficiently complete and consistent. The following categories were used. They are mutually exclusive.[103]
1? Primary source? An explicitly primary source was indicated as such in the thesis' bibliography, abstract or methodology. An implicit primary source was either mentioned in the abstract or methodology chapter in a manner implying that it was an ideologically crucial basis for the thesis' methodology; or was numerically significant [see category-n]. Each of a thesis's primary sources has an artificial count of "1" when counting citations for that particular thesis. The following, mutually exclusive categories were used.
P. Physical format. The following, mutually exclusive categories were applied.
Held?
# Status-Code.[109] The following, mutually exclusive categories were used. For later analyses, codes 9 through 6 are treated as held items, and codes 5 through 1 are treated as unheld items.
NOTIS the NOTIS call-number of the item, if there is a relevant record. All items held are now recorded on NOTIS, except for maps in the Geography Library.
vol Usually the volume number of a serial.
Title An abbreviation of the title; only initial articles
are omitted from the beginning of the title. For incompletely cited items
omitted from the bibliography, a page-number is given where the "reference"
may be found in the thesis, e.g., Milton \p.20
NB: the following abbreviations are always employed:
Lan Principle language of the work, if not English.[113] Note that many works have bilingual titles, but are actually in English. The codes used are
When a journal title was first encountered on a tally-sheet, its holdings were established. The title, holdings & NOTIS number were pasted to a separate document (which was later sorted alphabetically). This procedure saved time in any subsequent encounters.
Reliability suggests stability and consistency of measurement --- accuracy and replication. ...
Validity is often viewed in two contexts --- external or internal. The former suggests the extent to which the data are generalizable to a large setting, while the latter examines the extent to which the data measure what they purport to measure.[114]
This citation count is not only be replicatable, it is exactly repeatable. That is, an other researcher could perform his or her own analysis on exactly the same, identified material. On the other hand, had I instead performed a shelving study, say, the next researcher could only hope to approximately reproduce such a study by mimicking it, perhaps one academic year later when the materials, users and conditions are near to those of the first study. There is no way for a dubious, subsequent researcher to go back in time and perform an analysis on the very same usage which the original exercise claimed to be studying.
Of course this citation count is also merely replicatable, particularly if one wishes to ascertain whether the results are consistent, perhaps,
The accuracy was [minimally] tested by repeating the count for one thesis (Ed04) six days later, with a fresh tally sheet.[115] Only three items had a differing count:
Internal validity was supported by having a consistent policy as to what a citation was and what to count (see Appendix-1). There is no assurance of external validity. It is assumed that
A) The citation distribution for each subject's theses is a reflection of the overall contemporary citation patterns in that subject, at this University.
B) The citation patterns for English and History may be usefully extrapolated for other humanities subjects.
C) The citation patterns for Anthropology, Education, Political Studies, and Psychology may be usefully extrapolated for other social science subjects.
Assumption-B is probably the weakest of the three; it could be shored up by including data from other humanities theses into the Humanities distribution, even though there are insufficient theses in those subject to generate acceptable subject distributions. Assumption-A is essential to the relevance of this project.
There are other validity problems for citation counts:
The validity of use of citation analysis to assess use of journals is open to some reservations. ... Whether a citation of an article constitutes use of that article is a function of the honesty of the author. It is possible that some authors cite articles that they have never read. A second reservation is that, when a faculty member [or that faculty member's graduate student] cites an article in a journal, one doesn't know whether she or he acquired a reprint or preprint copy, personally subscribed to that journal, read a colleague's copy, or used some other library's copy.[116]
However, in consolation, by selecting all theses in each of the six subjects,[117] there should be little discrepancy between the population of 1996 theses in a given subject and a sample drawn from that population, since (at 1st March) the sample was the population.
103. This choice of categories was influenced by Buchanan and Herubel, "Profiling PhD Dissertation Bibliographies," 5; and, to a lesser extent, Budd, "Citation Study," 52.
104. includes monographic works by multiple, corporate, or anonymous authors; but excludes monographic works which are primarily edited (those are category-c).
105. After the decision of Cullars, "Characteristics of Foreign Literary Studies," 161-2:
This study treats a reference to an article cited in a collection of essays as a monographic reference because the author cited it from a book.
106. The publication rate, research orientation, and content all help to distinguish category-e from category-d --- albeit with several gray areas.
107. At the time, the exact details were recorded separately, so that any significant, new category could be introduced. Only category-i was introduced, after initially being considered for the formal project proposal & rejected because several Education theses were examined and none apparently cited ERIC documents.
108. This applies only to references in the bibliography which were nominated as "in press," "in preparation," or some such. Unpublished, uncatalogued drafts are "raw" data and are not counted or tabulated. Theses were often termed "unpublished" in the bibliographies but these have been classified as category-p (if not already held and known to be category-q).
109. NB: if an item is not listed in the NOTIS catalogue, it is assumed to be unavailable, even if an electronic version is accessible from the library (e.g., through Reuters Online, NEWZTEXT, or other Internet resources).
110. When a serial appears to cease at a given date, but is known to have a current subscription under a different, subsequent name and NOTIS number, it is still considered to be code-9, not code-8.
111. The number of a newspaper, magazine, or similar serial is not recorded. Any "volume number" of a multi-part monograph is noted as part of the title (if relevant).
112. This omission is not necessarily a mistake, since some serials use only the date rather than a volume number.
113. Again, this information is preferably taken from NOTIS, although it is possibly erroneous for older or shorter records, or for multi-part items covered by only one record. If an item is multilingual, the non-English language most relevant to the thesis has been given.
114. Peter Hernon, Statistics: A Component of the Research Process, Rev. ed., (Norwood, NJ: Ablex, 1994), 65-66.
115. This thesis was chosen because it was not too large, and I had a cold the first time it was counted, so mistakes would not be unexpected.
116. Swigger and Wilkes, "Use of Citation Data," 44.
117. That is to say, all excluding embargoed theses and those catalogued after 1st March 1997.