IMPROVING THE ACCURACY OF
BIODATA QUESTIONNAIRES. Frank
Kuschnereit and Barry Spinner. University
of New Brunswick.
Biographical information (Biodata) obtained from job applicant
forms has been in use since the turn of this century to predict future
job success. It has been only relatively recently that there has been
a concerted effort to improve Biodata's reliability and validity. Two
of the major concerns are how to minimize the incidence of faking
and how to reduce inaccurate responses due to social desirability.
Theoretical and empirical work have identified several factors which
might affect the accuracy of Biodata responses. Using a 2x2
(between) x2x2 (within) design, this study examined the effects of
(1) the consequences of inaccurate responses (warning); the amount
of detail requested in responses (Detail); (3) the relevance of the item
to the job (Relevancy); and (4) whether or not the information could
be independently verified (Verifiability). Participants were
volunteers registered in Introductory Psychology. It was
hypothesized that providing a warning of consequences and
requiring greater detail would reduce social desirability of responses,
but only for items that were job relevant and potentially verifiable.
Results are discussed with reference to the characteristics of
desirable Biodata items and the format in which they are presented.
Implications for future research are also discussed.
Introduction
There are many costs associated with hiring personnel who
subsequently are not successful. These costs can be measured not only
in dollar terms but also by their effect on the lives and careers of the
persons involved. Many organizations expend a great deal of time and
effort to match the right person to the job. There are a variety of
personnel selection techniques in common use. The interview is one
of the most popular but has been criticized as being too subjective.
The assessment centre approach has the applicant complete a battery
of selection tests but can be very time consuming and expensive. One
technique that is nonsubjective, inexpensive and effective involves
obtaining biographical information (Biodata) from job applicants to
predict future job success. While resembling the standard job
application form, a Biodata questionnaire seeks to obtain more
extensive and detailed information on significant events in applicant's
lives and especially on their past behaviours. The basis of Biodata's
predictive abilities is the theory that past behaviour is the best
predictor of future behaviour. Biodata cannot predict all future
behaviours but it can give an indication of the types of probable
behaviours based on an individual's prior learning history (Mumford
& Owens, 1987). Nichols (1994) stated that "Biodata measures may
predict performance across so many aspects of behaviour as well as
they do because responses to Biodata items may serve to capture
previous manifestations of the constructs and mechanisms that
ultimately determine predictive relationships with criteria."
While all job application forms collect some form of biographical
information and have been in use since the turn of this century, it has
been only relatively recently that there has been a concerted effort to
improve Biodata's reliability and validity. Two of the major concerns
are how to minimize the incidence of faking and how to reduce
inaccurate responses due to social desirability.
One recent approach to addressing both of these concerns
employs differing the instructions given to the applicants. Kluger and
Colella (1993) studied the effects of faking Biodata test items by
randomly warning 214 of 429 applicants for a nurse's assistant position
against faking. While the warning mitigated the propensity to fake, the
specific warning effects depended on item transparency. For
transparent items, warning reduced the extremeness of item means and
increased item variances. For nontransparent items, warning did not
have an effect on item means and reduced item variances. These faking
effects were best predicted when transparency was operationalized in
terms of item-specific job desirability in addition to the item-general
social desirability.
This study examined the instructions given to the role-play
applicants to see if a forceful caution could reduce the incidence of
faking good. It also examined the hypothesis that asking for additional
details in response to multiple-choice type items would increase the
accuracy of responses. In addition, the effects of warning conditions
and amount of supporting detail requested on faking behaviours were
examined in a simulated job application situation using a two by two
between and two by two within factorial design as shown below.
There were two levels of warning (minimum and maximum warnings).
In the maximum warning groups, the subjects were explicitly cautioned
that some of their responses will be verified and that any false
statements would be grounds for dismissal.
Each item was rated as to the degree it could be perceived as
being job related (JR) and verifiable (V). It was expected that the
means and correlations would be different for items that are highly job
related (HJR) and highly verifiable (HV) as opposed to items that are
low (LJR & LV). Within the instrument there was also a "fake"
(verifiable) item to further assess faking based on a study by Pannone
(1984).
Method
DESIGN
2 X 2 Between
|
Short Version
|
Extended Version
|
| Minimum Warning
|
56
|
53
|
| Maximum Warning
|
54
|
55
|
2 X 2 Within
Items
|
High Job
Relevance |
Low Job Relevance
|
| Highly Verifiable
|
6
|
6
|
| Slightly Verifiable
|
6
|
6
|
The subjects were 258 Psychology 1000 students. These students
volunteered to participate in return for receiving credit towards their
course participation marks. The participants were asked to role play
as a job applicant responding to a fictional job advertisement.
MATERIALS:
Once the nature of the study was described to the participants and
they agreed to take part, each was given a package of materials that
included:
A) A fictional job advertisement, indicating the nature of the
job and the desired job applicant qualifications.
A note that indicated they had made an initial application for
the job, and had been invited by the company to complete
the rest of the application process by answering a Biodata
form.
The Biodata form, which included 97 items tapping into
several aspects of the job applicant's past. Some of the
items (High Relevancy) were selected as they related back to
desirable characteristics identified in the job advertisement.
Others were selected to be less relevant to the job.
Similarly, some of the items were selected to be easily
verifiable (High Verifiability) while others were not easily
verified (Low Verifiability). Item relevancy and verifiability
were confirmed by having 48 participants (not in the current
study) rate them on each of these dimensions. In addition,
as a check on faking, we included a single item which asked
participants if they had experience with a (nonexistent)
computer software package whose name was fabricated for
this study.
Results
Initial Analysis
Univariate outliers (Z>3.0) were examined and adjusted. No
multi variate outliers were found. Although, there was no indication
from the literature to expect a difference due to sex, an initial analysis
was conducted using a series of ANOVAs using sex, warning condition
and version as independent variables. All tests were non significant
and it was decided to combine the sexes for the main analysis. The
individual test items were converted to Z scores and composite means
were computed for the four question types - High Verifiability/ High
Job Relevance, High Verifiability / Low Job Relevance, Low
Verifiability / High Job Relevance and Low Verifiability/Low Job
Relevance.
Main Analysis
The SPSS MANOVA procedure was used to examine the 2 X 2
between-subject factors of version type (extended or short version) and
warning condition (maximum or minimum warning). It also examined
the 2 X 2 within design of item verifiability by job relevance. None of
the effects in the design were significant at an alpha of .05.
A separate analysis was conducted for the single item designed to
check faking. While over 20% of the participants indicated they had
experience with this software, there were no differences as a function
of either between-subject factor (warning and extended version).
Discussion
None of the factors in this design (item verifiability, item-job relevance,
the consequences of misreporting, and requesting extended details behind the
answers to some items) had a systematic effect on Biodata responses. While
it is possible that these conditions have no effect in real-life situations,
previous literature suggests that these features should have an effect on
responses.
The most likely reason for the absence of significant effects is the nature
of the sample and the setting used for the study. It is difficult to determine
whether the results would have been different if the participants had been
older and more experienced as job seekers. Although there is some indication
in the literature that university students can effectively role play as job
seekers it makes intuitive sense that first year students are not ideal.
The setting for the role playing should be as realistic as possible.
Participants in this study completed the materials in a large group. In
retrospect, a smaller classroom could have been set up to better represent a
business location rather than have large groups of participants complete the
instrument in an amphitheater.
The level of warning failed to have an effect on the results. This could
have been an artifact of the warning not being meaningful to the participants.
The same could be said for the failure of Relevancy and Verifiability to be
significant. The effectiveness of these manipulations relied upon participants
"getting into their role", and taking the warnings seriously.
One possible explanation for the lack of differences between conditions
is that participants generally responded honestly to all items (therefore
conditions designed to reduce faking would have no effect). While this may
have happened to some extent (i.e., participants were not motivated enough),
it is unlikely that all answers were honest and accurate since over 20% of the
participants reported having some experience with a computer program called
Intrude 4.0. This was a check item and there is no such program in existence.
Clearly the responses must be some form of error but unfortunately the
manipulations in the study had no effect on this item as there were no
significant differences between the groups.
Unfortunately the absence of significant results prevents any
conclusions about the factors investigated in this study. Future research
should be orientated towards the factors investigating these factors in a more
realistic setting and ideally with participants who are actually applying for
a position which they want to obtain.
Go to References