March 22, 2001
This
paper presents the results of an online survey on how ICQ users use the
application, and attempts to discover some interesting trends and possible
implications for the usability of the software.
ICQ
is one of the first instant messaging application to appear on the Internet,
and to date is one of the most popular instant messaging applications in use,
with an estimated 87 million users world-wide. ICQ was acquired by American
Online (AOL) a few years ago, although the company released its own AOL Instant
Messenger instead of using the existing ICQ software. The software has gone
through several major revisions in the past, and has been ported from MS
Windows to other platforms such as the Macintosh, Linux and Java. Our team (Gek
Low, Di Yin Lu, Adam Yarnold and Steven Lurie) decided to conduct a
multi-pronged study to find out what it is that made ICQ so popular, and to
reveal usability problems with the software. This paper presents the findings
from a Web-based survey conducted as part of the overall study. The objective
of the survey was to understand a major group of ICQ users, namely college students,
and to get some general feedback on the various features from them. The survey
revealed some interesting results that warrants further study and also
highlighted a few possible usability problems with the software.
A
total of 24 people responded to the survey, of which 19 were college students.
The
survey was set up as a basic HTML form on a Stanford student Leland Unix
account, using the web server’s built-in support for conducting form-based
surveys. All the scales used in the survey were 7-point scales. All submissions
were logged automatically by the web server. The team members got their friends
to fill in the survey, but in order to get a wider variety of responses, I also
posted to the Stanford su.market newsgroup asking for volunteers. The survey
was anonymous, unless the respondents chose to give us their name and contact
for possible further study. This fact was emphasized when asking for
respondents, and also stated clearly on the survey form. The survey was run for
a week and the results were then processed manually.
A
Stanford Leland account for data-logging, MS Excel for analysis of data.
Why do people use
ICQ?
The
objective was to get a quick initial sampling to the users’ motivations for
using the software. The respondents presented a wide range of answers to this
free-form question. Here are just some of the responses.
·
“It’s
convenient, and it’s relaxing chatting in real-time”
·
“Easy,
non-demanding way to keep in touch”
·
“To
keep in touch with friends”
·
“It
is cheaper than talking on the phone”
·
“Because
my friends are on it”
·
“Friends
who use it rather than AIM”
Many
users thus see ICQ as a useful and convenient alternative to traditional means
of keeping in touch with friends. Then there are the few who use ICQ simply
because they had no choice – their friends are using it. ICQ is popular, but
some people prefer to use other instant messaging software. It might be that
ICQ is not as user-friendly as other instant messengers. This question was not
addressed in our study, but a comparison of the various instant messengers out
there with ICQ would be a very appropriate follow-up study.
|
Age group |
Frequency |
Percentage |
|
Gender |
Frequency |
Percentage |
|
19
to 23 |
13 |
54.2% |
|
Male |
14 |
60.9% |
|
24
to 28 |
11 |
45.8% |
|
Female |
9 |
39.1% |
13
of the 24 respondents (54.2%) are between 19 and 23 years of age, while the
rest are 24 to 28 years old. We did not get any other age groups, which is
expected given that most of the respondents are college students.
There
were a lot more male respondents than female respondents, but it is unwise to
make any general conclusions at this stage. The small number of respondents,
coupled with how we got people to fill in the survey (newsgroup users and ICQ
users may not correlate), makes this statistic meaningless on its own.
|
Ethnicity |
Frequency |
Percentage |
|
Major/Profession |
Frequency |
Percentage |
|
White |
13 |
54.2% |
|
Engineering/CS |
11 |
45.8% |
|
Black |
1 |
4.2% |
|
Physics/Chemistry |
3 |
12.5% |
|
Asian/Pacific |
7 |
29.2% |
|
Other
majors |
5 |
20.8% |
|
Native
American |
3 |
12.5% |
|
Non-student |
5 |
20.8% |
About
half of respondents were white, and a third were Asian/Pacific, which was
pretty much what we expected. As there was only one black respondent to the
survey, the number is not representative of the percentage of black ICQ users.
As mentioned above, we did not have the time to get a good sampling of users.
As
for the students’ major, slightly less than half of respondents were from the
engineering and computer science majors, but there were also a variety of other
majors. Still, we can’t conclude that the majority of ICQ college users are
from engineering or computer science, because of the way the survey was
conducted.
It
was not easy to get useful demographic data for the project, since we did not
have the resources to be able to reach a large number of people in a short
time. This does not mean that the demographic data is useless though. It is
important in that it provides a context for our analysis of the data.
|
Internet connection |
|
|
ICQ version |
|
|
|
|
Ethernet |
13 |
54.2% |
|
ICQ
ver 97 |
2 |
8.3% |
|
T1/T3 |
10 |
41.7% |
|
ICQ
ver 98 |
7 |
29.2% |
|
56K |
1 |
4.2% |
|
ICQ
ver 99b |
1 |
4.2% |
|
|
|
|
|
ICQ
ver 2000 |
14 |
58.3% |
All
the respondents use MS Windows, and only one has a modem connection, which is
surprising. Again, maybe this is the result of the sampling process. I would
have wished for a wider variety of users, but anyway the common platform makes
the analysis easier and makes the conclusions stronger, although they can’t be
applied to users on other platforms.
Not
all of our ICQ users use the latest 2000 version. A large percentage of users
still use the older 98 version. This suggests maybe a reluctance to upgrade to
the latest version, or maybe that the new features are not compelling enough to
justify the upgrade. This might also point to possible difficulties in
downloading the software. Steven’s heuristic evaluation of the ICQ download
site shows that the web site has a lot of usability problems and this may put
off potential upgraders.
|
Gender vs ICQ version |
|
|
Gender vs experience |
|
||
|
|
male |
female |
|
|
male |
female |
|
ver97 |
0 |
2 |
|
1
to 3 months |
2 |
0 |
|
ver98 |
2 |
4 |
|
3
months to 1 year |
1 |
2 |
|
ver99b |
1 |
0 |
|
1
to 2 years |
2 |
2 |
|
ver2000 |
11 |
3 |
|
more
than 2 years |
9 |
5 |
Even
more interesting is when we breakdown the ICQ version used by gender. Males
have a much greater tendency than females in choosing the latest version. It
may be due to the interface, the design of the download page or just simply a
tendency for males to always get the most updated software. It should also be noted
that the males tend to have more experience than the females. More study is
needed before any conclusions can be drawn.
|
ICQ experience |
|
|
|
ICQ usage frequency |
|
|
|
1
to 3 months |
2 |
8.3% |
|
Everyday |
17 |
70.8% |
|
3
months to 1 year |
3 |
12.5% |
|
3
or 4 times a wk |
3 |
12.5% |
|
1
to 2 years |
4 |
16.7% |
|
Once
a week |
1 |
4.2% |
|
more
than 2 years |
15 |
62.5% |
|
Once
a month |
3 |
12.5% |
|
Experience vs version |
|
|
|
Usage frequency vs version |
|
|||
|
|
ver97 |
ver98 |
ver2000 |
|
|
ver97 |
ver98 |
ver2000 |
|
1
to 3 months |
0 |
0 |
2 |
|
Everyday |
0 |
4 |
12 |
|
3
months to 1 year |
2 |
0 |
1 |
|
3
or 4 times a wk |
0 |
1 |
2 |
|
1
to 2 years |
0 |
4 |
0 |
|
Once
a week |
0 |
1 |
0 |
|
more
than 2 years |
0 |
3 |
11 |
|
Once
a month |
2 |
1 |
0 |
About
87% of our users have at least 1 year of experience with ICQ, and 70.8% use the
software everyday. Correlating these with the versions they use, we can see
that the more experienced and frequent users tend to use the latest version,
which is not surprising.
|
Experience vs reported expertise |
|
Major vs reported expertise |
|
|
Gender vs reported expertise |
|||||
|
|
Mean |
S.D. |
|
|
Mean |
S.D. |
|
|
Mean |
S.D, |
|
1
to 3 months |
4.50 |
0.71 |
|
Engineering/CS |
5.91 |
1.22 |
|
male |
6.00 |
1.30 |
|
3
months to 1 year |
4.67 |
2.52 |
|
Physics/Chemistry |
5.00 |
2.00 |
|
female |
4.89 |
1.62 |
|
1
to 2 years |
4.25 |
1.26 |
|
Other
majors |
4.80 |
2.59 |
|
|
|
|
|
more
than 2 years |
6.00 |
1.46 |
|
|
|
|
|
|
|
|
The
mean expertise as self-reported by the users is 5.42 (+/- 1.64). Even when
broken down by experience, users tend to rate themselves as above average. In
fact, only 3 users gave themselves a rating of less than 4 on the 7-point
scale. Males tend to give themselves higher expertise ratings, as well as users
majoring in engineering and computer science. The other studies conducted by
the team showed that there were many usability problems with the software, but
then again, the users in this survey are mostly experienced users hence the
higher reported expertise levels.
|
Connect to ICQ whenever online |
|
Maintains a webfront |
|
|
||
|
Yes |
16 |
66.7% |
|
Yes |
3 |
12.5% |
|
No |
8 |
33.3% |
|
No |
19 |
79.2% |
|
|
|
|
|
What
is a webfront? |
2 |
8.3% |
Most
users connect to ICQ automatically whenever they are online. This is the
default behavior of ICQ, but there is also an option to disable this feature.
There is a fairly large percentage of users who prefer to disable this
automatic connection, so it appears that the method to disable it is quite
well-known.
Most
users also know what an ICQ webfront is. Only 2 users do not know what it is.
This is somewhat surprising given the general lack of usability of the ICQ web
page. However, most users do not have an ICQ webfront. Either the feature is
not useful, or is too troublesome to use.
|
Uses other Instant Messengers |
|
|
|
Yes |
15 |
62.5% |
|
No |
9 |
37.5% |
|
|
|
|
|
AOL
users |
10 |
|
|
Yahoo
users |
6 |
|
|
MSN
users |
4 |
|
The
majority of the users use other instant messaging applications in addition to
ICQ, of which the most popular alternative is the AOL Instant Messenger (AIM).
It is not known if AIM is more popular than ICQ, but the fact that many ICQ
users also use AIM suggests a comparison of the features and usability is made
between the two programs.
|
Most frequent ICQ status |
|
|
|
Online |
8 |
33.3% |
|
Free
for chat |
1 |
4.2% |
|
Disconnect |
2 |
8.3% |
|
Away |
5 |
20.8% |
|
Invisible |
4 |
16.7% |
|
Occupied |
1 |
4.2% |
|
Not
Available |
3 |
12.5% |
|
Experience vs most frequent ICQ status |
|
Usage frequency vs most frequent status |
||||
|
|
Active |
Inactive |
|
|
Active |
Inactive |
|
1
to 3 months |
2 |
0 |
|
Everyday |
5 |
12 |
|
3
months to 1 year |
0 |
3 |
|
3
or 4 times a wk |
3 |
0 |
|
1
to 2 years |
2 |
2 |
|
Once
a week |
0 |
1 |
|
More
than 2 years |
5 |
10 |
|
Once
a month |
1 |
2 |
* Active statuses are “online” and
“chat”. The rest are “inactive”.
ICQ
has a user-selectable status, which is seen by the people on the user’s contact
list. The most frequent status is “online” which is the default. “Free for
chat” doesn’t seem to be a popular setting. However, most people prefer to use
the “inactive” status settings, which is interesting because instant messaging
software is supposed to help people interact. If we breakdown the users by
experience and usage frequency, we find that the more experienced and frequent
users of the software prefer the “inactive” settings. Maybe the more experienced
users simply do not want to be disturbed by other people easily. This sets up
an interesting situation where users want to contact other people but do not
want to be contacted easily themselves.
|
Number of contacts |
|
|
Number of active contacts |
|
||
|
1 to
5 |
8 |
33.3% |
|
1
to 5 |
15 |
62.5% |
|
6
to 10 |
3 |
12.5% |
|
6
to 10 |
6 |
25.0% |
|
11
to 16 |
5 |
20.8% |
|
11
to 16 |
1 |
4.2% |
|
17
or more |
8 |
33.3% |
|
17
or more |
2 |
8.3% |
The
number of people on the users’ contact lists are pretty well spread out, with
the majority of users having either 1 to 5 or more than 17 people on their
lists. However, the number of active contacts at any one time is usually just 1
to 5. Correlating this result with the previous one on ICQ statuses, it is no
surprise because most users prefer the “inactive” status settings. It would
seem that the way users might use ICQ is to use it like the way some people use
answering machines – send a message first, and if it’s important and the
receiver is actually physically there at the computer, he or she will reply
right away. Otherwise, you wait. This is also the way some people use email. It
would seem that instant messaging is basically just a faster version of email.
If this is true, then an ethnographic study of many ICQ users may be very
interesting to conduct.
|
Experience vs number of contacts |
|
|
||
|
|
1 to 5 |
6 to 10 |
11 to 16 |
17 or more |
|
1
to 3 months |
2 |
0 |
0 |
0 |
|
3
months to 1 year |
3 |
0 |
0 |
0 |
|
1
to 2 years |
2 |
1 |
1 |
0 |
|
more
than 2 years |
1 |
2 |
4 |
8 |
|
|
|
|
|
|
|
Experience vs number of active contacts |
|
|
||
|
|
1 to 5 |
6 to 10 |
11 to 16 |
17 or more |
|
1
to 3 months |
2 |
0 |
0 |
0 |
|
3
months to 1 year |
3 |
0 |
0 |
0 |
|
1
to 2 years |
4 |
0 |
0 |
0 |
|
More
than 2 years |
6 |
5 |
2 |
2 |
Breaking
the users down by experience again, we see that the more experienced users tend
to have more contacts on their lists, as well as active contacts. Users with
less experience tend to have most of their contacts active, but not the
experienced users. This matches well with the previous findings.
In
the survey, the users were given the choice of saying that they have never used
a certain feature, or that they do not know what the feature is. This was
intended to for users who do not have the feature in the version of ICQ they
use. However, it turns out that users overwhelming chose “never used” over
“don’t know”, even when the ICQ they use do not have the feature. This either
points to confusion over what features are actually available in the
application, or confusion over the survey questions. As the reason of not
clear, I analyzed only the positive responses.
|
Features usage |
Number of users with a usage rating of 3
or more |
|
Message |
23 |
|
File
transfer |
13 |
|
Email
messages |
6 |
|
URL
transfer |
7 |
|
Chat |
17 |
|
ICQ
phone |
2 |
|
SMS
messages |
3 |
|
ICQ
greeting card |
2 |
|
Organizer
features |
2 |
Usage
frequencies were rated on a scale of 1 to 7, with 1 being least frequent and 7
being most frequent. Counting only users with a rating of 3 or more, the most
popular features (in order) are messaging, chat and file transfer, which are
the core features of instant messaging software. The least popular features,
which are the phone, SMS messaging, greeting card and organizer features,
appears to be fairly redundant. Only SMS messaging is a feature found only on
the latest version of ICQ.
|
Features usage |
Mean |
SD |
|
Message |
5.74 |
1.32 |
|
File
transfer |
4.12 |
1.90 |
|
Email
messages |
2.31 |
1.97 |
|
URL
transfer |
3.29 |
2.09 |
|
Chat |
4.14 |
1.93 |
|
ICQ
phone |
2.29 |
2.21 |
|
SMS
messages |
4.75 |
2.50 |
|
ICQ
greeting card |
1.67 |
1.12 |
|
Organizer
features |
2.17 |
1.83 |
If
we look at the mean usage frequency ratings as given by the users, we see again
messaging, chat and file transfer as being the most frequently used features,
and the same redundant features falling far behind. The only exception to this
is the SMS messaging. Most of the users who use it tend to use it frequently,
so the SMS messaging feature is not redundant after all. Either the users do
not have a compelling need to use the non-core features, or the features may be
hard to locate or hard to use. Follow-up studies should be carried out.
|
Features liking |
Mean |
SD |
|
Message |
5.83 |
0.94 |
|
File
transfer |
6.11 |
1.08 |
|
Email
messages |
4.07 |
1.21 |
|
URL
transfer |
4.56 |
1.90 |
|
Chat |
5.00 |
1.52 |
|
ICQ
phone |
4.40 |
1.17 |
|
SMS
messages |
5.29 |
1.25 |
|
ICQ
greeting card |
4.27 |
0.65 |
|
Organizer
features |
4.56 |
1.01 |
As
for how much our users like the features (provided that they actually use
them), all the features are surprisingly likeable on average. Messages and file
transfer top the list, but other than those, there is no “bad” feature that can
be identified. Steven’s heuristic evaluation of ICQ 2000 revealed a lot of
usability problems, as did Di Yin’s observational study. Apparently, once users
have learnt how to use the features, they become used to them and grew to like
them or feel nonchalant about them. It looks like the users’ needs outweigh the
usability of the features.
There
are some users who gave low ratings to certain features. File transfer, email
message, and chat received several low ratings. Future studies can focus on
these features to find out if there are usability problems in these areas.
The
survey raises some interesting questions for further study. Ideally we would
have liked the other studies done by the team to follow-up on the results of
this survey, but due to lack of time, this was not done and so all the various
studies were done concurrently. Possible candidates for further research
include observing users as they use ICQ in their environment, to see how
frequently they perform different tasks, and how they respond to incoming
messages, while going about their work. As one user put it, ICQ is an “easy,
non-demanding way to keep in touch”. It would be interesting to see just how
easy and non-demanding it is. The other studies our team conducted addressed
the problem of ease-of-use, but they do not address the way users use the
software.
More
extensive surveys or interviews can also be carried out to answer some of the
other questions raised by this survey, such as the reason for the apparent
disparity between versions used by the male and female ICQ users, why certain
features are not popular, what are their main gripes with the various features,
etc. Usability tests cannot answer these questions, because they look at only a
small part of the overall picture.
The survey could have been designed better. I could have simply asked “why?” in many places (such as “why is this your most frequent ICQ status mode?”). However, there is often a tradeoff between how much you want to find out and the length of the survey. Longer surveys may result in the respondents not being serious enough when answering the questions.
The
data collection process could also be improved. Asking friends to fill in the
survey and looking in newsgroups for volunteers is not really a good method,
although it did get us a fairly large number of responses within a short period
of time. Perhaps even better would be to use paper surveys instead of
electronic ones and to go out into the streets or into the offices to look for
volunteers. However, that would require a lot more time and effort in data collection.
One
problem with the design of the survey is when I ask if the user has never used
the feature before or if he doesn’t know what the feature is. It might have
been confusing to people, judging by the number of “never used” responses when
the feature doesn’t even exist in that version of ICQ. The way the question was
phrased was changed in response to initial pilot testing of the survey, but
evidently problems still exist.
A. Survey
B. Raw data
C. Tabulated data