CS377A Assignment 4
Comparing the Efficiency of Various Microsoft Word
Help Features
Gek Siong Low
Abstract
This
is a follow-up study on a previous self-report study on surveying the usage
patterns of Microsoft Word users in using its help features. An experimental
study was carried out to determine the efficiency of the various help features
in Microsoft Word and the results presented here.
Introduction
In
my earlier study, I surveyed Microsoft Word users on how they use the help
features, and their perceptions of them. It was found that the Office Assistant
was perceived as more effective than the help index and help contents on
average as a source of help. In this study, I attempted to measure the time
spent on using the various help features in order to perform certain tasks, and
find out if any general conclusions can be drawn from the results.
Method
Participants
4
people from the class participated in the study. They are Joe Sacco, Di Yin Lu,
Mark Chavira and Nina Kim.
Procedures
The
same PC was used for all the experiments. Prior to the session, the Internet
Explorer history is cleared, because the help system in Microsoft Word 2000
uses the same HTML engine, and I do not want the users to get clues on where to
go from the link colors.
A
series of 4 tasks were presented to the participants. For each task, they were
asked to use a specific help feature to find out how to achieve the task. They
were timed on how long they took to find what they thought was the correct
solution and to perform the task. Three help features were used in the study,
namely the Office Assistant, the help index and the help contents. To prevent
the learning effect from using the same help feature repeatedly, the help
features that a participant were to use for each task were rotated between
participants. Also, only one help feature was used for each task-participant
pair, since it would not make sense to repeat the task with another help
feature once the task is learned.
Because
I didn’t want to give the participants clues on what to look or search for, I
basically presented the tasks as “make the page look like this” instead of
explicitly saying for example, “add shading to the text”.
Materials
Microsoft
Word 2000 running on a certain Windows 2000 PC in the PC/Mac cluster in the
Gates basement.
Results and
Discussion
There
wasn’t enough data to perform reasonable statistical analysis. There were wide
variances in what little data was gathered. The time taken to consult a
particular help depends quite a lot on that the user’s ontology matches that of
the software, especially for the help index. For example, Joe spent a much
longer time using the help index than Nina in performing the first task because
he didn’t immediately thought of the word “indent”.
The
most problematic task appeared to be the text borders and shading. Almost
everybody went for text boxes or graphically drawing a rectangle around the
text, which failed miserably because they are more complicated to do, and the
help features they used went very useful. Only Joe got the correct solution
using the help contents, and even then, he found the solution under the
“Graphics and Drawing” branch, so he probably had the same initial idea as the
rest. Nina actually didn’t made the connection even though she was looking at
the “Borders” help page. This mismatch in conceptual models made the task
especially confusing and frustrating.
The
easiest task was inserting a picture from a file. The concept is
straightforward and clear.
A
look at the analysis of central tendency would appear to imply that the help
index is worst and the Office Assistant is actually the best help feature.
While it would be nice to corroborate this finding with the results from the
earlier survey, I want to emphasize that there is simply not enough data to
make any sensible conclusion here. Every help feature has at least one scenario
where a participant spent a lot of time going down the wrong path. The help
index appears to be a big culprit in doing this, probably because of the large
number if possible results presented to the user. The help contents, while
presenting a structured view of things, takes time to navigate and read. The
Office Assistant presents less number of search results and also more relevant
ones too, which may explain its slightly higher efficiency. However, it too
suffers from the same problem as the help index, in that if the user has a
different conceptual model, then a lot of time will be wasted.
As
interesting as these theories may be, there is no way to conclude anything
definite from the data from this experiment. I would suggest doing a more
complete study with greater number of users, enough to actually separate the
users who have the same conceptual model as the application, and those who
don’t, so that the efficiencies can be compared between these two scenarios as
well.
Critique
This
is a poorly designed experiment, but it would have made a great observational
study, given the interesting things that each participant did. I didn’t my
participants to have so many problems with the tasks. The tasks were intended
to be something a user would possibly want to do but are not everyday tasks
that everybody already know how to do. I was quite surprised that not everybody
knows about paragraph indentation, although I probably shouldn’t be because of
another MS Word study I did while taking CS147.
Maybe
getting more participants will show a bell curve distribution, or maybe not. In
any case, more participants are definitely needed if any form of statistical
analysis is to be made. This is probably what’s so difficult about experimental
methods. A lot of investment is needed before any useful and repeatable
conclusions can be drawn.
I’ll
probably need to redesign the tasks to make them less prone to such a wide
variety of approaches, although it can also be argued that constraining users
to “obvious” tasks would skew the results and be much less generalizable.
Hence, this is a question of just exactly what I’m trying to test here. It
seems to me that experimental methods involve a tradeoff between validity and
generalizability. Sometimes it may be hard or impossible to test for what you
really want to know, and you have go for something more contrived and specific
but less generalizable to the real-world situation.
Appendix
Summary and Analysis
Participant/Tasks Chart
Participant\Task
|
1 |
2 |
3 |
4 |
|
1. Joe Sacco |
A
|
B |
C |
A |
|
2. Di Yin Lu |
B
|
C |
A |
B |
|
3. Mark Chavira |
C |
A |
B |
C |
|
4. Nina Kim |
A
|
B |
C |
A |
A – Help Index
B – Help Contents
C – Office Assistant
Analysis: Task vs Help Feature
Efficiency
Task\Help
Feature
|
Help Index |
Help Contents |
Office Assistant |
Mean |
|
1. Indenting
paragraphs |
7
min 43 s
2 min 48 s |
1 min 27 s |
1 min 36 s |
3 min 24 s |
|
2. Borders and
shading |
4
min 30 s
|
1 min 40 s 3 min 31 s |
4 min 14 s |
3 min 29 s |
|
3. Insert picture
from file |
0 min 23 s |
0 min 49 s |
0 min 54 s 0 min 49 s |
0 min 44 s |
|
4. Send picture
behind text |
1 min 36 s
0 min 51 s |
4 min 56 s |
2 min 02 s |
2 min 21 s |
Mean
|
2 min 59 s
|
2 min 29 s |
1 min 55 s |
|
Results
of experiment
Name of participant: Joe Sacco Date/Time: 2/20/2001 2:30pm
Task 1: Indenting
paragraphs
Help method used: Help Index / Help
Contents / Office Assistant
Time spent on looking up help and performing task : 7 min 43 s Outcome:
Success / Failure /
Partial
Notes: Looked up “space”
>> “Adjust line/paragraph spacing” >> Format menu
Looked up
“justify” >> “Tabs” >> “Indent paragraphs” >> “Hanging
indents”
A lot of time was
spent on wondering what to look for. Terminology is a big problem here.
Task 2: Borders and
shading
Help method used: Help
Index / Help Contents
/ Office Assistant
Time spent on looking up help and performing task : 1 min 40 s Outcome:
Success / Failure /
Partial
Notes: Graphics and Drawings
>> Borders >> Add or change or fill
Task 3: Inserting
picture from file
Help method used: Help
Index / Help Contents / Office
Assistant
Time spent on looking up help and performing task : 0 min 54 s Outcome:
Success / Failure /
Partial
Notes: Asked about “Insert
picture”. No problems here.
Task 4: Send picture
behind text
Help method used: Help Index / Help
Contents / Office Assistant
Time spent on looking up help and performing task : 1 min 36 s Outcome:
Success / Failure /
Partial
Notes: Looked up “background”
>> “Add a background” >> “Use a picture as a background”
Decided he has no
idea what to look for.
Results
of experiment
Name of participant: Di Yin Lu Date/Time: 2/20/2001 4:30pm
Task 1: Indenting
paragraphs
Help method used: Help
Index / Help Contents
/ Office Assistant
Time spent on looking up help and performing task : 1 min 27 s Outcome:
Success / Failure /
Partial
Notes: Formatting >>
Formatting paragraphs >> Vertical alignment of text, then to “Set tab
stops”
Task 2: Borders and
shading
Help method used: Help
Index / Help Contents / Office
Assistant
Time spent on looking up help and performing task : 4 min 14 s Outcome:
Success / Failure /
Partial
Notes: Asked about “textbox”.
Looked around for the draw toolbar, couldn’t find it. Drew a text box later.
Asked about “bring
to front”. Eventually gave up.
Task 3: Inserting
picture from file
Help method used: Help Index / Help
Contents / Office Assistant
Time spent on looking up help and performing task : 0 min 23 s Outcome:
Success / Failure /
Partial
Notes: Looked up “insert”
>> “Scan picture and insert”.
Task 4: Send picture
behind text
Help method used: Help
Index / Help Contents
/ Office Assistant
Time spent on looking up help and performing task : 4 min 56 s Outcome: Success / Failure / Partial
Notes: Working with Graphics >> Enhancing/Resizing >> Create transparent areas
>>
Edit picture
>>
Resize/Crop
>>
Aligning
Eventually gave
up.
Results
of experiment
Name of participant: Mark Chavira Date/Time: 2/21/2001 2:15pm
Task 1: Indenting
paragraphs
Help method used: Help
Index / Help Contents / Office
Assistant
Time spent on looking up help and performing task : 1 min 36 s Outcome:
Success / Failure /
Partial
Notes: Asked “How do I indent?”
>> Create a first
line indent
>>
Create a hanging indent
Task 2: Borders and
shading
Help method used: Help Index / Help
Contents / Office Assistant
Time spent on looking up help and performing task : 4 min 30 s Outcome:
Success / Failure /
Partial
Notes: Looked up “color”
>> “Change the color of text and numbers”, but couldn’t find toolbar on
screen.
Looked up
“background color” >> “Add a background”
Looked up
“rectangle” >> “Add a rectangle”
Added the drawing
toolbar. Drew rectangle. Now looked up “transparency”.
Eventually gave
up, but did the task with text boxes without looking up help.
Task 3: Inserting
picture from file
Help method used: Help
Index / Help Contents
/ Office Assistant
Time spent on looking up help and performing task : 0 min 49 s Outcome:
Success / Failure /
Partial
Notes: Working with Graphics
>> Inserting Graphics >> Insert a picture
Took a while to
read down the listing.
Task 4: Send picture
behind text
Help method used: Help
Index / Help Contents / Office
Assistant
Time spent on looking up help and performing task : 2 min 02 s Outcome:
Success / Failure /
Partial
Notes: Asked about “send to
background”, “move back”, “graphics back”, “graphics ordering”
>> Ways to
position text and graphics >> Layering text and graphics >> Change
the text-wrapping style
Results
of experiment
Name of participant: Nina Kim Date/Time: 2/21/2001 5:30pm
Task 1: Indenting
paragraphs
Help method used: Help Index / Help Contents
/ Office Assistant
Time spent on looking up help and performing task : 2 min 48 s Outcome:
Success / Failure /
Partial
Notes: Looked up “indent” >> Create a negative indent
>>
Increase left indent of entire paragraph
>>
Examples of paragraph
>>
Hanging indent
Moved entire
indent for different lines. Didn’t notice that there are two indent controls.
Task 2: Borders and
shading
Help method used: Help
Index / Help Contents
/ Office Assistant
Time spent on looking up help and performing task : 3 min 31 s Outcome:
Success / Failure / Partial
Notes: Working with Graphics
>> Borders, Fills, Shading (didn’t seem to see the solution)
Also looked at
“Inserting graphics”, “Insert WordArt”
Went back to
“Borders”, still didn’t see it.
Added drawing
toolbar, drew a textbox, filled it with color and then delete it.
Now drew a
textbox, cut and paste into it, and then fill it with color.
Task 3: Inserting
picture from file
Help method used: Help
Index / Help Contents / Office
Assistant
Time spent on looking up help and performing task : 0 min 49 s Outcome:
Success / Failure /
Partial
Notes: Asked about “insert a
picture” >> Insert a picture
Task 4: Send picture
behind text
Help method used: Help Index / Help
Contents / Office Assistant
Time spent on looking up help and performing task : 0 min 51s Outcome: Success / Failure /
Partial
Notes: Looked up “picture”
>> Change text wrapping style
Task 1: Make the paragraph look like this
Before:
The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog.
After:
The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog.
Task 2: Make the paragraph look
like this
Before:
The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog.
After:
The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog.
Task 3: Add a picture file from the hard disk
Before:
The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog.
After:
The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog.
Task 4: Arrange the picture like this
Before:
The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog.
After:
The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog. The quick brown fox jumped over the lazy dog.