Text By Robert A. Gates
Assistive Technology Specialist
Comments By SuperAdaptoid
Comments by SuperAdaptoid:
The tests are reasonable, the data is measurable and informative. However the study is set to an "ideal environment", assuming all things being equal, and all OCRs functioning at 100%. In a real- world setting the environment is always less than "ideal".
For instance, OCR performance can vary greatly with computer processer power. Newer OCR software is being written to accomodate, and take advantage of, ever higher processers. Even among older 386 OCR software performance improves greatly on 486 machines.
The study does not address the varible factors OCR Dependent user preferences. Individual preference among blind and low vision OCR Dependent users is likeky to include the critical human-factors of personal [Hands-On] experience and technical "Training Familiarity". We tend to like, [and want to keep], what we know best, as the "Tried and True". Even when presented with hard facts and superior alternatives. The hidden human factors are:
Personal learning curves
The learning level of effort required
Loss of functionality during re-training.
Security of things known and familiar
Insecurity of things unknown and unfamiliar
"Life is hard enough without change".
Regardless of these observations, the study does provide an "benchmark" for bench test performance among leading OCR software options. Just remember speed and accuracy can be relative to the power of the processer. Ease of use can be relative to the user's ability, disability, experience, expectations, and general level of frustration.
A clearly printed letter in one typeface and no columns.
A clearly printed page in four typefaces including two columns. The typeface fonts were Caslon and Helvetica. The actual fonts sizes were 9 point and 24 point.
A page of newsprint, with multiple fonts, pitches, columns and layout
A spreadsheet page
A magazine page with multiple fonts, pitches, columns and layout
Poor quality fax with headings, different font sizes and columns
Poor quality photocopy with headings, different font sizes and columns
The Labs looked at six aspects of each package tested:
Speed
Accuracy of OCR
General formatting
Overall ease of use
Ease of installation
Special features.
All products were timed on each scan to generate comparative results. No major changes were made to default settings. The only changes made were for original documents that had columns and documents that were degraded. These alterations were only made if the option was available on the product. No fine tuning was done and the OCR process was done through the Automatic OCR feature of the products.
General usability and ease of installation were also considered. Finally, any special features that the products claimed to have were investigated. Accuracy was tested by looking at two documents that were used in the test. The first one we looked at was a standard letter and the second one was the clearly printed text page in different typefaces and font sizes. We assessed accuracy by counting the percentage of characters that were successfully recognised.
General formatting applied to the ability of the OCR package to handle complex page layouts and multiple fonts; the closer that the scanned image was to the original, the better the package's formatting abilities were. The products were installed on a Apricot XEN 486 DX2-66 PC, with 12MB of RAM and a 540MB hard drive running Windows 95. The virtual memory setting was automatically managed by Windows 95. The scanner used with this test was a Hewlett Packard 4c connected to the PC via a SCSI controller card. The SCSI adapter was a Symbios Logic 53C400A (HP Version) supplied by Hewlett Packard with the scanner.
Assessment of Test Results
Speed:
The overall fastest engine came with CharacterEyes. This was identified by looking at the combined speed of scanning the different documents and then dividing the figure by the number of documents used. As can be seen from the graph below, Character Eyes has an engine speed of just over one minute.
TypeReader, EasyReader and Recognita all had very similar speeds with WordScan performing very badly. The total difference from the worst product and the best product was over 5 minutes, showing that CharacterEyes has in fact a very fast recognition engine. However, this should by no means be the acid test for assessing speed of the products. Looking at the ability of the products to acquire different documents individually gives a totally different picture. First of all when looking at the software's capability to acquire magazine and newspaper text, we found that TypeReader was the only product to do it within sixty seconds, as shown below.
The next test to look at is the software's ability to acquire degraded documents. It was found overall that the engine speed of most products declined dramatically in this test, as it took much longer to recognise the scanned image. Only CharacterEyes was able to carry out the OCR process within sixty seconds. It was also found, with the exception of Recognita, that the products also took longer to acquire fax documents than normal photocopies, as can be seen below.
In the final text page speed test TypeReader came out the fastest with ChracterEyes coming close behind. In all the speed performance tests WordScan Plus performed very badly. It can be concluded from this analysis that TypeReader has the fastest engine when it is presented with good quality original prints. But can it reproduce the original documents in it's original layout?
There is nothing worse than sitting with an acquired document and recorrecting all the mistakes the software made. So, it is not sufficient to look at speed alone. Although speed may be a factor, OCR software is supposed to save money and the time necessary to retype a document. Therefore accuracy during the OCR process is very important as mistakes can often take longer to find and rectify. In order to make OCR effective the vendors should be aiming for near 100% accuracy.
Accuracy:
Basically two documents were looked at when assessing the accuracy of each of the products. These documents were: a clearly printed letter in one typeface and no columns (180 characters) and a clearly printed page in four typefaces including two columns (4287 characters). The typeface fonts were Caslon and Helvetica. The actual fonts sizes were 9 point and 24 point. The accuracy rate was determined by looking at the number of characters per page and the software's ability to accurately reproduce these characters in it's original form as a percentage. The results are displayed below.
Nearly all the OCR products were capable of accurately reproducing the letter document, producing a 99% accuracy rate. The only exception was EasyReader Elite which had 25 character mistakes and an accuracy rate of 86.41%. TextBridge Pro, WordScan Plus and TypeReader Pro reproduced the original without any mistakes.
This test was however a very simple one for the packages, therefore we put the packages through a further test. This time we took the printed page with columns, different typefaces and font sizes to find out how good the packages really were at reproducing the original with minimal mistakes. ChracterEyes performed very badly in this test with a total number of character mistakes coming to 356, or 91.69% accuracy. Also performing badly was EasyReader Elite producing 184 character mistakes, or 95.71% accuracy. Performing exceptionally well was WordScan Plus with a mere 18 mistakes and a accuracy rate of 99.58%. In conclusion to this test it is safe to say that accuracy rates below 98.8% or greater than 50 mistakes per page, can mean time consuming finding and correcting of errors. From the results we can see that WordScan Plus, TextBridge Pro and TypeReader Pro both offer near perfect accuracy rates with Recognita Plus being acceptable.
Formatting:
Finally the Labs thought it was important to look at the formatting capabilities of each of the OCR products, especially for people who are looking for an identical copy of the original. In this test the Labs compared the general format of the OCR copy with the original to determine whether or not the basic formats were kept. Important aspects to consider are whether columns, font sizes, paragraph settings and the original text are maintained. All the documents included in the speed test were also used in this section. Not one of the packages was able to reproduce the spreadsheet into Excel with any meaning to it.
The ability of TextBridge Pro to maintain the scanned image in it's original form was not excellent. Original one page documents were reproduced over two pages, text was not wrapped around images well and some text was totally misplaced. On the plus side it did manage to maintain font sizes and in some cases font typeface.
Recognita Plus was able to maintain the original document on one page with good reproduction of font size, typeface and columns. It captured the layout of degraded documents well and maintained text in it's original spots. The downside was that titles of the magazine and newspaper pages were half omitted, and the layout of the simple typed letter was not perfect. Although TypeReader Pro was unable to maintain the reproduced document on one page, it does manage to reproduce all the text including sub-headings. Font sizes and typefaces were reproduced perfectly and columns were clearly separated. making the computer document look very similar to the original. OmniPage Pro was the only product that managed to display inserted pictures and incorporate it within the text document. With the exception of half recognised sub-headings it did manage to maintain the original format of magazine page and the typed letter very well. On degraded copies the software was able to pick up text on dark background well - although this was picked up as an image. The newspaper page was reproduced very badly with some sub-heading un-readable.
CharaterEyes Pro was very poor at maintaining even basic formatting like margins. Although the layout of the typed letter was fine, it did miss out some characters on the left margin because of a very narrow setting. The Labs were unable to make any sense of the photocopy page, the fax page, the magazine page or the newspaper cutting.
WordScan Plus as we have identified produced very accurate results in terms of text reproduction. Unfortunately as with it's poor speed performance it is also let down by very poor formatting. Even though the option of decolumnizing was not selected, the automatic OCR process was unable to maintain column settings. Columnar originals appeared as a one column text document. Even the simple typed letter was unable to be maintained in it's original form with extra spaces inserted in between paragraphs, a four line address placed on two lines and normal text displayed in bold. Finally, with EasyReader formatting is non-existent. It was unable to maintain column settings and basically displayed the scanned image as one big text file. The only document it was able to maintain near perfect formatting was the typed letter as it had no columns.
On each of the above tests the products were given a rating between one and ten, with ten being excellent. This enabled the Labs to get a better picture of the product that stood out the best. The rating of each product is shown below.
TypeReader
Speed 9
Accuracy 10
Formatting 9
TextBridge
Speed 7
Accuracy 10
Formatting 8
CharacterEyes
Speed 9
Accuracy 6
Formatting 6
EasyReader
Speed 8
Accuracy 7
Formatting 3
OmniPage
Speed 7
Accuracy 8
Formatting 8
WordScan Plus
Speed 3
Accuracy 10
Formatting 2
Recognita
Speed 7
Accuracy 9
Formatting 8
So it can be seen that TypeReader emerges as the overall top performer in
the labs test, with TextBridge, Recognita and OmniPage also doing well. A
point to note is that fine tuning on certain products may have improved its
formatting. However in the Labs test no fine tuning was done as we wanted
to see how well the software could reproduce documents with minimal user
interaction and also keep the tests to a certain standard.
Text Provided By:
Robert A. Gates
Assistive Technology Specialist
RGates@Goodnet.com
Arizona Center for the Blind and Visually Impaired
3100 E Roosevelt
Phoenix, AZ 85008
602-267-0453 voice
602-273-7410 fax