Paper presented at the Fourteenth International Conference on Testing Computer Software


Testing CDE In Sixty Languages: One Test Is All It Takes

Harry Robinson
Sankar Chakrabarti

Abstract

The Common Desktop Environment (CDE) product is internationalized. This means that CDE and the applications delivered with it must work properly in both English and non-English language environments without any changes in the application binaries. The problem is how to verify efficiently that the CDE product as delivered really functions properly in the variety of locales it claims to support. Our organization was confronted with the task of verifying CDE in 60 different locales. Traditional record-and-playback methods would not port well across locales. We needed an automated test method where the cost did not become prohibitive as the number of supported locales grew. This paper describes the development of a locale-smart testing procedure to solve this problem.

Testing Internationalized Applications

Internationalized applications employ two main principles [1]:

Internationalized applications are a boon to development groups, but testing these applications can become a nightmare for the test team. Simply put, since an application behaves slightly differently in each supported locale, the application should be tested in each of those locales. Doing this testing well and efficiently takes some effort.

Message Catalog Approach

We propose that tests become internationalized by using the information contained in the locale-specific message catalogs. The test assertions should be composed in terms of the information in the message catalog as often as practical. This adds no extra burden since the message catalogs are an integral part of any system supporting internationalized software. The test and its target application "feed off" the same message catalogs. Such an internationalized test will operate in any locale.

Rather than expressing the test assertions in terms of hard-coded values, a test assertion for an internationalized test program could be expressed in locale-independent terms as follows:

If routine X executes properly, it will output the string indexed by (set-Id Y and message-Id Z) contained in the message catalog specific for the locale of execution.

It is important to note what we are doing here. We are asserting that the invariant in this relationship is the location of the string in the message catalog. We suggest that this invariant be used in deriving the test assertion as often it is practical. This practice would result in more robust and locale-independent tests. 

Incorporating the message catalog locations into the test program might seem to tie the test program too closely to the application's implementation. On the contrary, we believe it is the least of several evils. It is true that the correspondence between the application and the test program will break if the application changes how it uses the message catalogs; it is far more likely, however, that the contents of the message catalogs themselves will change. Our program is immune to message catalog content changes because we do not concern ourselves with the contents, merely with the location in the catalog.


Testing CDE in Sixty Languages . . .

Construction of a locale-independent CDE test suite

This paper describes a fully automated system to touch test eight CDE applications in all available locales. When we started the project, extensive testing of each CDE application was already being performed in the C locale. Our modest goal in this project was to adapt touch tests to verify that the main CDE applications could successfully come up and close in each locale. These eight applications were: File Manager, Style Manager, Text Editor, Calendar Manager, Mailer, Window Manager, Application Manager, and Dtterm. We started with an existing touch test suite designed to execute in the C locale. We modified the test suite to operate reliably in all 60 locales supported by CDE.

For each locale available on the test system, the test program performs the following actions:

Implementing even these modest tests posed significant challenges in the internationalized arena.

Set the desktop locale

The first hurdle in testing CDE in multiple locales is developing a method of setting locales automatically.

The user or test program logging in to CDE chooses the desired locale from language menus under the "Option" button of the login screen (see Figure 1). To select a locale, one must first know where the desired locale appears on the menu. Finding the desired locale is difficult for a test program because CDE constructs the language menus dynamically, based on how many locales are available on the system. The same locale can appear in different menu locations on different systems.

For example, Figures 1 and 2 show the login screens for two test machines. Figure 1 has two fewer locales installed than Figure 2. Note that es_ES.iso88591 (Spanish) locale appears in Figure 1 at the bottom of menu 1; in Figure 2, the same locale appears near the top of menu 2.

Figure 1: Login screen (Note locale es_ES.iso88591 at the bottom of submenu 1)

 

Test programs equipped with only static representations of the language menus cannot execute properly in these dynamic situations. The test must query the target environment at run-time to determine where to find the desired locale.

Our test tool, Synlib, [2] provides a structure known as a focusmap that allows us to navigate hierarchical menu systems by specifying menu locations. Once Synlib is given a selection's location, it computes the keystrokes needed to reach the targeted menu item.

.

Figure 2: Login screen (Note locale es_ES.iso88591 at the top of submenu 2)

 

However, since the structure of the menus and the position of each locale can only be determined at run-time, a static focusmap may be good for one system but may not be good for another system. The test program therefore uses a generic focusmap (Figure 3) for the language menus. The generic focusmap does not associate a locale name with any particular menu or position.

The executing test program queries the target machine with "locale -a" to determine how many locales are present on the machine. With this number, the test program calculates how many menus and submenus will appear on the login screen. By sorting the output of "locale -a", the test program also determines which menu and which item correspond to the target locale. For instance, if we want to run our tests in de_DE.iso88591, the above procedure might determine that there are 57 locales available on the target machine and that de_DE.iso88591 appears as Item 12 on Submenu 1. Our call to the focusmap utility would then specify that we wished to select the focus item:

Options.Options_menu.Language.Lang_submenu.menu1.menu1.12

Synlib then selects the appropriate item for us.

Figure 3: Generic Focusmap for the Option Menu and Submenus

 

This generic approach works well. The test program does not need to be told which locales or how many locales are present on the target system to be tested. Rather it is instructed to scan the system and determine what the system's language menu will look like. We can either choose a particular locale in which to test or, by feeding a loop construct with the output of "locale -a", we can cycle through all the available locales on a machine without knowing ahead of time what locales might be available. The latter technique was used in our automated test system.

The reader may recognize that the variable menu structure presented by the login screen is not strictly an effect of internationalization of CDE. Rather, the structure is determined by the number of locales configured on the target system. Nevertheless, setting the locale is a significant hurdle which had to be overcome to build a fully automated touch test suite for CDE in all locales.


. . . One Test Is All It Takes

Open designated applications

Once the test program has logged in to the desired locale, we would like to open the designated CDE applications. To do this, we click on an item in the Front Panel and watch for the application window to appear. This section of the paper deals with verifying that the target application window has opened. The general question we are answering is this: in an internationalized application, how do we verify that the application has behaved correctly?

In CDE, all applications have distinct titles. In our touch test, we open the application and verify that the application has the expected title. The test program should be constructed to retrieve the expected title string from the appropriate message catalog files. If an application window does not appear or displays an incorrect title, then the application has a bug. The example below illustrates the idea.

Here is a hard-coded version of a verification routine:

                /* Wait for the window to map */
                SynWaitWindowMap( dpy, "Editor", HOWLONG );

                /* Fetch the title from the window */
                SynGetWindowTitle( dpy, "Editor", &title_ptr);

                /* Verify the window title */ 
               if ( strstr(title_ptr, "Text Editor") != NULL )
                       isCorrectWindow = True;
                else 
                        isCorrectWindow = False;

The first level of verification, SynWaitWindowMap(), determines whether the application window actually appeared. The test reads the title of the target window using SynGetWindowTitle(). The string returned by this routine is then compared with the expected string. The problem is that the title strings are dependent on the locale of execution of the application (Table 1). The only way to make one test verify the behavior of the application in all its supported locales is to construct the test such that it can seek out the appropriate message catalog and then figure out the title string specified for that locale using the set-index and message-index for the desired string.

 

Table 1: Text Editor Title in Different Locales

 

Note that we are proposing here that the location of the window title string in the Text Editor message catalog be used as the test invariant, rather than the string itself. Once we know that the title is located in message set 7, message number 2 of the dtpad.cat catalog, we can re-code our test program as follows:

        	/* Open the message catalog */
                my_cat=catopen("/usr/dt/lib/nls/msg/%L/dtpad.cat",NL_CAT_LOCALE);

	        /* Fetch the expected title string from the message catalog */
                expected_title=catgets(my_cat,7,2,"Text Editor" );

	        /* Close the message catalog */
	        catclose(my_cat);

        	/* Wait for the window to map */
	        SynWaitWindowMap( dpy, "Editor", HOWLONG );

	        /* Fetch the title from the window */
                SynGetWindowTitle( dpy, "Editor", &title_ptr);

	        /* Verify the window title */
	        if ( strstr(title_ptr, expected_title) != NULL )
		        isCorrectWindow = True;
	        else
                        isCorrectWindow = False;

This verification will now work in all supported locales. In fact, because the test uses the message catalog locations, its results are immune to changes in the strings themselves.

Close each application

In this section of the paper, we demonstrate locale-dependent input by closing the application using its menu mnemonics. Locale-dependent inputs are those in which an application's concept of what is a valid input depends on the locale of execution. An input which is considered valid in one local can be wrong or meaningless in another locale. The example below involving menu mnemonics of the CDE Text Editor File menu illustrates the point. The internationalized code fragment shows how message catalogs can be used effectively to determine the valid inputs in a specified locale and how using this knowledge we can easily write locale-independent tests.

Take the example of the Text Editor's File menu. Figure 4 shows the File menu in the English and French locales. To close the application window in the English locale, the test program (or a human user) presses "C" for "Close". Here is the hard-coded version in Synlib:

	/* Select the "Close" action */
        SynClickKey( dpy, "C" );

The same input however will fail to close the application in the French locale; in French, the appropriate input to close the application is "F" for "Fermer". To verify the Close behavior, therefore, the test should press "F" and not "C''. How can the test provide the valid input in a given situation? Either all the possibilities are to be coded in the test -- which becomes increasingly complex with increasing number of locales; or the test may be constructed to consult the locale-specific message catalog to determine which is the valid input for the situation. It is our opinion that this latter approach is the cleanest and least expensive alternative.

Figure 4: File Menu in English (left) and French (right)

The illustration below shows how this test can be written in a locale-independent way using the information in the message catalogs. In the C locale, the message catalog contains the following:

$set 11
29 C
30 Close

In the fr_FR.iso88591 (French) locale, the dtpad.cat message catalog contains

$set 11
29 F
30 Fermer
 

Table 2: Mnemonic for the Close Action in Different Locales

As shown in Table 2, the location of the mnemonic is invariant across all locales. The test program can use this invariance to adapt itself to various locales.

        /* Open the message catalog */
        my_cat=catopen("/usr/dt/lib/nls/msg/%L/dtpad.cat",NL_CAT_LOCALE);

	/* Get the mnemonic for the Close action */
        sprintf(file_close,   "<%s>", catgets(my_cat, 11, 29, "C") );

	/* Close the message catalog */
        catclose(my_cat);

	/* Select the "Close" action */
        SynClickKey( dpy, file_close );

 

Results

Only one set of tests is all it took to verify aspects of CDE applications in all 60 supported locales. The set of tests is fully automated and executes unattended. The test can be targeted to any system supporting CDE; it is not necessary to know which locales are available on the system.

The test suite in its current state is fairly simple. It merely sets the locale and then opens and closes the main CDE applications. In spite of its simplicity, this touch test suite uncovered a large number of defects of various levels of severity:

 

These defects were uncovered by our test suite even after the CDE system was thoroughly tested and debugged using other test suites.


Conclusions

To internationalize a test we need to follow the same principles employed by the internationalized applications; i.e. the tests must fight fire with fire. If the applications use locale-smart functions - the tests must do likewise. If environmental variables determine an application's appearance and behavior, then tests must also use the same environmental variables to predict the expected properties of the application and to verify such expectations. The tests should be attuned to all the environmental variables, such as LANG, that are used by the application. Only then tests will be robust enough and portable across locales.

A major suggestion of this work is that tests should use the structure of the message catalogs to compose test assertions. Without this approach there is very little hope of creating locale-independent tests and reducing the cost of testing target software. We propose that tests adopt the use of message catalogs in forming assertions as often as it is practical. The touch tests presented here for individual CDE applications were very simple. In most cases, the test consisted only of verifying that when launched in the locale of interest, the application started properly and displayed its window with the appropriate title. This test suite did not explore extensive functional testing of the applications concerned. It has become apparent to us, however, that the experience gained from this exercise can be easily adapted to create fully operational locale-independent functional tests suites for more thorough verification of the CDE applications.


Acknowledgments

The authors would like to acknowledge Mohammed Rafi whose touch test program formed the basis of the test suite described in this paper.


References

[1] Sankar L. Chakrabarti and Sunil Girdhar: Testing Internationalized GUI Applications. The X Resource, Issue 17, pp.211-227, Winter Issue, 1996

[2] Sankar L. Chakrabarti: Testing X-Clients Using Synlib and FocusMaps. The X Resource, Issue 13, pp. 7-21, Winter Issue, 1995


Harry Robinson  is currently a software test engineer with Microsoft's Intelligent Search Group

Sankar Chakrabarti is an R&D engineer with Hewlett Packard


Return to Harry Robinson's software testing web page

1