Vol. 4. No. 3 M-2 May 2000

Return to Table of Contents Return to Main Page

Alexis, the Encomium TOEIC Test Preparation System

According to the official TOEIC web site, http://www.toeic.com:

The TOEIC (Test of English for International Communication) Test is an English language proficiency test for people whose native language is not English. It is the world standard for the assessment of English used in the global workplace, with 1.5 million tests administered each year (TOEIC.com, 2000).

According to Gilford (1996), this test came about because of requests from the Japanese Ministry of International Trade and Industry to the Educational Testing Service (ETS) in the mid-1970's. It measures the listening comprehension and reading skills of individuals whose native language is not English. Businesses worldwide use TOEIC scores for evaluating employees.

Given the growing popularity of the TOEIC, the growth in the number of test-preparation courses and software packages was inevitable. According to my own quick research, a Google (http://www.google.com) search of the words "TOEIC preparation" returns 2,080 entries. The Alexis System is just one among hundreds, apparently, trying to address the need for test preparation.

The Alexis System

The Alexis system is a browser-based program that allows users to prepare for the TOEIC using 10 Picture Modules, 5 Question-Response Modules, 15 Conversation Modules, 6 Short Talk exercises, 15 Reading Modules, 13 Error Recognition Modules, and 4 Reading Activities. These activities mirror the item-types found in the TOEIC. We will look first at the browser delivery system, then at each of these activity types.

This CD-ROM comprises web pages and Shockwave/Director files (interactive animation that is delivered by a free browser plug-in), delivered by a standard browser interface. This is an advantage for two reasons: it requires no software installation on the user's machine, and it uses software the user is likely to have already.

There are, however, problems with the implementation. First, a Macromedia Shockwave plug-in is needed to run the program. This plug-in is not available on the CD-ROM; instead, the user must go to Macromedia's site to download it. (The Windows version is 633K, and the Mac version, 829K as of May 1, 2000.)

Second, although the program can be run off-line, there are a few links that require an Internet connection. Neither the CD-ROM jewel case nor the program itself indicates that the user will need to be online to access certain links, nor, more importantly, are Internet links marked as such from the pages of the program. This is likely to confuse some offline users who will click on one link and wait. A simple parenthetical statement saying, for example, "requires Internet connection" would easily have solved this problem.

One minor problem with the Internet links is that one points to ETS's home page (http://www.ets.org) as a source of information about the TOEIC. In fact, the user will find little information about the TOEIC on this site. The TOEIC is administered by the Chauncey Group, a subsidiary of ETS, thus the TOEIC website is found at http://www.toeic.com.

Finally, the web design of the program itself is inelegant. The home page is an overly long list of contents that is hard to navigate. A better navigation would have taken advantage of a menu system or a frames architecture to limit the amount of text on a single page. Navigation between the modules requires using the "back" button on the browser to the main menu and selecting the next module. Similarly, links that state merely "click here" make the web design look amateurish.

The listening modules consist of a picture and a set of four choices describing the picture. We see the first item from Picture Module 1.1 below, along with the text for the correct answer given.

Picture Module Example 1. Click on the picture to see a larger image.

The user is given the option of seeing a hint, seeing the written script, and getting substantive feedback on all answers. These reflect good instructional design. However, the navigation is confusing. When hovering over the photograph, a "pointing finger" appears, which would normally indicate that there is a link available. There isn't. The hint button stays illuminated once it is used, giving no indication that it can be used over again for multiple hints. Finally, after finishing a set of items, the user can review the questions, but unfortunately cannot do so selectively. For example, I wanted to look at Question 7 again, but had to not only page through Questions 1-6, but also answer each of them correctly a second (or third) in order to return to #7. This will be extremely frustrating for users who are doing well, and only need or want to review the occasional item.

More troubling is the content. For the same item presented above, the choices are:

1. The computer is broken.
2. The man is welding.
3. The man is wealthy.
4. He is taking a break.

The feedback for item C says: "No, wealthy means RICH, but this is clearly a workman."

I'm disturbed by the implications here. Although the distractor is trying to address a possible confusion between the pronunciation of "welding" and "wealthy," the feedback implies that the understanding should be based on the implication that a "workman" [sic] wouldn't be wealthy.

This sort of problem isn't unique. In Listening 1.2, Picture 2, there is a photograph of girls in tutus. The first distractor asks about their shoes, which can't be seen. The feedback indicates this. The third distractor asks about whether they are jumping, and the feedback says they aren't. But, (and this may be nitpickingÑbut these tests are about nitpicking), if we can't see their feet, are we sure they aren't jumping? The correct answer states, "they are happy." However, one can see only one face out of the three (and she doesn't look all that happy.)

Picture Module Example 2. Click on the picture to see a larger image.

Another item in this section is culturally biased: students who don't know the difference between a basketball and a soccer ball won't be able to answer the question about the picture. Their inability will have nothing to do with their listening skill, and everything to do with experience with sporting equipment and vocabulary.

Unfortunately, this is only a sample of the weaknesses in the Picture Modules. There are many more.

The listening modules are less problematic. Although they suffer from the same navigational problems, the sound quality is excellent. After completing a module consisting of 10 questions, the user is told how many were correctly done and in what time. This is especially useful for students trying to improve their time management skills in testing situations. Again, though, the user cannot choose which items to review, but must page through and answer each question correctly before being allowed to move on.

The Short Conversations are also well done in terms of sound quality and of the content of the conversations. Some attempts at humor, however, fall flat. In item 3.14, there is a question about a calendar date of a restaurant reservation. One of the distractors is "Friday the 13th." The feedback says, "No. This is an unlucky day. They might choke on their steak tartare." This seems a gratuitous attempt at humor, and presupposes a superstitious viewpoint. What's worse is that many users may not understand the cultural reference, and thus not understand the humor.

There are six short talks, each followed by a series of questions. Again, while the sound quality is good, the quality of the test items is less consistent. In Item 4.2 (Weather Report #1), one question requires that the user listen for two different reported temperatures, and then perform a subtraction exercise to get the correct answer. Although a clever task, it requires more than listening skill to complete. Granted, it is a simple subtraction exercise, but it turns the question into a memory and math item, rather than a listening one. In Question 4 for this same listening passage, the user has to select the correct Fahrenheit temperature related to a Celsius temperature. This, in fact, can be done without listening at all if the student has a basic understanding of the two temperature measurement systems.

One solution to all of these problems is to ensure that the test items do indeed test the skill intended. If a listening item can be done without listening to the passage, for example, then it should be revised.

The reading exercises are the single sentence, fill-in-the-blank type favored in this type of testing. The navigation for these items is a little smoother than in the listening items, since there are no unlinked "hotspots" and the "hint" button doesn't stay illuminated after use. The feedback is generally good, although a few feedback items don't supply enough information for the user to understand fully why a choice is right or wrong.

This is also a type of test item familiar not only to TOEIC takers, but TOEFL takers as well. In fact, the Alexis system uses the same interface as the TOEFL practice found online at Students International (formerly part of the Okanagan website, now at http://www.studint.net/). This isn't surprising, since the same writers and designers at Okanagan University College created it. An example from the Students International site is below:

TOEFL Test Prep available at Students International. click on image to see a larger version

These types of items are particularly complicated to write, and the authors did a fairly good job with most of them. But, they slipped in a few places. For example:

Nobody, in spite of the warnings, didn't wear protective clothing while sterilizing the equipment.

Which underscored item "must be corrected or rewritten," as per the program instructions? Although it appears that the answer should be (C)"didn't", in fact, there is no easy way to rewrite or revise that one word to come up with a correct sentence. It is the entire verb phrase, "didn't wear," that needs correction. Thus, given the strictures of the test, I would answer (A) "Nobody", since it can easily be corrected or rewritten to produce a correct sentence, such as: "My friend, in spite of the warnings, didn't wear protective clothing while sterilizing the equipment." The feedback, however, indicates that C is the answer.

There are a few other items with similar problems. Given the difficulty of creating this type of item, it is especially important that each distractor be fully considered.

In these activities, the user must interpret information presented in advertisements, signs, and posters. Unfortunately, these few readings are again filled with culturally-specific ideas, such as barn dances and St. Patrick's Day. Given the context of the TOEIC examination, these kinds of questions seem to miss the mark. They have little to do with international business, and are deeply embedded cultural constructs that are not contextualized for the user.

Additional Comments

Other features about this CD-ROM are notable:

1. The entire CD-ROM uses only 16 MB of space (a standard CD-ROM will hold up to 660 MB). The largest file is 332K, and there are only a very few files over 100K. The vast majority are smaller than 50K. It is unclear why this material is presented on a CD-ROM rather than a website with a CD-ROM option. (The CD-ROM option is a good feature for those who must pay local telephone charges for Internet access.) Given the expense of the CD-ROM, much more could have been done with the CD-ROM technology and available space.

2. A great number of the right answers are also the longest answers. This is a common flaw in distractor-writing. In fact, test-taking workshops tell students to be on the lookout for this feature.

3. There is a credit screen at the beginning of each item. However, it scrolls by so quickly that it is difficult to read it before the next screen comes up. There isn't a way to get to those credits other than to reload the page and read fast. If you read fast enough, you'll see that the items were written and designed by Brian Rhodes and his colleagues at Okanagan University College.

4. The focus of TOEIC questions is consistently on business and workplace English. This is not the case with the Alexis system, which more than not presents general English items more suitable for the TOEFL.(See the sample test at http://www.toiec.com.)

Conclusion

Test item writing is a lot harder than one might think. What are often dismissed as mere "multiple guess" tests are in fact extremely difficult to design from a number of standpoints. They require:

• distractors that tap into users' likely confusions,
• items that test specific skills and not cultural knowledge,
• items that test the skill intended, and not other skills (such as memory or mathematics), and
• answers that are clear-cut.

This is why organizations such as ETS have large groups of item writers and thorough systems of testing to determine item validity and reliability in testing.

Unfortunately, the Alexis system fails on a number of accounts, in ways that many "home-grown" practice test systems do: They don't appear to have taken into account the rigor that is required by good testing design. They don't seem to have done the type of validity and reliability tests on these items that one would hope for in published software.

For the number of activities and high quality sound, this CD-ROM will no doubt be useful for test-takers looking for practice. However, the responses and the feedback to many of the items, as well as the navigation, will probably confuse and frustrate them.

All "big tests" and test preparation books undergo constant revision. Items that are deemed insufficient are replaced. A companion website that published updated modules and offered additional content would improve the Alexis system greatly.

References

M.E. Sokolik
University of California, Berkeley
<sokolik@socrates.berkeley.edu>