Requirements for lexicon building

However high the descriptive goals, and however sound the theory behind the software, lexicography is primarily task-driven, and lexica are primarily purpose-built. Consequently, the prime issue underlying a lexicographic project, like a software development project, is the requirements specification, i.e. the statement of practical goals which will later be used to evaluate the result of the project. A lexicon for use in automatic dictation software will have a different requirements specification from the paper dictionary consulted by a Scrabble player: the Scrabble player needs no information about pronunciation, certainly no statistics, and may effectively need no more than a list of word forms in their standard orthography, while other lexicon types require many different types of lexical information.

The spoken language system designer, for example, will require a lexicon as a piece of software (lingware, i.e. machine-readable linguistic data and models, and tools for lexicon construction and access) containing information mainly about the following:

• statistical details of the relation between words and acoustic features, embodied in a Hidden Markov Model (a variety of probabilistic finite state machine), coupled with
• statistical information about the co-occurrence of words in sequence, coupled with
• other kinds of lexical information, depending on the application, e.g. as front end to a text editing programme, or to a database querying system.

Based on a requirements specification, the design of a lexicon covers the next important set of conditions on lexicon construction:

• lexicon structure in terms of relations between lexical entries (macrostructure), seen for instance in the distinction between the semasiological form-oriented, often word-based lexicon with orthographic alphabetic sorting, the pronunciation lexicon or the rhyming lexicon, to the onomasiological hierarchical taxonomies which are characteristic of thesauri of semantically related word families.
• types of lexical information for each entry (microstructure), from surface details of pronunciation and orthography, to details of word structure as simple or compound words, the syntactic properties of lexical entries, their meaning and the pragmatics of their use in real contexts.

Both macrostructure and microstructure are designed to relate as easily as possible to the application detailed in the requirements specification.

Finally, the lexicon design is realised in a specific, concrete, operational implementation, for which the following are developed:

• a project definition in terms of acquisition, construction and formatting tasks and their interdependencies, and a mapping of tasks on to a time-scale;
• an evaluation schedule in terms of the requirements specifications and the design, and in respect of the estimated life-cycle of the lexicon as a product;
• a maintenance schedule (revision and extension) in order to attain -- perhaps update -- the requirements specification.

The clearest illustrations of this development process are to be found in the domains of speech synthesis ([Quazza & van den Heuvel (this volume)]) and automatic speech recognition ([Adda-Decker & Lamel (this volume)]). The requirements specification includes vocabulary size or coverage (both extensional coverage, i.e. the number of entries, and intensional coverage, i.e. the number of information fields), the semantic domain covered by the application, and the speaker category with reference to the intended system users. Starting from these, the design and implementation procedures are strictly adhered to, through the development to the evaluation of the lexicon within the context of an operational system, using criteria of word error rate or ergonomic integration within an overall application (such as word processing) in speech recognition, or naturalness, comprehensibility and acceptability in speech synthesis.