Spectrum Lexicography Workshop 2004

Timetable

First few weeks of term: Introduction to computational corpus lexicography:
- Terminology
- Concordancing (see below)

Tuesday 18 May 2004, 12:15h (Phonetics Lab)
- Nadine Borchardt: Shoebox - practical aspeccts.

Tuesday 25 May 2004, 12:15h (Phonetics Lab)
- (LREC conference period)

Tuesday 01 June 2004, 12:15h (Phonetics Lab)
- Nadine Borchardt: Shoebox - practical aspects.

Tuesday 08 June 2004, 12:15h (Phonetics Lab)
- Caroline Sporleder: Machine learning and lexicography.

Tuesday 15 June 2004, 12:15h (Phonetics Lab)
Shoebox (continuation; Nadine Borchardt and others):
- defining and modifying database structure
- microstructure definitions (types of lexical information)
Owing to illness, in fact we discussed the following aspects of the
Ibibio lexicography work:
- Structure of printed version of Ibibio electronic dictionary
- Lexicographic procedure from tabular capture of lexical information to
  adding format markup and printing the dictionary.
- UNIX script for format markup script.

Tuesday 22 June 2004 at 12:15h (Phonetics Lab)
Discussion of current lexicographic projects:
- Ibibio project (Eno-Abasi Urua, Moses Ekpenyong)
- Saba Amsalu
- Nadine Borchardt
- Anna Garbar
- Thorsten Trippel (state of the lexicon in the Modelex project)

Tuesday 29 June 2004 at 12:15h (Phonetics Lab)
Dafydd Gibbon: Formalising the computational lexicon (theory)
- generalisation as an operation over feature structures,
- generalisation in inheritance networks,
- the ILEX (Integrated Lexicon with Exceptions) lexicon model.

Tuesday 6 July at 12:15h (Phonetics Lab)
Dafydd Gibbon: Implementing the formal computational lexicon (practice)
- DATR resources (website)
- ZDATR web interface
- ZDATR UNIX and Windows command line interface

Tuesday 13 July at 12:15h (Phonetics Lab)
Dafydd Gibbon & Thorsten Trippel: The TAMINO XML lexical database.

Tuesday 20 July at 12:15h (Phonetics Lab)
Continuation of project oriented discussions
(I will be at the EMELD conference in Detroit)

Tuesday 29 July at 12:15h (Phonetics Lab)
- Final meeting: coordination of summer months
- Plans for follow-up to the ModeLex project

Corpus lexicography notes

1. The general rule in corpus-based lexicography is: the larger the corpus, the better.
2. Higher order n-grams can be generated using the same techniques; the higher the value of n, the larger the corpus should be.
3. Note that the script is not optimised in any way but is intended to illustrate the use of a fairly wide range of UNIX tool techniques (my excuse for hacking this script).
4. For further information see:
   
   • Gibbon, Dafydd, Roger Moore & Richard Winski, eds. (1997). Handbook of Standards and Resources for Spoken Language Systems. Berlin: Mouton de Gruyter. [Especially my chapter on the lexicon.]
   • van Eynde, Frank & Dafydd Gibbon, eds. (2000). Lexicon Development for Speech and Language Processing. Dordrecht: Kluwer Academic Publishers. [Especially my introductory chapter.]
   • Gibbon, Dafydd Inge Mertins & Roger Moore, eds. (2000). Handbook of Multimodal and Spoken Dialogue Systems. New York &c.: Kluwer Academic Publishers. [Especially the chapter on terminology.]
5. lexicography2004-04-27.txt (3.0k)

Frequency dictionary (using UNIX script)

INPUT: text
OUTPUT: wordlists and digram lists
METHOD: remove punctuation, put each word on separate line, sort and count doublets, work out frequency information.

• lexscript01.sh (5.0k; archaic HTML wrapper needs to be removed and file renamed)

INPUT

The input corpus from which the output files were generated is simply the lexicography notes file linked above.

OUTPUT

Automatically generated word output files in order of generation:

1. lex_out_wordlist.txt (3.0k)
2. lex_out_alphasorted.txt (4.0k)
3. lex_out_numericsorted.txt (4.0k)
4. lex_out_ranked.txt (6.0k)

1. digram_out_list.txt (5.0k)
2. digram_out_alphasorted.txt (8.0k)
3. digram_out_numericsorted.txt (8.0k)
4. digram_out_ranked.txt (12k)

KWIC concordance (using UNIX script)

INPUT: plain ASCII text
OUTPUT: normalised text, wordlist, concordance
METHOD: normalise text into punctuation-delimited phrases, create wordlist (see above), for each word list phrases in which it occurs.

• kwic01.html

INPUT

• conc_test.txt

1. conc_test-norm.txt (4.0k, 156 lines, 3 pages)
2. conc_test-words.txt (4.0k, 423 lines, 9 pages)
3. conc_test-conc.txt (73k, 1725 lines, 35 pages!)

The Shoebox lexical database management system (DBMS)

Ibibio dictionary project

Formalising lexical structure

Overview

• Lexical structures require formalisation for
  1. better scientific understanding of lexica,
  2. modelling the mental lexicon,
  3. acquisition of lexical information in the creation of databases,
  4. semi-automatic production of dictionaries from databases.
• Lexical structures are:
  1. Macrostructure (usually consisting of table and tree structures)
  2. Mesostructure (generalisations over lexical items in the form of
     1. relations between lexical items defined in terms of different elements of microstructures,
     2. definitions of categories used in microstructures
     3. concordance-like contexts in which tokens of lexical items occur.
  3. Microstructure:
     1. types of lexical information
     2. structured as a vector (n-tuple), a tree of finite depth.
• A suitable formalism for formalising lexical structure is the attribute-value logic.

Modelling conventions

1. a set of objects to be modelled,
2. a set of relations between these objects,
3. a function which interprets the formalism in terms of these objects.

1. standard lexical database conventions,
2. inheritance knowledge base conventions.

Lexical database modelling conventions

1. Macrostructure: table (i.e. a vector of rows), where
   1. each row represents a lexical entry,
   2. each column represents a type of lexical information.
2. Microstructure: none.
3. Microstructure: row of fields in a table, each field representing a lexical information value of the type defined by the column in which it occurs.

1. homography,
2. homophony,
3. homonymy,
4. polysemy,
5. semantic relations (hyponymy, syntonymy, antonymy, ...)
6. definition of categories,
7. provision of examples.

Lexical inheritance network modelling conventions

1. The macrostructure is a set of nodes, partitioned into entry nodes representing lexical entries, and class nodes, representing lexical classes or fields.
2. The mesostructure is a pair of relations over these nodes, class inheritance and entry inheritance, represented as edges between nodes in a graph. The relations represent generalisations over lexical entries.
3. The microstructures of nodes are attribute-value structures; values are concatenations (or other structure-creating operations) of atomic or node references. The attribute-value structures represent types of lexical information.

Coding