Handbook of Standards and Resources for Spoken Language Systems
Dafydd Gibbon, Roger Moore and Richard Winski (Editors)
Contents
Preliminaries
Editorial preface
Main technical authors
List of Figures
List of Tables
User's guide
Background
E
AGLES
objectives
E
AGLES
organisational structure
E
AGLES
workplan
Spoken Language systems, standards and resources
Spoken Language systems
Standards and resources for Spoken Language systems
Spoken Language standards and resources in Europe
The ESPRIT S
AM
project
Other EU projects
The European Language Resources Association
Spoken Language standards and resources worldwide
COCOSDA
The Linguistic Data Consortium
The E
AGLES
Spoken Language Working Group (WG5)
Subgroups of the E
AGLES
Spoken Language Working Group
Relationships with the other E
AGLES
Working Groups
Workshops
Production of the handbook
Consultation with the R&D Community
Overview of the handbook
Intended readership
Scope
Spoken Language system design and specification
Spoken Language resources
Assessment of Spoken Language systems
The reference material
The main chapters of the handbook
System design
Spoken Language corpus design
Spoken Language corpus collection
Spoken Language corpus representation
Spoken Language lexica
Language models
Physical characterisation and description
Assessment methodologies and experimental design
Assessment of recognition systems
Assessment of speaker verification systems
Assessment of synthesis systems
Assessment of interactive systems
The current state of play
Possible future actions
Revision and completion of existing documentation
Extended survey of existing practice
Extension of language base
Terminology
Move to prescriptive recommendations
Publication and dissemination
Coordination with other bodies
Contact points
Acknowledgements
Spoken language system and corpus design
System design
Introduction
System capability profile versus application requirement profile
Technical features versus system capabilities
System in operation versus laboratory prototype
Speech recognition systems
Speaker dependency
Speaker-adapted systems
Speaking aspects
Speaking mode
Speaking fluency
Speaking rate
Non-speech sounds
Vocabulary aspects
Lexicon size
Speech training material
Speech data collection
Lexicon generation
Speech modelling and syntactic rules
Field speech data (in-situ recording)
Branching factor (perplexity factor)
Rejection mode
Application vocabulary and confusion matrix
Speaker selection
Language modelling
Channel adaptation/Environment adaptation
Task/Application adaptation
Speech recognition and application interfaces
Speech input and speech signal acquisition
Microphone interfaces
Telephone interfaces
Cut-through versus voice-stop (anticipation)
Error measure presentation
Error handling
Time out
Signal-to-noise ratio
Response time
Speaker verification/identification
Speaker verification versus speaker identification
Speaker characteristics verification or identification
Error measure
Speaker verification errors
Speaker identification error
Training
Exploitation
Text dependent approach
Text independent approach
Speech quantity and quality
Adding / Removing speakers
Speech synthesis
Speech recording, storage, and playback
Canned speech
Text-to-speech synthesis
Linguistic part
Phonetic part
Acoustic module
Quality measures
Multi-linguality
Interactive voice systems
Menu-driven systems
Sophisticated dialogue manager
Interaction with an information database
Error recovery strategy
Routing to an operator
DTMF, pulse detection, and voice recognition
System platform
Software aspects
Operating systems
Drivers
Application programming interfaces (APIs)
Application generators
Hardware aspects
Platforms
Speech processing boards
Speech input/output interfaces
Connectivity
Real-time aspects
Planning for expansion
System simulation and prototyping
Host-computer interfaces
Computer telephony integration
Multi-lingual aspects
System dimension configuration
Statistical tools
Conclusion
Recommendation: Requirement profiles
SL corpus design
Introduction
Spoken language corpus
About this chapter
Eight main differences between collections of written and spoken language data
Durability of text, volatility of speech
Different production times for text and speech
Correcting errors in the production of text and speech
Orthographic identity and phonetic variability of lexicalised units
Printable ASCII-strings and continuously sampled speech
Size differences between NL and SL data
The different legal status of written texts and spoken words
The different nature of categories and time functions
Applications of spoken language corpora
Speech corpora for research purposes
Phonetic research
Sociolinguistic research
Psycholinguistic research
First language acquisition
Second language acquisition
General linguistic research
Audiology
Speech pathology
Speech corpora for technological applications
Speech synthesis
Speech recognition
Knowledge-based
vs.
stochastic systems
Speaker-independent
vs.
speaker-dependent systems
Isolated words
vs.
continuous speech
Corpora for speech recognition research
Spoken language systems
Speaker recognition/verification
Channel conditions
Psychological and physiological conditions
Specification of the linguistic content
Different types of speech data
Read aloud isolated phonemes
Read aloud isolated words
Read aloud isolated sentences
Read aloud text fragments
Semi-spontaneous speech
Spontaneous speech about a predetermined subject
The Wizard of Oz technique
Spontaneous speech
Factorial experiments and corpus studies
Specification of number and type of speakers
Corpus size in terms of speakers
Speech corpora with few speakers
Speech corpora with about 5 to 50 speakers
Speech corpora with more than 50 speakers
General remarks
Speaker characteristics
Stable / transient speaker characteristics
Demographic coverage
Male / female speakers
Age
Voice quality
Vocabulary and syntax
Weight and height
Smoking and drinking habits
Pathological speech
Professional vs. untrained speakers
Geographical and sociolinguistic factors
Final comments
SL corpus collection
Introduction
Data collection dimensions
Visibility: open vs. secret
Clandestine microphone recordings
Recommendations
Visible recordings
Recommendations
Environment: Studio vs. on location
Recording in a studio
Recommendations
Recording on location
Recommendations
Interaction and control
Random recordings
Spontaneous dialogue recordings
Interview recordings
Read speech recordings
Speaker prompting
Recommendations
Monitoring and validation
Multi-channel recording
Laryngography
Electropalatography
Electromagnetic articulography
Cineradiography
Air-flow measurements
X-ray microbeam
Nuclear magnetic resonance imaging
Ultrasound imaging
Procedures
Equipment
Microphone
Recommendations
Amplifier/Processor
Recommendations
Recording device
Recommendations
Management
Speaker database
Speaker recruitment
Recommendations
Scheduling speakers
Recommendations
Cost
Recommendations
Wizard of Oz
Recommendations
Legal aspects
SL corpus representation
Introduction
The transcription of read speech versus the transcription of spontaneous speech
Transcription of dialogues
Levels and types of transcription
Concerning the segmentation and labelling of speech signals
Background
A caveat
Use of segmented speech databases
The C
RIL
conventions
Manual segmentation
Automatic and semi-automatic segmentation
Segmentation and labelling in the V
ERBMOBIL
project
The levels and types of symbolic representations of speech
Recording script
Orthographic transcription
Reduced word forms
Dialect forms
Numbers
Abbreviations and spelled words
Interjections
Orthographic transcription of read speech
Orthographic transcription as the first of many levels
Morphological, syntactic, semantic and pragmatic representations
Citation-phonemic representation
Broad phonetic or phonotypic transcription
Narrow phonetic transcription
Acoustic-phonetic transcription
Physical transcription
Prosodic transcription
Types of approach to prosodic labelling
Examples of the two types of approach
The ToBI labelling system
The M
ARSEC
labelling system
The IPO approach
Prosodic labelling in the V
ERBMOBIL
project
Provisional recommendations for prosodic transcription
Non-linguistic and other phenomena
List of recommendations
Spoken language characterisation
Spoken language lexica
Introduction
Lexica for spoken language systems
Lexical information as properties of words
Types of application for spoken language lexica
Spoken language lexical databases as a general resource
Lexica in selected spoken language systems
Recommendations on resources
What is a spoken language lexicon?
Basic features of a spoken language lexicon
Lexical databases and system lexica for spoken language
Spoken language and written language lexica
Basic lexicographic coverage criteria
The lexicon in spoken language recognition systems
Recommendations on defining spoken language lexica
Types of lexical information in spoken language lexica
Lexicon models and lexical representation
A simple sign model for lexical properties
Lexical units
Kinds of lexical unit
Fully inflected form lexica
Stem and morph lexica
The notion of ``lexical lemma''
Lexical properties and lexical relations in spoken language
Recommendations on types of lexical information
Lexical surface information
Orthographic information
Pronunciation information
Prosodic information
Recommendations on lexical surface information
Morphological information
Types of morphological information
Applications of morphology
Recommendations on morphology
Grammatical information
Statistical language models
Sentence syntax information
Recommendations on grammatical information
Lexical content information
Lexical semantic information
Pragmatic information
Idiomatic information
Recommendations on semantic information
Lexicon structure
Spoken language lexicon formalisms
Lexicon architecture and lexical database structure
Lexicon architecture and the structure of lexical databases
A simple database type: Pronunciation tables
More complex lexical databases
Recommendations on lexicon structure
Lexical knowledge acquisition for spoken language
Stages in lexical knowledge acquisition
Types of knowledge source
Dictionaries
Corpora
Acquisition tools
Recommendations on lexicon construction
Outlook
Language models
Introduction
Goals of the chapter
Advice to the reader
System architecture for speech recognition
Bayes decision rule
Stochastic language modelling
Types of language models
Why does the trigram language model work?
Perplexity definition
Formal definition
Implications of the perplexity definition
Extensions of the perplexity definition
Recommendations
Language model smoothing: modelling unseen events
Problem formulation
Linear discounting and backing-off
Linear interpolation
Absolute discounting and backing-off
Conclusion
Multilevel smoothing for trigram models
Problem formulation
The full trigram model
Practical issues
Cache
Experimental results
Recommendations:
m
-gram language models
Refined language models
Equivalence histories and word classes
Grammar based language models
Recommendations: Refined language models
Language models and search
Sources of recognition errors
Search: Finding the single best sentence
Search: Word graphs
Recommendations: Language models and search
Final note: the mathematics of smoothing
Linear discounting and backing-off
Linear interpolation
Absolute discounting and backing-off
Physical characterisation and description
Introduction
The communication chain
Specifications for speech corpora production and speech assessment environments
Recommendations on recording strategies
Basic notations and terminology
Periodic and aperiodic motion
The dB scale
Phones and sones
Analog and digital speech signal representation
Sampling
Quantisation and coding
Talker / listener descriptors
General (demographic) descriptors
Task specific descriptors
Talker descriptors
General talker descriptors
Anatomical (voice) descriptors
Habitual speech descriptors
Audiometric descriptors
Listener descriptors
General listener descriptors
Audiometric descriptors
Transducer characteristics
Microphones and speech recordings
Microphones
Conversion principles
Directional characteristics
Recording environment
Small room acoustics
Recording rooms
Recording chain: studio vs. on-site
Studio
On-site
Data collection
Recording procedure
Recommendations on microphones and speech recordings
Recommendations on microphones
Recommendations on the recording environment
Recommendations on the recording chain: studio
Recommendations on the recording chain: on-site
Recommendations on data collection
Recommendations on the recording procedure
Parallel recordings
Pitch Determination Instruments (PDIs)
Physiological measurements
Mimic and gesture recordings
Recommendations on parallel recordings
Environment characteristics
Artificial vs. natural environment
Recommendations on artificial vs. natural environment
Auditory environment
Use and compensation of headphones
Open vs. closed headphones
Using diffuse-field equalised headphones
Insertion-loss compensation
Feedback compensation
Modelling of the auditory perspective
Talker mirroring in the virtual room
Subject conditioning
Recommendations on the auditory environment
Visual environment
Head mounted visual displays (``eyephones'')
Frontal visualisation
Recommendations on the visual environment
Miscellaneous environmental factors
Recording chain characteristics
Signals and systems
Linear distortions
Amplitude distortions
Phase distortions
Non-linear distortions
Recommendations on recording chain characteristics
Reproducibility assurance procedures
Description of recording conditions
References and standards
Reference signals
Reference channels
Recommendations on reproducibility
Analysis and conditioning tools for speech signals
Signal analysis and measurement
Hardware devices
Software tools
Signal conditioning
Modification of speech parameters
Signal contamination
Binaural processing
Simulation of environmental factors
Recommendations on analysis and conditioning tools for speech signals
Spoken language system assessment
Assessment methodologies and experimental design
Introduction
How to read this chapter
Role of statistical analysis and experimentation in Language Engineering Standards (LES)
Statistical and experimental procedures for analysing data corpora
Statistical analysis
Populations, samples and other terminology
Sampling
Biases
Estimating sample means, proportions and variances
Estimating means
Estimating proportions
Estimating variance
Ratio of sample variances
Hypothesis testing
Simple hypothesis testing
Analysis of Variance
Non-parametric tests
Experimental procedures
Experimental selection of material
Segmentation
Classification
Assessing recognisers
Baseline performance
Progress
Functional adequacy and user acceptance
Methodology
Application oriented
Reference oriented
Calibrated databases
Manipulated, artificial and diagnostic databases
Experimental design
Assessing speaker verification and recognition systems
Sampling rare events in speaker verification and recognition systems
Employing expert judgments to augment speaker verification and assessment for forensic aspects of speaker verification and recognition
Interactive dialogue systems
Wizard of Oz (WOZ)
Audio-only simulations
Requirements
Subject variables
Wizard variables
Multimodal
Dialogue metrics
Psycholinguistic metrics
Acoustic-based measures
Assessment of recognition systems
Introduction
Classification of recognition systems
Speech quality and conditions
Capability profile versus requirement profile
Assessment purpose versus methodology
Definitions and nomenclature
The performance measure as percentage
Recognition score
Isolated word scoring
Connected or continuous word scoring
Confusions
Vocabulary
Analysis of Variance design
Description of methodologies
Representative databases
Reference methods
Specific calibrated databases
Diagnostic methods with a specific vocabulary
Artificial test signals
Parameters
Pre-production parameters
Post-production parameters
Linguistic parameters
Recogniser specific parameters
Assessment parameters
Experimental design of small vocabulary word recognition
Technical set-up
Training
Test procedure
Addition of noise
Automatic gain control
Scoring the results
Analysis of results
Basic statistics, ANOVA
Confusion matrices
Experimental design of large vocabulary continuous speech recognition
Training material
Acoustic training
Lexicon
Language model
Development test
Dry run
Test material selection
Evaluation protocol
Scoring method
Assessment of speaker verification systems
Presentation
Speaker classification tasks
General definitions
A taxonomy of speaker recognition systems
Task typology
Speaker identification versus speaker verification
Related tasks
Types of error
Levels of text dependence
Interaction mode with the user
Definitions
Examples
Text-dependent systems
Fixed-vocabulary systems
Unrestricted text-independent systems
Influencing factors
Speech quality
Temporal drift
Speech quantity and variety
Speaker population size and typology
Speaker purpose and other human factors
Recommendations
Example
Scoring procedures
Notation
Registered speaker population
Test impostor population
Closed-set identification
Misclassification rates
Mistrust rates
Confidence ranks
Comments
Example
Verification
False rejection rates
False acceptance rates and imposture rates
Relative unreliability, vulnerability and imitation ability
Comments
Example
Expected benefit
Threshold setting
System operating characteristic
System characteristic modelling
Example
Open-set identification
Recommendations
Comparative and indirect assessment
Reference systems
Human calibration
Transformation of speech databases
Applications, systems and products
Terminology
Typology of applications
Telecommunication (remote) applications
On-site (local) applications
Examples of speaker verification systems
Examples of speaker verification products
Alternative techniques
Conclusions
System and product assessment
System assessment
Product assessment
Recommendations
Forensic applications
Listener method
Spectrographic method
Semi-automatic method
Recommendations
Conclusions
Assessment of synthesis systems
Introduction
What are speech output systems?
Why speech output assessment?
Users of this chapter
Towards a taxonomy of assessment tasks and techniques
Glass box vs. black box
Recommendations on choice of test methodology
Laboratory vs. field
Recommendations on choice of test methodology
Linguistic vs. acoustic
Human vs. automated
Judgment vs. functional testing
Global vs. analytic assessment
Methodology
Subjects
Recommendations on choice of subjects
Test procedures
Recommendations on choice of response modality
Benchmarks
Reference conditions
Segmental reference conditions
Recommendations on choice of segmental reference conditions
Prosodic reference conditions
Recommendations on choice of temporal reference conditions
Recommendations on choice of melodic reference conditions
Voice characteristics reference conditions
Overall quality reference conditions
Recommendations on choice of overall quality reference conditions
Comparability across languages
Black box approach
Laboratory testing
Functional laboratory tests
Recommendations on functional testing of overall output quality
Judgment laboratory tests
Recommendations on judgment testing of overall output quality
Field testing
Preliminary remarks
Field tests
Glass box approach
Linguistic aspects
Pre-processing
Grapheme-phoneme conversion
Word stress
Morphological decomposition
Syntactic parsing
Sentence accent
Acoustic aspects
Segments
Functions of segments
Segmental tests
Segmental tests at the word level
Segmental tests at the sentence level
Recommendations on specific segmental quality tests
Prosody
Functions of prosody
Judgment tests of prosody
Functional tests of prosody
Voice characteristics
Functions of voice characteristics
Voice characteristics tests
Relationships among tests
Further developments in speech output testing
Introduction
Long-term strategy: Towards predictive tests
From human to automated testing
Predicting functional behaviour from judgment testing
Predicting global from analytic testing
Predicting field performance from laboratory testing
Linguistic testing: Creating test environments for linguistic interfaces
Acoustic testing: Developments for the near future
Segmental quality testing
Prosodic quality testing
Voice characteristics testing
Overall quality testing
Conclusion: summary of test descriptions
S
AM
Standard Segmental Test
CLuster IDentification Test (CLID)
The Bellcore Test
Diagnostic Rhyme Test (DRT)
Modified Rhyme Test (MRT)
Haskins Syntactic Sentences
S
AM
Semantically Unpredictable Sentences (SUS)
Harvard Psychoacoustic Sentences
S
AM
Prosodic Form Test
S
AM
Prosodic Function Test
S
AM
Overall Quality Test
ITU-T Overall Quality Test
Assessment of interactive systems
Introduction
About this chapter
Purpose
Scope
Terminology
Reading guide
Different needs addressed
Comparing existing systems
Improving existing systems
Designing new systems
Section summary
Interactive dialogue systems
Definitions
Interactive dialogue systems compared to command systems
Interactive complexity of dialogue systems
Menu dialogue systems
Spoken language dialogue systems
Multimodal dialogue systems including speech
Specification and design
Design by intuition
Design by observation
Design by simulation
The Wizard of Oz technique
Requirements for WOZ simulations
Variables in spoken WOZ experiments
Subject variables
Wizard variables
Communication channel variables
An iterative WOZ methodology
The pre-experimental phase
The first experimental phase
Second or subsequent experimental phases
WOZ conclusions
Iterative design methodology for spoken language dialogue systems
Interactive voice response systems: recommendations
Recommendations on design methodology
Recommendations on prompt design
Spoken language dialogue systems: recommendations
Recommendations on design methodology
Additional recommendations
Readings in interactive dialogue system specification
Evaluation
Background
Characterisation
Dialogue system characterisation
Knowledge databases
Dialogue strategies
Task characterisation
Task type
Task complexity
User characterisation
Environment characterisation
Result corpus characterisation
Overall system characterisation
Assessment framework
Assessment metrics
The environment in which tests take place
The degree of simulation or system integration
How much is being evaluated?
Glass box assessment
Black box assessment
Quantitative measures
Qualitative measures
A core set of metrics for system comparison
Dialogue duration
Turn duration
Contextual appropriateness
Correction rate
Transaction success
Recommendations on evaluation methodology
Readings in interactive dialogue system evaluation
References
Spoken language reference materials
Character codes and computer readable alphabets
Introduction
ASCII
Problems
Multi-Byte encodings
ISO 10646
The Unicode standard
Translation of code tables
Specific vs. general converters
MIME
Computer readable phonetic alphabets
IPA subsets
Full IPA alphabet encoding
References
SAMPA computer readable phonetic alphabet
Introduction
Notation issues
Transcription
Coding
Further languages
SAMPA: Present status
The phonemic notation of individual languages
Bulgarian
Danish
Dutch
English
Estonian
French
German
Greek
Hungarian
Italian
Norwegian
Polish
Portuguese
Romanian
Spanish
Swedish
Levels of annotation and extension of SAMPA
SAMPA as a phonemic system
Detailed phonetic or acoustic annotation
SAMPROSA (SAM PROSodic Alphabet): symbol set
References
SAM file formats
Introduction
Speech file and associated description file formats
Label file format
Label file header format
Label file body format
Label types
Label File example
Files used in a recording session using EUROPEC
Corpus File: CORPUS.DBF
Speaker File: SPEAKERS.DBF
Prompt file
Protocol description file
Codes used for presentation protocol (Protocol Description File)
Example of Protocol Description File
Recording conditions file
PRIVATE.PEC file
SAM recording protocols
Definition of terms
Classification of general strategies for recording and prompting
Recording mode
Prompting style
Timing strategy
Recording protocol
Microphone
Other sensors
Speech data capture
Recording environment
Recording mode and prompting style
Recording control
Recording procedure
Integrity checks
Backup procedures
Retrieval procedures
Calibration
Inter site consistency and recording procedure verification
Collation of recordings
SAM software tools
DKISALA (Interactive Semi-Automatic Labelling Software)
ELSA (ESPRIT Labelling System Assessment software)
EUROPEC (European Program d'Enregistrement de Corpus)
PTS (Progiciel de Traitement de Signal)
RESAM
SAMITPRO (S
AM
Iterative Proportional Fitting)
SAM_REC0 Isolated Word Recogniser
SAM_SCOR (Sam Input Assessment Scoring Software)
SAM_SLM
SAM_SPEX (Speech Parameter Extractor)
SAMTRA (S
AM
TRanscription Analysis)
SOAP (Speech Output Assessment Package)
PTM (Parametric Test Manager)
EUROPEC recording tool
Digital storage media
Storage media dimensions
Permanence
Order of access
Other storage media characteristics
Compression
CD-ROM
Physical CD formats
Logical data organisation
CD-R software
CD-R hardware
Future developments
Summary
References
Database Management Systems (DBMSs)
Data model
Hierarchical data model
Summary
Network data model
Summary
Relational data model
Example
Summary
Object-oriented data model
Summary
Deductive data model
Summary
Safe storage of data
Application-independent storage of data
Controlled access to data
Summary
References
Speech standards
Introduction
Signal theory recalls
Speech research
Computer hardware and software
Conclusion
Appendix: Useful anonymous ftp sites
EUROM-1 database overview
Polyphone project overview
Introduction
Reusable resources
The Dutch P
OLYPHONE
corpus
Recording workstation
Speaker selection
The speech material
Postprocessing
Use of P
OLYPHONE
in application development
Train time table information
Training of the recogniser
Building a phonemic lexicon
A model for
yes/no
expressions
Time and date expressions
Phone card services
Automating card services
Speaker verification
Requirements for future corpora
European speech resources
Overview
Criteria for assessment of the situation of Spoken Language Resources
Types and specificities of corpora
Actors in speech resource production
Summary of the current situation on a per language basis
DANISH (Denmark)
Existing databases and their actors
Ongoing projects & new initiatives
Conclusion
DUTCH (The Netherlands)
Existing databases and their actors
Ongoing projects & new initiatives
Conclusion
ENGLISH (United Kingdom)
Existing databases and their actors
Ongoing projects & new initiatives
Conclusion
FRENCH (France, Belgium, Switzerland)
Existing databases and their actors
Ongoing projects & new initiatives
Conclusion
GERMAN (Germany)
Existing databases and their actors
Ongoing projects & new initiatives
Conclusion
GREEK (Greece)
Existing databases and their actors
Ongoing projects & new initiatives
Conclusion
ITALIAN (Italy)
Existing databases and their actors
Ongoing projects & new initiatives
Conclusion
NORWEGIAN (Norway)
Existing databases and their actors
Ongoing projects & new initiatives
Conclusion
PORTUGUESE (Portugal)
Existing databases and their actors
Ongoing projects & new initiatives
Conclusion
SPANISH (Spain)
Existing databases and their actors
Ongoing projects & new initiatives
Conclusion
SWEDISH (Sweden)
Existing databases and their actors
Ongoing projects & new initiatives
Conclusion
General conclusions
Production costs
Transcription and documentation conventions for Speechdat
Introduction
Points of departure
Markings required for scoring
Case
Spelling
Number sequences
Letter sequences
Punctuation
Mispronunciations
Unintelligible words
Word fragments
Verbal deletions
Non-speech acoustic events
Prosodic annotation
Truncated waveforms
Assessment (optional)
Procedure
Speaker information
Some language-specific transcription options
French
English
Italian
The Bavarian Archive for Speech Signals
Introduction
General information
Corpora
Corpora in preparation
Other activities
Cooperations
Future plans
More information
Glossary
List of abbreviations
Copyright © Walter de Gruyter GmbH & Co.
III.98