Structured text: hierarchy and coding

1. Characters:
   • smallest units (cf. phonemes)
   • in many document description languages coded with ASCII code sequences (exception: internal representations of word processors)
2. Tokens, symbols:
   • smallest interpretable units (cf. morphemes)
   • identified by tokenisation, and
   • generally describable with a regular grammar (or a finite state automaton)
3. Objects:
   • groups of tokens with complex meaning (cf. phrases)
   • identified by parsing and
   • generally describable with a context-free grammar (or a push-down automaton)