5 Notational system

Two important criteria for transcription are machine readability and rendering for human readability. We first propose a feature-based vector notation for machine readability. Abbreviations for human readability are under development.

A gesture respresents a set of values of the gesture attributes in the following order:

1. source:
   • location
   • hand shape

2. trajectory:
   • directionality
   • shape
   • hand shape
   • modifiers
     • size
     • repetitions
     • speed

3. target:
   • location
   • hand shape

4. symmetry

A gesture can consist of a source, a trajectory and a target. Whereas for dynamic gestures all three features need to be specified, for static gestures only the source needs to be described, since there is no trajectory and thus the source and the target are identical.

5.0.0.1 Source and Target.

When making a complete annotation without gaps (e.g. empty or unsystematic notations), the target location and target hand shape of a gesture and the source location and source hand shape of the following gesture are identical. Thus, the target labels of the gestures can be omitted for reasons of efficiency.

The categories source and target both consist of the features location and hand shape. The category trajectory contains the features directionality, shape, hand shape, and modifiers.
The attributes size, repetitions, and speed are subsumed in the category of modifiers.
If a gesture is produced with a pair of body parts, e.g. hands, it must be added whether it is produced either in a parallel or symmetrical fashion, or whether the limbs act independently.

The transcription system is tentative and is still under development. We currently propose that transcription symbols are concatenated in two ways: in order to connect values on different attribute levels, a ``,'' is used. The following ASCII-based labels for values apply:

5.0.0.2 Location.

Location is described by using a modified and extended matrix-like version of the HamNoSys 4.0 charts on Hand and Body Position and on Distance from the Body (see Table 2).

Table 2: Possible Hand and Body Positions according to HamNoSys 4.0

	left of left side	left side of	middle of	right side of	right of right side
above the head	1ll	1l	1m	1r	1rr
head	2ll	2l	2m	2r	2rr
forehead	3ll	3l	3m	3r	5rr
nose	4ll	4l	4m	4r	4rr
under the nose	5ll	5l	5m	5r	5rr
mouth	6ll	6l	6m	6r	6rr
tongue	7ll	7l	7m	7r	7rr
teeth	8ll	8l	8m	8r	8rr
chin	9ll	9l	9m	9r	9rr
under the chin	10ll	10l	10m	10r	10rr
neck	11ll	11l	11m	11r	11rr
shoulder region	12ll	12l	12m	12r	12rr
breast region	13ll	13l	13m	13r	13rr
stomach region	14ll	14l	14m	14r	14rr
lap	15ll	15l	15m	15r	15rr
thighs	16ll	16l	16m	16r	16rr
knees	17ll	17l	17m	17r	17rr
lower legs	18ll	18l	18m	18r	18rr
feet	19ll	19l	19m	19r	19rr

5.0.0.3 Hand Shape.

Hand shape symbols are taken from the FORM hand shape inventory (see Martell, 2002). The label A5, for instance, stands for a relaxed hand. The label A7 for folded hands has been added to the categories.

Figure 4: FORM inventory of hand shapes (Martell, 2002).

5.0.0.4 Directionality.

The labels for directionality are:

• $up$ (up)
• $do$ (down)
• $fo$ (forward)
• $ba$ (backward)
• $le$ (left)
• $ri$ (right)

Combinations of these directions are joined with a slash (``$/$''), i.e. $up/fo$ for an up-forward direction.

5.0.0.5 Shape.

The labels for trajectory shape are:

• $ci$ (circle)
• $li$ (straight line)
• $wl$ (wavy line)
• $ar$ (arch)
• $zl$ (zig-zag line)
• $el$ (ellipsis)
• $sq$ (square)

Further categories may, of course, be added if necessary.

5.0.0.6 Modifiers.

The basic functional transcription as just outlined can be supplemented by additional symbols, some of which only apply to particular parts of the body (e.g. hands):

• $r(x)$ describes repeated gesture, $x$ indicating the number of repetitions. The value $r(0)$ is taken as the default and can be omitted.

• Since the size of gestures tends to be subject to strong variation, further specification of this may be of interest. Hence the following labels: $xs$ (very small), $s$ (small), $m$ (medium), $l$ (large), and $xl$ (very large). The value $m$ is considered the default and thus can be omitted.
• The speed of a movement can also vary. The annotation markers for this additional category are: $sl$, (slow), $me$ (medium), $fa$ (fast). Again, the default, $me$, can be left out.

5.0.0.7 Symmetry.

When describing a gesture which is executed with a pair of body parts or limbs, e.g. the hands, this category becomes relevant because it is necessary to define which part of the pair is used or how both parts are related to each other when moving about. If a gesture is executed with only one body part, the category can be assigned the value $rp$ (right part, relative to the speaker) or $lp$ (left part). If a gesture is executed with both parts, the general case would be first to describe the gesture produced with the right part, then the gesture produced with the left part, concatenated by ``$;$''. For gestures where the body parts move independently within the same time frame, this annotation does not have to be changed. However, a great number of gestures tend to be of a symmetrical or parallel nature, i.e. the body parts move either symmetrically or parallelly. In these cases, the two notational strings are merged and are assigned the values $sy$ (symmetrical) or $pa$ (parallel) under this category. This can, however, also be added on to. It might, for example, be necessary to define the type of symmetry as symmetrical to an axis (radial or bilateral symmetry) vs. symmetrical to a center.

If it is deemed necessary by the individual researcher to make an inter-rater comparison, all attributes need to be assigned a value. So even the optional ones would get their default value. In order to avoid notational ambiguity, it is necessary to keep in mind that the notational order is fixed since some values can occur more than once in a single string.

Thus, a gesture can be described in form of a vector with a maximum of values as listed below:

1. location of source: see Table 2
2. hand shape of source: see Figure 4
3. directionality of trajectory: $ri, le, fo, ba, up, do$ and combinations of those
4. shape of trajectory: $ci, li, wl, ar, zl, el, sq$
5. hand shape of trajectory: see Figure 4
6. size of trajectory: $xs, s, m, l, xl$
7. repetitions of trajectory: $r(0), r(1), r(2),$ etc.
8. speed of trajectory: $sl, me, fa$

9. (location of target: see Table 2)
10. (hand shape of target: see Figure 4)

11. symmetry: $rp, lp, sy, pa$

To give an example, the transcription

$15m,5A,ri,ci,1B,l,r(0),me,15m,5A,rp$

for the hands describes an unrepeated gesture carried out with medium speed with the right hand tracing a large circle with a pointed index finger, which starts and ends with the hands in a relaxed position on the lap.