7.2 Pilot evaluation

A two-fold pilot evaluation of the CoGesT system has been carried out. Three independent annotators annotated a 15 minute video of a professional story-teller telling the fairy tale ``Das Eselein'' in German. The annotations were compared with regard to the segmentation and classification of gestures.

Three annotators transcribed a roughly pre-segmented annotation file, which, despite the pre-defined boundaries, could be changed or further subsegmented in case it was deemed necessary. The overall agreement with regard to the segmentation was 85 percent. As to the actual classification of those segments, the average inter-annotator agreement was only 23.4 percent. This was due to two types of reasons. As for the definition of the categories,

• the differences between neighbouring body locations are often very subtle; thus some locations can be described with different labels meaning almost the same (e.g. lap vs. thighs when the speaker is seated);
• some movements can be described in different ways, e.g. what one annotator sees as a circle, another annotator may classify as a string of arches; thus, the segmentation as well as the classification is not well-defined in some areas;
• the classification of trajectories with regard to their modifiers size, repetitions and speed is highly subjective and thus may vary depending on the different annotators;
• typing errors lead to inconsistencies in the annotation process.

As the specification logic is concerned, there were weaknesses with regard to the specification of symmetry since, in some cases, is not clear which axes are referred to.

The system has now been reviewed with regard to most of these points. However, further additions and modifications will still have to be made within the course of further evaluations.