Evaluating Inter-Annotator Agreement for Non-Manual Markers in Sign Languages

This paper is part of a larger project that aims to create a standardized procedure for annotating non-manual markers (NMMs) in sign language data. The paper describes two approaches to evaluating inter-annotator agreement, the event-based approach and the frame-based approach, and uses a combination of these two approaches to evaluate the annotation guidelines introduced in Oomen et al. (2023). The evaluation reveals that for several labels in the annotation scheme inter-annotator agreement is rather low. This indicates that the annotations guidelines need to be further improved. We present concrete recommendations for how this may be achieved, and intend to implement these recommendations in future work. All data and analysis scripts are available.

This is a pre-print of a conference paper. It has been accepted for the 11th Workshop on the Representation and Processing of Sign Languages: Evaluation of Sign Language Resources at LREC-COLING 2024.