Technical report: Evaluating inter-annotator agreement for non-manual markers in sign languages

This report is part of a larger project that aims to create a standardized procedure for annotating non-manual markers (NMMs) in sign language data. The initial steps we took as part of this project—developing annotation guidelines and creating a dataset annotated according to these guidelines by two annotators—were previously reported in Oomen et al. (2023). The present report concerns the next step: a thorough evaluation of inter-annotator agreement, yielding substantial recommendations for improvement of the guidelines.

The report is structured as follows. In Section 1, we briefly outline our general motivations for developing a new protocol for annotating NMMs, and provide a brief summary of the first steps towards such a protocol as reported in Oomen et al. (2023). Section 2 describes general methods for evaluating inter-annotator agreement that can be applied to sign language data. Sections 3 and 4 discuss the results of applying these methods to our test dataset, where Section 3 focuses on general findings and Section 4 on tier-specific results and recommendations. Section 5 then introduces the proposed new tier structure for the updated version of the annotation guidelines. Section 6 considers two ways of fine-tuning the way in which inter-annotator agreement is calculated, Section 7 provides a general discussion, and Section 8 concludes.

Update 30-04-2024:
- Version 1.2 is updated to fine-tune recommendations and to include the proposed new tier structure.