University of Amsterdam / Amsterdam University of Applied Sciences

sorry, we can't preview this file

Technical_report___Inter_Rater_Reliability_v1_2.pdf (5.5 MB)

Technical report: Evaluating inter-annotator agreement for non-manual markers in sign languages

Download (5.5 MB)
Version 2 2024-04-30, 10:03
Version 1 2024-04-08, 15:03
online resource
posted on 2024-04-30, 10:03 authored by L.D. EsselinkL.D. Esselink, M. OomenM. Oomen, Floris RoelofsenFloris Roelofsen

This report is part of a larger project that aims to create a standardized procedure for annotating non-manual markers (NMMs) in sign language data. The initial steps we took as part of this project—developing annotation guidelines and creating a dataset annotated according to these guidelines by two annotators—were previously reported in Oomen et al. (2023). The present report concerns the next step: a thorough evaluation of inter-annotator agreement, yielding substantial recommendations for improvement of the guidelines.

The report is structured as follows. In Section 1, we briefly outline our general motivations for developing a new protocol for annotating NMMs, and provide a brief summary of the first steps towards such a protocol as reported in Oomen et al. (2023). Section 2 describes general methods for evaluating inter-annotator agreement that can be applied to sign language data. Sections 3 and 4 discuss the results of applying these methods to our test dataset, where Section 3 focuses on general findings and Section 4 on tier-specific results and recommendations. Section 5 then introduces the proposed new tier structure for the updated version of the annotation guidelines. Section 6 considers two ways of fine-tuning the way in which inter-annotator agreement is calculated, Section 7 provides a general discussion, and Section 8 concludes.

Update 30-04-2024:
- Version 1.2 is updated to fine-tune recommendations and to include the proposed new tier structure.


Questions in sign language

Dutch Research Council

Find out more...


Retention period


Usage metrics

    University of Amsterdam / Amsterdam University of Applied Sciences



    Ref. manager