University of Amsterdam / Amsterdam University of Applied Sciences
Browse

sorry, we can't preview this file

Esselink_et_al_2024_Evaluating_Inter_Annotator_Agreement_for_Non_Manual_Markers_in_Sign_Languages.pdf (278.13 kB)

Evaluating Inter-Annotator Agreement for Non-Manual Markers in Sign Languages

Download (278.13 kB)
conference contribution
posted on 2024-05-15, 12:18 authored by L.D. EsselinkL.D. Esselink, M. OomenM. Oomen, Floris RoelofsenFloris Roelofsen


This paper is part of a larger project that aims to create a standardized procedure for annotating non-manual markers (NMMs) in sign language data. The paper describes two approaches to evaluating inter-annotator agreement, the event-based approach and the frame-based approach, and uses a combination of these two approaches to evaluate the annotation guidelines introduced in Oomen et al. (2023). The evaluation reveals that for several labels in the annotation scheme inter-annotator agreement is rather low. This indicates that the annotations guidelines need to be further improved. We present concrete recommendations for how this may be achieved, and intend to implement these recommendations in future work. All data and analysis scripts are available.


This is a pre-print of a conference paper. It has been accepted for the 11th Workshop on the Representation and Processing of Sign Languages: Evaluation of Sign Language Resources at LREC-COLING 2024.

Funding

Questions in sign language

Dutch Research Council

Find out more...

History

Retention period

2054-01-01

Usage metrics

    University of Amsterdam / Amsterdam University of Applied Sciences

    Licence

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC