Call for Participation: The 4th Workshop on Human Evaluation of NLP Systems
(HumEval’24)
Date: 21 May 2024 (full day)
Venue: Lingotto Conference Centre, Turin, Italy
Registration
Registration is mandatory for attending the workshop. Find more info about
registration on the LREC-COLING 2024 website: https://lrec-coling-2024.org/
Workshop description
Human evaluation plays a central role in NLP, from the large-scale
crowd-sourced evaluations to the much smaller experiments routinely
encountered in conference papers. Yet there is growing unease about how
human evaluations are conducted in NLP. Researchers have pointed out the
less-than-perfect experimental and reporting standards that prevail (van
der Lee et al., 2019 https://aclanthology.org/W19-8643/; Gehrmann et al.,
2023 https://www.jair.org/index.php/jair/article/view/13715/26927), and
that low-quality evaluations with crowdworkers may not correlate well with
high-quality evaluations with domain experts (Freitag et al., 2021
https://aclanthology.org/2021.tacl-1.87). Only a small proportion of
papers provide enough detail for reproduction of human evaluations, and in
many cases the information provided is not even enough to support the
conclusions drawn (Belz et al., 2023
https://aclanthology.org/2023.insights-1.1).
The HumEval workshop (previously at EACL 2021, ACL 2022, and RANLP 2023)
aim to create a forum for current human evaluation research and future
directions, a space for researchers working with human evaluations to
exchange ideas and begin to address the issues human evaluation in NLP
faces in many respects, including experimental design, meta-evaluation and
reproducibility.
Programme
Find the detailed programme on the workshop website:
https://humeval.github.io/2024/programme
Invited speakers
Mark Diaz (Google Research)
Sheila Castilho (ADAPT/DCU)
Organising Committee
Simone Balloccu, Charles University, CZ
Anya Belz, ADAPT Centre, Dublin City University, Ireland
Rudali Huidrom, Dublin City University, Ireland
Ehud Reiter, University of Aberdeen, UK
João Sedoc, New-York University
Craig Thomson, ADAPT Centre, Dublin City University, Ireland
For questions and comments regarding the workshop please contact Simone
Balloccu at balloccu@ufal.mff.cuni.cz and humeval.ws@gmail.com.
--
Kind regards, Simone Balloccu.