Call for Participation: The 4th Workshop on Human Evaluation of NLP Systems (HumEval’24) Date: 21 May 2024 (full day) Venue: Lingotto Conference Centre, Turin, Italy Registration Registration is mandatory for attending the workshop. Find more info about registration on the LREC-COLING 2024 website: https://lrec-coling-2024.org/ Workshop description Human evaluation plays a central role in NLP, from the large-scale crowd-sourced evaluations to the much smaller experiments routinely encountered in conference papers. Yet there is growing unease about how human evaluations are conducted in NLP. Researchers have pointed out the less-than-perfect experimental and reporting standards that prevail (van der Lee et al., 2019 <https://aclanthology.org/W19-8643/>; Gehrmann et al., 2023 <https://www.jair.org/index.php/jair/article/view/13715/26927>), and that low-quality evaluations with crowdworkers may not correlate well with high-quality evaluations with domain experts (Freitag et al., 2021 <https://aclanthology.org/2021.tacl-1.87>). Only a small proportion of papers provide enough detail for reproduction of human evaluations, and in many cases the information provided is not even enough to support the conclusions drawn (Belz et al., 2023 <https://aclanthology.org/2023.insights-1.1>). The HumEval workshop (previously at EACL 2021, ACL 2022, and RANLP 2023) aim to create a forum for current human evaluation research and future directions, a space for researchers working with human evaluations to exchange ideas and begin to address the issues human evaluation in NLP faces in many respects, including experimental design, meta-evaluation and reproducibility. Programme Find the detailed programme on the workshop website: https://humeval.github.io/2024/programme Invited speakers Mark Diaz (Google Research) Sheila Castilho (ADAPT/DCU) Organising Committee Simone Balloccu, Charles University, CZ Anya Belz, ADAPT Centre, Dublin City University, Ireland Rudali Huidrom, Dublin City University, Ireland Ehud Reiter, University of Aberdeen, UK João Sedoc, New-York University Craig Thomson, ADAPT Centre, Dublin City University, Ireland For questions and comments regarding the workshop please contact Simone Balloccu at balloccu@ufal.mff.cuni.cz and humeval.ws@gmail.com. -- Kind regards, Simone Balloccu.