Standard
Rater Cohesion and Quality from a Vicarious Perspective. / Panditha, Deepak ; Weerasooriya, Tharindu ; Dutta, Sujan et al.
Findings of the Association for Computational Linguistics: EMNLP 2024. ed. / Yasal Al-Onaizan; Mohit Bansal; Yun-Nung Chen. Kerrville, Texas: Association for Computational Linguistics (ACL Anthology), 2024. p. 5149-5162.
Research output: Contribution in Book/Report/Proceedings - With ISBN/ISSN › Conference contribution/Paper › peer-review
Harvard
Panditha, D, Weerasooriya, T, Dutta, S, Luger, S
, Ranasinghe, T, KhudaBukhsh, A, Zampieri, M & Homan, C 2024,
Rater Cohesion and Quality from a Vicarious Perspective. in Y Al-Onaizan, M Bansal & Y-N Chen (eds),
Findings of the Association for Computational Linguistics: EMNLP 2024. Association for Computational Linguistics (ACL Anthology), Kerrville, Texas, pp. 5149-5162, The 2024 Conference on Empirical Methods in Natural Language Processing, Miami, United States,
12/11/24. <
https://aclanthology.org/2024.findings-emnlp.296/>
APA
Panditha, D., Weerasooriya, T., Dutta, S., Luger, S.
, Ranasinghe, T., KhudaBukhsh, A., Zampieri, M., & Homan, C. (2024).
Rater Cohesion and Quality from a Vicarious Perspective. In Y. Al-Onaizan, M. Bansal, & Y.-N. Chen (Eds.),
Findings of the Association for Computational Linguistics: EMNLP 2024 (pp. 5149-5162). Association for Computational Linguistics (ACL Anthology).
https://aclanthology.org/2024.findings-emnlp.296/Vancouver
Panditha D, Weerasooriya T, Dutta S, Luger S
, Ranasinghe T, KhudaBukhsh A et al.
Rater Cohesion and Quality from a Vicarious Perspective. In Al-Onaizan Y, Bansal M, Chen YN, editors, Findings of the Association for Computational Linguistics: EMNLP 2024. Kerrville, Texas: Association for Computational Linguistics (ACL Anthology). 2024. p. 5149-5162
Author
Panditha, Deepak ; Weerasooriya, Tharindu ; Dutta, Sujan et al. /
Rater Cohesion and Quality from a Vicarious Perspective. Findings of the Association for Computational Linguistics: EMNLP 2024. editor / Yasal Al-Onaizan ; Mohit Bansal ; Yun-Nung Chen. Kerrville, Texas : Association for Computational Linguistics (ACL Anthology), 2024. pp. 5149-5162
Bibtex
@inproceedings{76599f45bfbe441bad23af74a7f13ebf,
title = "Rater Cohesion and Quality from a Vicarious Perspective",
abstract = "Human feedback is essential for building human-centered AI systems across domains where disagreement is prevalent, such as AI safety, content moderation, or sentiment analysis. Many disagreements, particularly in politically charged settings, arise because raters have opposing values or beliefs. Vicarious annotation is a method for breaking down disagreement by asking raters how they think others would annotate the data. In this paper, we explore the use of vicarious annotation with analytical methods for moderating rater disagreement. We employ rater cohesion metrics to study the potential influence of political affiliations and demographic backgrounds on raters{\textquoteright} perceptions of offense. Additionally, we utilize CrowdTruth{\textquoteright}s rater quality metrics, which consider the demographics of the raters, to score the raters and their annotations. We study how the rater quality metrics influence the in-group and cross-group rater cohesion across the personal and vicarious levels.",
author = "Deepak Panditha and Tharindu Weerasooriya and Sujan Dutta and Sarah Luger and Tharindu Ranasinghe and Ashiqur KhudaBukhsh and Marcos Zampieri and Christopher Homan",
year = "2024",
month = nov,
day = "12",
language = "English",
pages = "5149--5162",
editor = "Yasal Al-Onaizan and Mohit Bansal and Yun-Nung Chen",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
publisher = "Association for Computational Linguistics (ACL Anthology)",
note = "The 2024 Conference on Empirical Methods in Natural Language Processing ; Conference date: 12-11-2024 Through 16-11-2024",
url = "https://2024.emnlp.org/",
}
RIS
TY - GEN
T1 - Rater Cohesion and Quality from a Vicarious Perspective
AU - Panditha, Deepak
AU - Weerasooriya, Tharindu
AU - Dutta, Sujan
AU - Luger, Sarah
AU - Ranasinghe, Tharindu
AU - KhudaBukhsh, Ashiqur
AU - Zampieri, Marcos
AU - Homan, Christopher
PY - 2024/11/12
Y1 - 2024/11/12
N2 - Human feedback is essential for building human-centered AI systems across domains where disagreement is prevalent, such as AI safety, content moderation, or sentiment analysis. Many disagreements, particularly in politically charged settings, arise because raters have opposing values or beliefs. Vicarious annotation is a method for breaking down disagreement by asking raters how they think others would annotate the data. In this paper, we explore the use of vicarious annotation with analytical methods for moderating rater disagreement. We employ rater cohesion metrics to study the potential influence of political affiliations and demographic backgrounds on raters’ perceptions of offense. Additionally, we utilize CrowdTruth’s rater quality metrics, which consider the demographics of the raters, to score the raters and their annotations. We study how the rater quality metrics influence the in-group and cross-group rater cohesion across the personal and vicarious levels.
AB - Human feedback is essential for building human-centered AI systems across domains where disagreement is prevalent, such as AI safety, content moderation, or sentiment analysis. Many disagreements, particularly in politically charged settings, arise because raters have opposing values or beliefs. Vicarious annotation is a method for breaking down disagreement by asking raters how they think others would annotate the data. In this paper, we explore the use of vicarious annotation with analytical methods for moderating rater disagreement. We employ rater cohesion metrics to study the potential influence of political affiliations and demographic backgrounds on raters’ perceptions of offense. Additionally, we utilize CrowdTruth’s rater quality metrics, which consider the demographics of the raters, to score the raters and their annotations. We study how the rater quality metrics influence the in-group and cross-group rater cohesion across the personal and vicarious levels.
M3 - Conference contribution/Paper
SP - 5149
EP - 5162
BT - Findings of the Association for Computational Linguistics: EMNLP 2024
A2 - Al-Onaizan, Yasal
A2 - Bansal, Mohit
A2 - Chen, Yun-Nung
PB - Association for Computational Linguistics (ACL Anthology)
CY - Kerrville, Texas
T2 - The 2024 Conference on Empirical Methods in Natural Language Processing
Y2 - 12 November 2024 through 16 November 2024
ER -
