TY - GEN

T1 - Evaluating Large Language Models in Relationship Extraction from Unstructured Data: Empirical Study from Holocaust Testimonies

AU - Nanomi Arachchige, Isuri

AU - Ha, Le An

AU - Mitkov, Ruslan

AU - Nahar, Vinitar

PY - 2023/9/9

Y1 - 2023/9/9

N2 - Relationship extraction from unstructured data remains one of the most challenging tasks in the field of Natural Language Processing (NLP). The complexity of relationship extraction arises from the need to comprehendthe underlying semantics, syntactic structures, and contextual dependencies within the text. Unstructured data poses challenges with diverse linguistic patterns, implicit relationships, contextual nuances, complicating accuraterelationship identification and extraction.The emergence of Large Language Models (LLMs), such as GPT (Generative Pre-trained Transformer), has indeed marked a significant advancement in the field of NLP. In this work, we assess and evaluate the effectiveness of LLMs in relationship extraction in the Holocaust testimonies within the context of the Historical realm. By delving into this domain specific context, we aim to gain deeper insights into the performance and capabilities of LLMs in accurately capturing and extracting relationships within the Holocaust domain bydeveloping a novel knowledge graph to visualise the relationships of the Holocaust. To the best of our knowledge, there is no existing study which discusses relationship extraction in Holocaust testimonies. The majority ofcurrent approaches for Information Extraction (IE) in historic documents are either manual or Optical Character Recognition (OCR) based.Moreover, in this study, we found that the Subject-Object-Verb extraction using GPT3- based relations produced more meaningful results compared to the Semantic Role labeling based triple extraction.

AB - Relationship extraction from unstructured data remains one of the most challenging tasks in the field of Natural Language Processing (NLP). The complexity of relationship extraction arises from the need to comprehendthe underlying semantics, syntactic structures, and contextual dependencies within the text. Unstructured data poses challenges with diverse linguistic patterns, implicit relationships, contextual nuances, complicating accuraterelationship identification and extraction.The emergence of Large Language Models (LLMs), such as GPT (Generative Pre-trained Transformer), has indeed marked a significant advancement in the field of NLP. In this work, we assess and evaluate the effectiveness of LLMs in relationship extraction in the Holocaust testimonies within the context of the Historical realm. By delving into this domain specific context, we aim to gain deeper insights into the performance and capabilities of LLMs in accurately capturing and extracting relationships within the Holocaust domain bydeveloping a novel knowledge graph to visualise the relationships of the Holocaust. To the best of our knowledge, there is no existing study which discusses relationship extraction in Holocaust testimonies. The majority ofcurrent approaches for Information Extraction (IE) in historic documents are either manual or Optical Character Recognition (OCR) based.Moreover, in this study, we found that the Subject-Object-Verb extraction using GPT3- based relations produced more meaningful results compared to the Semantic Role labeling based triple extraction.

U2 - 10.26615/978-954-452-092-2_013

DO - 10.26615/978-954-452-092-2_013

M3 - Conference contribution/Paper

T3 - International Conference Recent Advances in Natural Language Processing, RANLP

SP - 117

EP - 123

BT - International Conference Recent Advances in Natural Language Processing, RANLP 2023

A2 - Angelova, Galia

A2 - Kunilovskaya, Maria

A2 - Mitkov, Ruslan

PB - Association for Computational Linguistics

ER -