Home > Research > Publications & Outputs > Evaluating Large Language Models in Relationshi...
View graph of relations

Evaluating Large Language Models in Relationship Extraction from Unstructured Data: Empirical Study from Holocaust Testimonies

Research output: Contribution in Book/Report/Proceedings - With ISBN/ISSNConference contribution/Paperpeer-review

Published
Publication date9/09/2023
Host publicationInternational Conference Recent Advances in Natural Language Processing, RANLP 2023
EditorsGalia Angelova, Maria Kunilovskaya, Ruslan Mitkov
PublisherAssociation for Computational Linguistics
Pages117-123
Number of pages7
ISBN (electronic)9789544520922
<mark>Original language</mark>English

Publication series

NameInternational Conference Recent Advances in Natural Language Processing, RANLP
ISSN (Print)1313-8502

Abstract

Relationship extraction from unstructured data remains one of the most challenging tasks in the field of Natural Language Processing (NLP). The complexity of relationship extraction arises from the need to comprehend
the underlying semantics, syntactic structures, and contextual dependencies within the text. Unstructured data poses challenges with diverse linguistic patterns, implicit relationships, contextual nuances, complicating accurate
relationship identification and extraction.
The emergence of Large Language Models (LLMs), such as GPT (Generative Pre-trained Transformer), has indeed marked a significant advancement in the field of NLP. In this work, we assess and evaluate the effectiveness of LLMs in relationship extraction in the Holocaust testimonies within the context of the Historical realm. By delving into this domain specific context, we aim to gain deeper insights into the performance and capabilities of LLMs in accurately capturing and extracting relationships within the Holocaust domain by
developing a novel knowledge graph to visualise the relationships of the Holocaust. To the best of our knowledge, there is no existing study which discusses relationship extraction in Holocaust testimonies. The majority of
current approaches for Information Extraction (IE) in historic documents are either manual or Optical Character Recognition (OCR) based.
Moreover, in this study, we found that the Subject-Object-Verb extraction using GPT3- based relations produced more meaningful results compared to the Semantic Role labeling based triple extraction.