TY - UNPB

T1 - Who’s the Fairest of them All?

T2 - A Comparison of Methods for Classifying Tone and Attribution in Earnings-related Management Discourse

AU - Young, Steven

AU - Walker, Martin

AU - Athanasakou, Vasiliki

AU - El-Haj, Mahmoud

AU - Rayson, Paul

AU - Schleicher, Thomas

PY - 2020

Y1 - 2020

N2 - We compare the relative and absolute performance of various machine learning algorithms and wordlists at replicating manual coding results for tone and attribution by domain experts in management performance commentary. Our suite of learning classifiers comprises Naïve Bayes, random forest, support vector machines, and an artificial neural network called multilayer perceptron. We use wordlists proposed by Henry (2006, 2008) and Loughran and McDonald (2010) to classify tone. Wordlists for attribution are based on the causal reasoning list from Language Inquirer and Word Count (LIWC), together with two self-constructed lists. We use a self-constructed wordlist to distinguish between internal and external attributions. We train learning classifiers using a large sample of manually annotated performance sentences. Results for all classifiers are assessed using a separate manually annotated holdout sample. Conclusions regarding the best classification method vary according to the classification task. None of the approaches are capable of identifying the presence of an attribution reliably. Even for more reliable classification tasks such as tone and attribution type, absolute measurement errors often exceed 20%. We conclude that while automated textual analysis methods offer important opportunities in certain settings, manual content analysis remains an essential tool for researchers interested in studying the properties and consequences of financial discourse.

AB - We compare the relative and absolute performance of various machine learning algorithms and wordlists at replicating manual coding results for tone and attribution by domain experts in management performance commentary. Our suite of learning classifiers comprises Naïve Bayes, random forest, support vector machines, and an artificial neural network called multilayer perceptron. We use wordlists proposed by Henry (2006, 2008) and Loughran and McDonald (2010) to classify tone. Wordlists for attribution are based on the causal reasoning list from Language Inquirer and Word Count (LIWC), together with two self-constructed lists. We use a self-constructed wordlist to distinguish between internal and external attributions. We train learning classifiers using a large sample of manually annotated performance sentences. Results for all classifiers are assessed using a separate manually annotated holdout sample. Conclusions regarding the best classification method vary according to the classification task. None of the approaches are capable of identifying the presence of an attribution reliably. Even for more reliable classification tasks such as tone and attribution type, absolute measurement errors often exceed 20%. We conclude that while automated textual analysis methods offer important opportunities in certain settings, manual content analysis remains an essential tool for researchers interested in studying the properties and consequences of financial discourse.

M3 - Working paper

SP - 1

EP - 47

BT - Who’s the Fairest of them All?

ER -