Home > Research > Publications & Outputs > Bringing replication and reproduction together ...

Electronic data

  • bringing_replication_reproduction

    Submitted manuscript, 533 KB, PDF-document

    Available under license: CC BY: Creative Commons Attribution 4.0 International License

  • bringing-replication-reproduction

    Accepted author manuscript, 535 KB, PDF-document

    Available under license: CC BY: Creative Commons Attribution 4.0 International License

Links

View graph of relations

Bringing replication and reproduction together with generalisability in NLP: Three reproduction studies for Target Dependent Sentiment Analysis

Research output: Contribution in Book/Report/Proceedings - With ISBN/ISSNConference contribution/Paper

Published

Standard

Bringing replication and reproduction together with generalisability in NLP : Three reproduction studies for Target Dependent Sentiment Analysis. / Moore, Andrew; Rayson, Paul Edward.

Proceedings of the 27th International Conference on Computational Linguistics. Santa Fe, New Mexico, USA : Association for Computational Linguistics, 2018. p. 1132-1134.

Research output: Contribution in Book/Report/Proceedings - With ISBN/ISSNConference contribution/Paper

Harvard

Moore, A & Rayson, PE 2018, Bringing replication and reproduction together with generalisability in NLP: Three reproduction studies for Target Dependent Sentiment Analysis. in Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguistics, Santa Fe, New Mexico, USA, pp. 1132-1134, Conference on Computational Linguistics, Santa Fe, United States, 20/08/18.

APA

Moore, A., & Rayson, P. E. (2018). Bringing replication and reproduction together with generalisability in NLP: Three reproduction studies for Target Dependent Sentiment Analysis. In Proceedings of the 27th International Conference on Computational Linguistics (pp. 1132-1134). Santa Fe, New Mexico, USA: Association for Computational Linguistics.

Vancouver

Moore A, Rayson PE. Bringing replication and reproduction together with generalisability in NLP: Three reproduction studies for Target Dependent Sentiment Analysis. In Proceedings of the 27th International Conference on Computational Linguistics. Santa Fe, New Mexico, USA: Association for Computational Linguistics. 2018. p. 1132-1134

Author

Moore, Andrew ; Rayson, Paul Edward. / Bringing replication and reproduction together with generalisability in NLP : Three reproduction studies for Target Dependent Sentiment Analysis. Proceedings of the 27th International Conference on Computational Linguistics. Santa Fe, New Mexico, USA : Association for Computational Linguistics, 2018. pp. 1132-1134

Bibtex

@inproceedings{38519329f632435392390bef070a190a,
title = "Bringing replication and reproduction together with generalisability in NLP: Three reproduction studies for Target Dependent Sentiment Analysis",
abstract = "Lack of repeatability and generalisability are two significant threats to continuing scientific development in Natural Language Processing. Language models and learning methods are so complex that scientific conference papers no longer contain enough space for the technical depth required for replication or reproduction. Taking Target Dependent Sentiment Analysis as a case study, we show how recent work in the field has not consistently released code, or described settings for learning methods in enough detail, and lacks comparability and generalisability in train, test or validation data. To investigate generalisability and to enable state of the art comparative evaluations, we carry out the first reproduction studies of three groups of complementary methods and perform the first large-scale mass evaluation on six different English datasets. Reflecting on our experiences, we recommend that future replication or reproduction experiments should always consider a variety of datasets alongside documenting and releasing their methods and published code in order to minimise the barriers to both repeatability and generalisability. We have released our code with a model zoo on GitHub with Jupyter Notebooks to aid understanding and full documentation, and we recommend that others do the same with their papers at submission time through an anonymised GitHub account.",
keywords = "NLP , Sentiment Analysis, Reproducibility",
author = "Andrew Moore and Rayson, {Paul Edward}",
year = "2018",
month = "8",
day = "20",
language = "English",
pages = "1132--1134",
booktitle = "Proceedings of the 27th International Conference on Computational Linguistics",
publisher = "Association for Computational Linguistics",

}

RIS

TY - GEN

T1 - Bringing replication and reproduction together with generalisability in NLP

T2 - Three reproduction studies for Target Dependent Sentiment Analysis

AU - Moore, Andrew

AU - Rayson, Paul Edward

PY - 2018/8/20

Y1 - 2018/8/20

N2 - Lack of repeatability and generalisability are two significant threats to continuing scientific development in Natural Language Processing. Language models and learning methods are so complex that scientific conference papers no longer contain enough space for the technical depth required for replication or reproduction. Taking Target Dependent Sentiment Analysis as a case study, we show how recent work in the field has not consistently released code, or described settings for learning methods in enough detail, and lacks comparability and generalisability in train, test or validation data. To investigate generalisability and to enable state of the art comparative evaluations, we carry out the first reproduction studies of three groups of complementary methods and perform the first large-scale mass evaluation on six different English datasets. Reflecting on our experiences, we recommend that future replication or reproduction experiments should always consider a variety of datasets alongside documenting and releasing their methods and published code in order to minimise the barriers to both repeatability and generalisability. We have released our code with a model zoo on GitHub with Jupyter Notebooks to aid understanding and full documentation, and we recommend that others do the same with their papers at submission time through an anonymised GitHub account.

AB - Lack of repeatability and generalisability are two significant threats to continuing scientific development in Natural Language Processing. Language models and learning methods are so complex that scientific conference papers no longer contain enough space for the technical depth required for replication or reproduction. Taking Target Dependent Sentiment Analysis as a case study, we show how recent work in the field has not consistently released code, or described settings for learning methods in enough detail, and lacks comparability and generalisability in train, test or validation data. To investigate generalisability and to enable state of the art comparative evaluations, we carry out the first reproduction studies of three groups of complementary methods and perform the first large-scale mass evaluation on six different English datasets. Reflecting on our experiences, we recommend that future replication or reproduction experiments should always consider a variety of datasets alongside documenting and releasing their methods and published code in order to minimise the barriers to both repeatability and generalisability. We have released our code with a model zoo on GitHub with Jupyter Notebooks to aid understanding and full documentation, and we recommend that others do the same with their papers at submission time through an anonymised GitHub account.

KW - NLP

KW - Sentiment Analysis

KW - Reproducibility

M3 - Conference contribution/Paper

SP - 1132

EP - 1134

BT - Proceedings of the 27th International Conference on Computational Linguistics

PB - Association for Computational Linguistics

CY - Santa Fe, New Mexico, USA

ER -