Home > Research > Publications & Outputs > FIESTA

Research

Associated organisational units

Electronic data

  • 1906.12230v1

    Accepted author manuscript, 389 KB, PDF document

Links

Keywords

View graph of relations

FIESTA: Fast IdEntification of State-of-The-Art models using adaptive bandit algorithms

Research output: Contribution in Book/Report/Proceedings - With ISBN/ISSN › Conference contribution/Paper › peer-review

Published

Standard

FIESTA: Fast IdEntification of State-of-The-Art models using adaptive bandit algorithms. / Moss, Henry B.; Moore, Andrew; Leslie, David S. et al.
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg PA: Association for Computational Linguistics, 2019. p. 2920-2930.

Research output: Contribution in Book/Report/Proceedings - With ISBN/ISSN › Conference contribution/Paper › peer-review

Harvard

Moss, HB, Moore, A, Leslie, DS & Rayson, P 2019, FIESTA: Fast IdEntification of State-of-The-Art models using adaptive bandit algorithms. in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Stroudsburg PA, pp. 2920-2930. <https://www.aclweb.org/anthology/P19-1281>

APA

Moss, H. B., Moore, A., Leslie, D. S., & Rayson, P. (2019). FIESTA: Fast IdEntification of State-of-The-Art models using adaptive bandit algorithms. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 2920-2930). Association for Computational Linguistics. https://www.aclweb.org/anthology/P19-1281

Vancouver

Moss HB, Moore A, Leslie DS, Rayson P. FIESTA: Fast IdEntification of State-of-The-Art models using adaptive bandit algorithms. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg PA: Association for Computational Linguistics. 2019. p. 2920-2930

Author

Moss, Henry B. ; Moore, Andrew ; Leslie, David S. et al. / FIESTA : Fast IdEntification of State-of-The-Art models using adaptive bandit algorithms. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg PA : Association for Computational Linguistics, 2019. pp. 2920-2930

Bibtex

@inproceedings{e4af638571fb4b6d8659f4610c7fa5ed,
title = "FIESTA: Fast IdEntification of State-of-The-Art models using adaptive bandit algorithms",
abstract = "We present FIESTA, a model selection approach that significantly reduces the computational resources required to reliably identify state-of-the-art performance from large collections of candidate models. Despite being known to produce unreliable comparisons, it is still common practice to compare model evaluations based on single choices of random seeds. We show that reliable model selection also requires evaluations based on multiple train-test splits (contrary to common practice in many shared tasks). Using bandit theory from the statistics literature, we are able to adaptively determine appropriate numbers of data splits and random seeds used to evaluate each model, focusing computational resources on the evaluation of promising models whilst avoiding wasting evaluations on models with lower performance. Furthermore, our user-friendly Python implementation produces confidence guarantees of correctly selecting the optimal model. We evaluate our algorithms by selecting between 8 target-dependent sentiment analysis methods using dramatically fewer model evaluations than current model selection approaches. ",
keywords = "Machine Learning, Model selection, Multi-armed bandit, NLP",
author = "Moss, {Henry B.} and Andrew Moore and Leslie, {David S.} and Paul Rayson",
year = "2019",
month = jul,
day = "30",
language = "English",
pages = "2920--2930",
booktitle = "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics",
publisher = "Association for Computational Linguistics",

}

RIS

TY - GEN

T1 - FIESTA

T2 - Fast IdEntification of State-of-The-Art models using adaptive bandit algorithms

AU - Moss, Henry B.

AU - Moore, Andrew

AU - Leslie, David S.

AU - Rayson, Paul

PY - 2019/7/30

Y1 - 2019/7/30

N2 - We present FIESTA, a model selection approach that significantly reduces the computational resources required to reliably identify state-of-the-art performance from large collections of candidate models. Despite being known to produce unreliable comparisons, it is still common practice to compare model evaluations based on single choices of random seeds. We show that reliable model selection also requires evaluations based on multiple train-test splits (contrary to common practice in many shared tasks). Using bandit theory from the statistics literature, we are able to adaptively determine appropriate numbers of data splits and random seeds used to evaluate each model, focusing computational resources on the evaluation of promising models whilst avoiding wasting evaluations on models with lower performance. Furthermore, our user-friendly Python implementation produces confidence guarantees of correctly selecting the optimal model. We evaluate our algorithms by selecting between 8 target-dependent sentiment analysis methods using dramatically fewer model evaluations than current model selection approaches.

AB - We present FIESTA, a model selection approach that significantly reduces the computational resources required to reliably identify state-of-the-art performance from large collections of candidate models. Despite being known to produce unreliable comparisons, it is still common practice to compare model evaluations based on single choices of random seeds. We show that reliable model selection also requires evaluations based on multiple train-test splits (contrary to common practice in many shared tasks). Using bandit theory from the statistics literature, we are able to adaptively determine appropriate numbers of data splits and random seeds used to evaluate each model, focusing computational resources on the evaluation of promising models whilst avoiding wasting evaluations on models with lower performance. Furthermore, our user-friendly Python implementation produces confidence guarantees of correctly selecting the optimal model. We evaluate our algorithms by selecting between 8 target-dependent sentiment analysis methods using dramatically fewer model evaluations than current model selection approaches.

KW - Machine Learning

KW - Model selection

KW - Multi-armed bandit

KW - NLP

M3 - Conference contribution/Paper

SP - 2920

EP - 2930

BT - Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

PB - Association for Computational Linguistics

CY - Stroudsburg PA

ER -