Final published version
Licence: CC BY: Creative Commons Attribution 4.0 International License
Research output: Contribution to Journal/Magazine › Journal article › peer-review
Research output: Contribution to Journal/Magazine › Journal article › peer-review
}
TY - JOUR
T1 - Audio-based AI classifiers show no evidence of improved COVID-19 screening over simple symptoms checkers
AU - Coppock, H.
AU - Nicholson, G.
AU - Kiskin, I.
AU - Koutra, V.
AU - Baker, K.
AU - Budd, J.
AU - Payne, R.
AU - Karoune, E.
AU - Hurley, D.
AU - Titcomb, A.
AU - Egglestone, S.
AU - Tendero Cañadas, A.
AU - Butler, L.
AU - Jersakova, R.
AU - Mellor, J.
AU - Patel, S.
AU - Thornley, T.
AU - Diggle, P.
AU - Richardson, S.
AU - Packham, J.
AU - Schuller, B.W.
AU - Pigoli, D.
AU - Gilmour, S.
AU - Roberts, S.
AU - Holmes, C.
PY - 2024/2/7
Y1 - 2024/2/7
N2 - Recent work has reported that respiratory audio-trained AI classifiers can accurately predict SARS-CoV-2 infection status. However, it has not yet been determined whether such model performance is driven by latent audio biomarkers with true causal links to SARS-CoV-2 infection or by confounding effects, such as recruitment bias, present in observational studies. Here we undertake a large-scale study of audio-based AI classifiers as part of the UK government’s pandemic response. We collect a dataset of audio recordings from 67,842 individuals, with linked metadata, of whom 23,514 had positive polymerase chain reaction tests for SARS-CoV-2. In an unadjusted analysis, similar to that in previous works, AI classifiers predict SARS-CoV-2 infection status with high accuracy (ROC–AUC = 0.846 [0.838–0.854]). However, after matching on measured confounders, such as self-reported symptoms, performance is much weaker (ROC–AUC = 0.619 [0.594–0.644]). Upon quantifying the utility of audio-based classifiers in practical settings, we find them to be outperformed by predictions on the basis of user-reported symptoms. We make best-practice recommendations for handling recruitment bias, and for assessing audio-based classifiers by their utility in relevant practical settings. Our work provides insights into the value of AI audio analysis and the importance of study design and treatment of confounders in AI-enabled diagnostics.
AB - Recent work has reported that respiratory audio-trained AI classifiers can accurately predict SARS-CoV-2 infection status. However, it has not yet been determined whether such model performance is driven by latent audio biomarkers with true causal links to SARS-CoV-2 infection or by confounding effects, such as recruitment bias, present in observational studies. Here we undertake a large-scale study of audio-based AI classifiers as part of the UK government’s pandemic response. We collect a dataset of audio recordings from 67,842 individuals, with linked metadata, of whom 23,514 had positive polymerase chain reaction tests for SARS-CoV-2. In an unadjusted analysis, similar to that in previous works, AI classifiers predict SARS-CoV-2 infection status with high accuracy (ROC–AUC = 0.846 [0.838–0.854]). However, after matching on measured confounders, such as self-reported symptoms, performance is much weaker (ROC–AUC = 0.619 [0.594–0.644]). Upon quantifying the utility of audio-based classifiers in practical settings, we find them to be outperformed by predictions on the basis of user-reported symptoms. We make best-practice recommendations for handling recruitment bias, and for assessing audio-based classifiers by their utility in relevant practical settings. Our work provides insights into the value of AI audio analysis and the importance of study design and treatment of confounders in AI-enabled diagnostics.
U2 - 10.1038/s42256-023-00773-8
DO - 10.1038/s42256-023-00773-8
M3 - Journal article
VL - 6
SP - 229
EP - 242
JO - Nature Machine Intelligence
JF - Nature Machine Intelligence
SN - 2522-5839
ER -