Accepted author manuscript, 1.56 MB, PDF document
Available under license: CC BY: Creative Commons Attribution 4.0 International License
Final published version
Research output: Contribution to Journal/Magazine › Journal article › peer-review
Research output: Contribution to Journal/Magazine › Journal article › peer-review
}
TY - JOUR
T1 - Quantitative assessment of inconsistency in meta-analysis using decision thresholds with two new indices
AU - Sousa-Pinto, Bernardo
AU - Neumann, Ignacio
AU - Vieira, Rafael José
AU - Bognanni, Antonio
AU - Marques-Cruz, Manuel
AU - Gil-Mata, Sara
AU - Mordue, Simone
AU - Nevill, Clareece
AU - Baio, Gianluca
AU - Whaley, Paul
AU - Schwarzer, Guido
AU - Steele, James
AU - Stewart, Gavin
AU - Schünemann, Holger J
AU - Azevedo, Luís Filipe
PY - 2025/3/11
Y1 - 2025/3/11
N2 - OBJECTIVES: In evidence synthesis, inconsistency is typically assessed visually and with the I 2 and the Q statistics. However, these measures have important limitations (i) if there are few primary studies of small sample sizes or (ii) if there are multiple studies with precise estimates. In addition, with the increasing use of decision thresholds (DT), for example in Grading of Recommendations Assessment, Development and Evaluation evidence to decision (EtD) frameworks, inconsistency judgments can be anchored around DTs. In this article, we developed quantitative measures to assess inconsistency based on DTs. STUDY DESIGN AND SETTING: We developed two measures to quantify inconsistency based on DTs - the decision inconsistency (DI) and the across-studies inconsistency (ASI) indices. The DI and the ASI are based on the distribution of the posterior samples studies' effect sizes (ES) across interpretation categories defined by DTs. We developed these indices for the Bayesian context, followed by a frequentist extension.RESULTS: The DI informs on the overall inconsistency of ESs across interpretation categories, while the ASI quantifies how different studies are compared to each other (in relation to interpretation categories) based on absolute effects. A DI ≥ 50% and an ASI ≥ 25% are suggestive of important inconsistency. We provide an R package (metainc) and a web tool (https://metainc.med.up.pt/) to support the computation of the DI and ASI, including in the context of sensitivity analyses assessing the impact of potential uncertainty in inconsistency.CONCLUSION: The DI and the ASI can contribute to quantitatively assess inconsistency, particularly as DTs are gaining recognition in evidence synthesis and health decision-making.
AB - OBJECTIVES: In evidence synthesis, inconsistency is typically assessed visually and with the I 2 and the Q statistics. However, these measures have important limitations (i) if there are few primary studies of small sample sizes or (ii) if there are multiple studies with precise estimates. In addition, with the increasing use of decision thresholds (DT), for example in Grading of Recommendations Assessment, Development and Evaluation evidence to decision (EtD) frameworks, inconsistency judgments can be anchored around DTs. In this article, we developed quantitative measures to assess inconsistency based on DTs. STUDY DESIGN AND SETTING: We developed two measures to quantify inconsistency based on DTs - the decision inconsistency (DI) and the across-studies inconsistency (ASI) indices. The DI and the ASI are based on the distribution of the posterior samples studies' effect sizes (ES) across interpretation categories defined by DTs. We developed these indices for the Bayesian context, followed by a frequentist extension.RESULTS: The DI informs on the overall inconsistency of ESs across interpretation categories, while the ASI quantifies how different studies are compared to each other (in relation to interpretation categories) based on absolute effects. A DI ≥ 50% and an ASI ≥ 25% are suggestive of important inconsistency. We provide an R package (metainc) and a web tool (https://metainc.med.up.pt/) to support the computation of the DI and ASI, including in the context of sensitivity analyses assessing the impact of potential uncertainty in inconsistency.CONCLUSION: The DI and the ASI can contribute to quantitatively assess inconsistency, particularly as DTs are gaining recognition in evidence synthesis and health decision-making.
KW - Systematic review
KW - GRADE
KW - Inconsistency
KW - Heterogeneity
KW - Meta-analysis
U2 - 10.1016/j.jclinepi.2025.111725
DO - 10.1016/j.jclinepi.2025.111725
M3 - Journal article
C2 - 39955079
VL - 181
JO - Journal of clinical epidemiology
JF - Journal of clinical epidemiology
SN - 0895-4356
M1 - 111725
ER -