TY - JOUR

T1 - A permutation test for assessing the presence of individual differences in treatment effects

AU - Chang, C.

AU - Jaki, T.

AU - Sadiq, M.S.

AU - Kuhlemeier, A.

AU - Feaster, D.

AU - Cole, N.

AU - Lamont, A.

AU - Oberski, D.

AU - Desai, Y.

AU - Lee Van Horn, M.

N1 - The final, definitive version of this article has been published in the Journal, Statistical Methods in Medical Research, 30 (11), 2021, © SAGE Publications Ltd, 2021 by SAGE Publications Ltd at the Statistical Methods in Medical Research page: https://journals.sagepub.com/doi/10.1177/09622802211033640 on SAGE Journals Online: http://journals.sagepub.com/

PY - 2021/11/1

Y1 - 2021/11/1

N2 - An important goal of personalized medicine is to identify heterogeneity in treatment effects and then use that heterogeneity to target the intervention to those most likely to benefit. Heterogeneity is assessed using the predicted individual treatment effects framework, and a permutation test is proposed to establish if significant heterogeneity is present given the covariates and predictive model or algorithm used for predicted individual treatment effects. We first show evidence for heterogeneity in the effects of treatment across an illustrative example data set. We then use simulations with two different predictive methods (linear regression model and Random Forests) to show that the permutation test has adequate type-I error control. Next, we use an example dataset as the basis for simulations to demonstrate the ability of the permutation test to find heterogeneity in treatment effects for a predicted individual treatment effects estimate as a function of both effect size and sample size. We find that the proposed test has good power for detecting heterogeneity in treatment effects when the heterogeneity was due primarily to a single predictor, or when it was spread across the predictors. Power was found to be greater for predictions from a linear model than from random forests. This non-parametric permutation test can be used to test for significant differences across individuals in predicted individual treatment effects obtained with a given set of covariates using any predictive method with no additional assumptions.

AB - An important goal of personalized medicine is to identify heterogeneity in treatment effects and then use that heterogeneity to target the intervention to those most likely to benefit. Heterogeneity is assessed using the predicted individual treatment effects framework, and a permutation test is proposed to establish if significant heterogeneity is present given the covariates and predictive model or algorithm used for predicted individual treatment effects. We first show evidence for heterogeneity in the effects of treatment across an illustrative example data set. We then use simulations with two different predictive methods (linear regression model and Random Forests) to show that the permutation test has adequate type-I error control. Next, we use an example dataset as the basis for simulations to demonstrate the ability of the permutation test to find heterogeneity in treatment effects for a predicted individual treatment effects estimate as a function of both effect size and sample size. We find that the proposed test has good power for detecting heterogeneity in treatment effects when the heterogeneity was due primarily to a single predictor, or when it was spread across the predictors. Power was found to be greater for predictions from a linear model than from random forests. This non-parametric permutation test can be used to test for significant differences across individuals in predicted individual treatment effects obtained with a given set of covariates using any predictive method with no additional assumptions.

KW - Predicted individual treatment effects

KW - heterogeneity in treatment effects

KW - personalized medicine

KW - permutation test

KW - Random Forests

U2 - 10.1177/09622802211033640

DO - 10.1177/09622802211033640

M3 - Journal article

VL - 30

SP - 2369

EP - 2381

JO - Statistical Methods in Medical Research

JF - Statistical Methods in Medical Research

SN - 0962-2802

IS - 11

ER -