TY - JOUR

T1 - A simple method for implementing Monte Carlo tests

AU - Ding, Dong

AU - Gandy, Axel

AU - Hahn, Georg

PY - 2016/11/5

Y1 - 2016/11/5

N2 - We consider a statistical test whose p-value can only be approximated using Monte Carlo simulations. We are interested in deciding whether the p-value for an observed data set lies above or below a given threshold such as 5%. We want to ensure that the resampling risk, the probability of the (Monte Carlo) decision being different from the true decision, is uniformly bounded. This article introduces a simple method with this property, the confidence sequence method (CSM). We compare our approach to SIMCTEST (Gandy, 2009, JASA) which also guarantees a uniform bound on the resampling risk. A key advantage of CSM is its simplicity, which comes at the expense of being conservative and leads to seemingly uniformly larger stopping boundaries in CSM compared to SIMCTEST. Since the stopping boundaries in CSM are only marginally larger, we nevertheless conclude that the simplicity of CSM makes it a useful method for practical applications.

AB - We consider a statistical test whose p-value can only be approximated using Monte Carlo simulations. We are interested in deciding whether the p-value for an observed data set lies above or below a given threshold such as 5%. We want to ensure that the resampling risk, the probability of the (Monte Carlo) decision being different from the true decision, is uniformly bounded. This article introduces a simple method with this property, the confidence sequence method (CSM). We compare our approach to SIMCTEST (Gandy, 2009, JASA) which also guarantees a uniform bound on the resampling risk. A key advantage of CSM is its simplicity, which comes at the expense of being conservative and leads to seemingly uniformly larger stopping boundaries in CSM compared to SIMCTEST. Since the stopping boundaries in CSM are only marginally larger, we nevertheless conclude that the simplicity of CSM makes it a useful method for practical applications.

KW - stat.ME

M3 - Journal article

JO - arxiv.org

JF - arxiv.org

ER -