We present a method for assigning a statistical significance to detection candidates in targeted searches for continuous gravitational waves from known pulsars, without assuming the detector noise is Gaussian and stationary. We take advantage of the expected Doppler phase modulation of the signal induced by Earth’s orbital motion, as well as the amplitude modulation induced by Earth’s spin, to effectively blind the search to real astrophysical signals from a given location in the sky. We use this “sky shifting” to produce a large number of noise-only data realizations to empirically estimate the background of a search and assign detection significances, in a similar fashion to the use of time slides in searches for compact binaries. We demonstrate the potential of this approach by means of simulated signals, as well as hardware injections into real detector data. In a study of simulated signals in non-Gaussian noise, we find that our method outperforms another common strategy for evaluating detection significance. We thus demonstrate that this and similar techniques have the potential to enable a first confident detection of continuous gravitational waves.