Polarized cross-correlation spectroscopy on a quantum dot charged with a single hole shows the sequential emission of photons with common circular polarization. This effect is visible without magnetic field, but becomes more pronounced as the field along the quantization axis is increased. We interpret the data in terms of electron dephasing in the X+ state caused by the Overhauser field of nuclei in the dot. We predict the correlation time scale can be increased by accelerating the emission rate with cavity QED.