The presence of stress raisers in bearing steels is known to be highly detrimental to rolling contact fatigue performance. In VIM-VAR (vacuum induction melted-vacuum arc remelted) M50, primary carbides inherited from the solidification stage can develop damage in the form of butterflies, possibly leading to failure by spalling. A ball-on-rod rolling contact fatigue test rig was used to induce damage under different loading conditions. The butterfly distribution as well as the spatial variation of number, size, and orientation relative to the racetrack surface was investigated. The effect of the load on the butterfly formation is also detailed. Finally, the nucleation and growth of butterflies was observed throughout a series of tests suspended after different number of stress cycles. The trends unveiled in this paper could highly benefit the development and validation of models aiming at predicting the effect of stress raisers on the performance of materials subjected to rolling contact fatigue.