Fluctuations in a periodically driven oscillator whose frequency of eigenoscillation depends nonmonotonically on its energy are investigated. Their character, and their evolution with the amplitude or frequency of the driving force, can be very different from those of oscillators with a monotonic dependence: the distinction reflects a transition between conventional, and a novel type of, nonlinear resonance.