It is shown theoretically and by analog electronic experiment that, in dissipative oscillatory systems for which the frequency of eigenoscillation displays an extremum as a function of energy, the dynamics of nonlinear resonance can differ markedly from the conventional case. Transitions between the conventional and novel types of nonlinear resonance, as parameters vary, correspond to changes in the topology of basins of attraction. With added noise, they can result in drastic changes in fluctuational transition rates between small- and large-amplitude regimes.