In this paper we explore the consequences of a heterogeneous immune response in individuals on the evolution of a rapidly mutating virus. We show that several features of the incidence and phylogenetic patterns typical of influenza A may be understood in this framework. In our model, limited diversity and rapid drift of the circulating viral strains result from the interplay of two interacting subpopulations with different types of immune response, narrow or broad, upon infection. The subpopulation with the narrow immune response acts as a reservoir where consecutive mutations escape immunity and can persist. Strains with a number of accumulated mutations escape immunity in the other subpopulation as well, causing larger epidemic peaks in the whole population, and reducing strain diversity. Overall, our model produces a modulation of epidemic peak heights and patterns of antigenic drift consistent with reported observations, suggesting an underlying mechanism for the evolutionary epidemiology of influenza, in particular, and other infectious diseases, more generally.