We present a comprehensive analysis of the ability of current stellar population models to reproduce the optical (ugriz) and near-infrared (JHK) colours of a small sample of well-studied nearby elliptical and S0 galaxies. We find broad agreement between the ages and metallicities derived using different population models, although different models show different systematic deviations from the measured broad-band fluxes. Although it is possible to constrain simple stellar population models to a well-defined area in age-metallicity space, there is a clear degeneracy between these parameters even with such a full range of precise colours. The precision to which age and metallicity can be determined independently, using only broad-band photometry with realistic errors, is Δ[Fe/H] ≃ 0.18 and Δlog Age ≃ 0.25. To constrain the populations and therefore the star formation history further, it will be necessary to combine broad-band optical-IR photometry with either spectral line indices, or else photometry at wavelengths outside this range.