Background: People with an intellectual impairment experience high levels of social and health inequalities. We investigated the impact of COVID-19 on the physical and mental health of people with intellectual impairment, controlling for demographic risk, socio-economic circumstances and pre-pandemic health levels. Method: Data were drawn from two UK birth cohorts that surveyed their participants on the impact of COVID-19 in May 2020: the Millennium Cohort Study (20-year-old participants) and the British Cohort Survey (50-year-old participants). Health outcomes (COVID-19 infection, COVID-19 symptoms, self-reported physical health, mental health, health service use and impact on health behaviours) were compared between people with and without intellectual impairment, adjusting for gender and ethnicity. Differences were further adjusted for self-reported health pre-pandemic and the impact of COVID-19 on socio-economic circumstances. Results: Controlling for gender and ethnicity, poor health was reported less often by younger adults [relative risks (RR): 0.44 95% confidence interval (CI) 0.23, 0.86] and more often by older adults (RR: 1.99 95% CI 1.45, 2.73) with intellectual impairment compared with peers. Older adults were also more likely to experience fever and loss of taste/smell. Adjusting for pre-pandemic health and socio-economic circumstances eliminated some differences in the older cohort, but not in the younger one. Conclusion: In young adulthood, the impact of COVID-19 on health outcomes was not negative. The pattern was reversed in later adulthood, although differences were mostly eliminated after adjustment suggesting a socio-economic and age gradient of COVID-19 impacts on intellectual impairment.