Understanding the socioeconomic gradient in physical inactivity is essential for effective health promotion. This paper exploits data on over one million individuals (1,002,216 people aged 16 and over) in England drawn from the Active People Survey (2004–11). We identify the separate associations between a variety of measures of physical inactivity with education and household income. We find high levels of physical inactivity. Further, both education and household income are strongly associated with inactivity even when controlling for local area deprivation, the availability of physical recreation and sporting facilities, the local weather and regional geography. Moreover, the gap in inactivity between those living in high and low income households is already evident in early adult life and increases up until about age 85. Overall, these results suggest that England is building up a large future health problem and one that is heavily socially graded.