When glaciers retreat they expose barren substrates that become colonized by organisms, beginning the process of primary succession. Recent studies reveal that heterotrophic microbial communities occur in newly exposed glacial substrates before autotrophic succession begins. This raises questions about how heterotrophic microbial communities function in the absence of carbon inputs from autotrophs. We measured patterns of soil organic matter development and changes in microbial community composition and carbon use along a 150-year chronosequence of a retreating glacier in the Austrian Alps. We found that soil microbial communities of recently deglaciated terrain differed markedly from those of later successional stages, being of lower biomass and higher abundance of bacteria relative to fungi. Moreover, we found that these initial microbial communities used ancient and recalcitrant carbon as an energy source, along with modern carbon. Only after more than 50 years of organic matter accumulation did the soil microbial community change to one supported primarily by modern carbon, most likely from recent plant production. Our findings suggest the existence of an initial stage of heterotrophic microbial community development that precedes autotrophic community assembly and is sustained, in part, by ancient carbon.