Understanding the processes that drive the formation of black holes (BHs) is a key topic in observational cosmology. While the observed MBH–MBulge correlation in bulge-dominated galaxies is thought to be produced by major mergers, the existence of an MBH–M⋆ relation, across all galaxy morphological types, suggests that BHs may be largely built by secular processes. Recent evidence that bulge-less galaxies, which are unlikely to have had significant mergers, are offset from the MBH–MBulge relation, but lie on the MBH–M⋆ relation, has strengthened this hypothesis. Nevertheless, the small size and heterogeneity of current data sets, coupled with the difficulty in measuring precise BH masses, make it challenging to address this issue using empirical studies alone. Here, we use Horizon-AGN, a cosmological hydrodynamical simulation to probe the role of mergers in BH growth over cosmic time. We show that (1) as suggested by observations, simulated bulge-less galaxies lie offset from the main MBH–MBulge relation, but on the MBH–M⋆ relation, (2) the positions of galaxies on the MBH–M⋆ relation are not affected by their merger histories, and (3) only ∼35 per cent of the BH mass in today's massive galaxies is directly attributable to merging – the majority (∼65 per cent) of BH growth, therefore, takes place gradually, via secular processes, over cosmic time.