Understanding galaxy formation and evolution requires studying the interplay between the growth of galaxies and the growth of their black holes across cosmic time. Here we explore a sample of Ha-selected star-forming galaxies from the HiZELS survey and use the wealth of multi-wavelength data in the COSMOS field (X-rays, far-infrared and radio) to study the relative growth rates between typical galaxies and their central supermassive black holes, from z=2.23 to z=0. Typical star-forming galaxies at z~1-2 have black hole accretion rates (BHARs) of 0.001-0.01 Msun/yr and star formation rates (SFRs) of ~10-40 Msun/yr, and thus grow their stellar mass much quicker than their black hole mass (~3.3 orders of magnitude faster). However, ~3% of the sample (the sources detected directly in the X-rays) show a significantly quicker growth of the black hole mass (up to 1.5 orders of magnitude quicker growth than the typical sources). BHARs fall from z=2.23 to z=0, with the decline resembling that of star formation rate density or the typical SFR. We find that the average black hole to galaxy growth (BHAR/SFR) is approximately constant for star-forming galaxies in the last 11 Gyrs. The relatively constant BHAR/SFR suggests that these two quantities evolve equivalently through cosmic time and with practically no delay between the two.