Lyman alpha emitters (LAEs) and Lyman break galaxies (LBGs) represent the most common groups of star-forming galaxies at high z, and the differences between their inherent stellar populations (SPs) are a key factor in understanding early galaxy formation and evolution. We have run a set of SP burst-like models for a sample of 1558 sources at 3.4 < z < 6.8 from the Survey for High-z Absorption Red and Dead Sources (SHARDS) over the GOODS-N field. This work focuses on the differences between the three different observational subfamilies of our sample: LAE–LBGs, no-Ly α LBGs, and pure LAEs. Single and double SP synthetic spectra were used to model the spectral energy distributions, adopting a Bayesian information criterion to analyze under which situations a second SP is required. We find that the sources are well modelled using a single SP in ∼79 per cent of the cases. The best models suggest that pure LAEs are typically young low-mass galaxies (⁠t∼26⁺⁴¹₋₂₅ Myr; M_star∼5.6^+12.0_−5.5×10⁸ M_⊙⁠), undergoing one of their first bursts of star formation. On the other hand, no-Ly _α LBGs require older SPs (t ∼ 71 ± 12 Myr), and they are substantially more massive (M_star ∼ 3.5 ± 1.1 × 10⁹ M⊙). LAE–LBGs appear as the subgroup that more frequently needs the addition of a second SP, representing an old and massive galaxy caught in a strong recent star-forming episode. The relative number of sources found from each subfamily at each z supports an evolutionary scenario from pure LAEs and single SP LAE–LBGs to more massive LBGs. Stellar mass functions are also derived, finding an increase of M* with cosmic time and a possible steepening of the low-mass slope from z ∼ 6 to z ∼ 5 with no significant change to z ∼ 4. Additionally, we have derived the SFR–M_star relation, finding an SFR∝M^β_star behaviour with negligible evolution from z ∼ 4 to z ∼ 6.