We investigate the clustering and halo properties of $\sim 5000$ Ly$\alpha$-selected emission line galaxies (LAEs) from the Slicing COSMOS 4K (SC4K) and from archival NB497 imaging of SA22 split in 15 discrete redshift slices between $z \sim 2.5 - 6$. We measure clustering lengths of $r_0 \sim 3 - 6\ h^{-1}$ Mpc and typical halo masses of $\sim 10^{11}$ M$_\odot$ for our narrowband-selected LAEs with typical $L_{\rm{Ly}\alpha} \sim 10^{42 - 43}$ erg s$^{-1}$. The intermediate band-selected LAEs are observed to have $r_0 \sim 3.5 - 15\ h^{-1}$ Mpc with typical halo masses of $\sim 10^{11 - 12}$ M$_\odot$ and typical $L_{\rm{Ly}\alpha} \sim 10^{43 - 43.6}$ erg s$^{-1}$. We find a strong, redshift-independent correlation between halo mass and Ly$\alpha$ luminosity normalized by the characteristic Ly$\alpha$ luminosity, $L^\star(z)$. The faintest LAEs ($L \sim 0.1\ L^\star(z)$) typically identified by deep narrowband surveys are found in $10^{10}$ M$_\odot$ halos and the brightest LAEs ($L \sim 7\ L^\star(z)$) are found in $\sim 5 \times 10^{12}$ M$_\odot$ halos. A dependency on the rest-frame 1500 \AA~UV luminosity, M$_\rm{UV}$, is also observed where the halo masses increase from $10^{11}$ to $10^{13}$ M$_\odot$ for M$_\rm{UV} \sim -19$ to $-23.5$ mag. Halo mass is also observed to increase from $10^{9.8}$ to $10^{12.3}$ M$_\odot$ for dust-corrected UV star formation rates from $\sim 0.6$ to $10$ M$_\odot$ yr$^{-1}$ and continues to increase up to $10^{13.5}$ M$_\odot$ in halo mass, where the majority of those sources are AGN. All the trends we observe are found to be redshift-independent. Our results reveal that LAEs are the likely progenitors of a wide range of galaxies depending on their luminosity, from dwarf-like, to Milky Way-type, to bright cluster galaxies. LAEs therefore provide unique insight into the early formation and evolution of the galaxies we observe in the local Universe.