In pursuit of efficiency, we propose a new way to construct least squares estimators, as the minimizers of an augmented objective function that takes explicitly into account the variability of the error term and the resulting uncertainty, as well as the possible existence of heteroskedasticity. We initially derive an infeasible estimator which we then approximate using Ordinary Least Squares (OLS) residuals from a first-step regression to obtain the feasible “HOLS” estimator. This estimator has negligible bias, is consistent and outperforms OLS in terms of finite-sample Mean Squared Error, but also in terms of asymptotic efficiency, under all skedastic scenarios, including homoskedasticity. Analogous efficiency gains are obtained for the case of Instrumental Variables estimation. Theoretical results are accompanied by simulations that support them.