We investigate the change in mean stellar population age and metallicity ([Z/H]) scaling relations for quiescent galaxies from intermediate redshift ($0.60\leq z\leq0.76$) using the LEGA-C Survey, to low redshift ($0.014\leq z\leq0.10$) using the SAMI Galaxy Survey. We find that, similarly to their low-redshift counterparts, the stellar metallicity of quiescent galaxies at $0.60\leq z\leq 0.76$ closely correlates with $M_*/R_\mathrm{e}$ (a proxy for the gravitational potential or escape velocity), in that galaxies with deeper potential wells are more metal-rich. This supports the hypothesis that the relation arises due to the gravitational potential regulating the retention of metals, by determining the escape velocity required by metal-rich stellar and supernova ejecta to escape the system and avoid being recycled into later stellar generations. On the other hand, we find no correlation between stellar age and $M_*/R_\mathrm{e}^2$ (stellar mass surface density $\Sigma$) in the LEGA-C sample, despite this being a strong relation at low redshift. We consider this change in the age--$\Sigma$ relation in the context of the redshift evolution of the star-forming and quiescent populations in the mass--size plane, and find our results can be explained as a consequence of galaxies forming more compactly at higher redshifts, and remaining compact throughout their evolution. Furthermore, galaxies appear to quench at a characteristic surface density that decreases with decreasing redshift. The $z\sim 0$ age--$\Sigma$ relation is therefore a result of building up the quiescent and star-forming populations with galaxies that formed at a range of redshifts and so a range of surface densities.