It has been observed that a large lepton asymmetry, n(L)/s approximate to 0.1, can prevent the Higgs expectation value from going to zero at high temperature, resulting in the non-restoration of the SU(2) x U(1) gauge symmetry and other symmetries. This could allow for the elimination of domain walls and other dangerous topological defects. Here we show that if the reheating temperature after inflation, T-R is sufficiently low then symmetry non-restoration will occur with an asymmetry of the order of the observed baryon asymmetry, n(B)/s approximate to 10(-10). For this to occur T-R less than or similar to 1 GeV is necessary. Remarkably, this happens to be the reheating temperature expected in Affleck-Dine baryogenesis along a d = 6 flat direction of the MSSM. As an example, we show that this can neatly solve the Z(3), domain wall problem of the NMSSM.