We have investigated the photoluminescence (PL) of self-assembled InAs/GaAs quantum dots (QDs) in high magnetic fields of up to 50 T and as a function of temperature. Our data clearly indicate that two different mechanisms are at work. At low temperatures (T < 80 K), the zero-field PL is increasingly dominated by lower energy dots. High-field measurements however demonstrate that these dots are larger in size only in the growth direction. At temperatures above 100 K, a strong decrease of the PL peak energy shift with field is observed, while the zero-field PL is characterized by a redshift according to the changes in the bandgap. We discuss these contradictory observations in terms of a phenomenon that we call field-assisted enhancement of the QD barrier potential. Since this effect is much stronger for small high-energy QDs, the latter progressively dominate the PL emission when temperature and magnetic field are increased.