We have studied the magnetic field (< 50 T) dependence of the photoluminescence (PL) of self-assembled InAs/GaAs quantum dots as a function of temperature (T). As the temperature is raised from 4.2 up to 80 K thermal redistribution causes the PL to be increasingly dominated by dots with a lower PL energy. Magneto-PL demonstrates that these low energy dots are larger in size only in the growth direction, and not in the plane of the sample. At high temperatures (T > 100 K) a different physical phenomenon emerges: we see an anomalous decrease of the PL shift in magnetic field, which is attributed to field-enhancement of the quantum dot barrier potential. This mechanism strongly favors excitons in small dots with a weak PL shift in magnetic field, hence laterally smaller dots increasingly dominate the PL at high temperatures and high fields.