Early life stress (ELS) is known to have considerable influence on brain development and affective functioning. Previous studies in clinical populations have shown that hippocampus and amygdala, two central structures of limbic emotion processing circuits, are predominantly affected by early stress exposure. Given the inconsistent findings on ELS-related effects in healthy populations and the associations of ELS and affective functioning, the question arises which additional emotion-relevant variables need to be considered to better understand the effects of ELS. We, therefore, investigated the volume of hippocampus and amygdala in 25 high alexithymic (h-ALEX) and 25 low alexithymic (l-ALEX) individuals, which were matched with regard to ELS, but significantly differed in their degree of emotional functioning. Volumetric analyses were performed using FSL-FIRST, a method to automatically segment subcortical structures on T1-weighted magnetic resonance images. Alexithymia was assessed using the Toronto Alexithymia Scale and Bermond-Vorst Alexithymia Questionnaire. ELS was assessed by Childhood Trauma Questionnaire (CTQ) and Early Trauma Inventory. Our data showed that ELS was negatively associated with right hippocampus volume in h-ALEX individuals, while there was no such association in the l-ALEX group. Furthermore, ELS was positively associated with left amygdala volume in l-ALEX individuals, but not in individuals with high levels of alexithymia. The present study emphasizes a substantial relationship between intrapersonal factors, such as alexithymia and neural alterations related to the experience of ELS. Longitudinal study designs are necessary to pursue the question of how emotional abilities interact with individual adaptations to early stress exposure on the neural level.