In the past decade, many studies have focused on the relationship between emotional valence and vertical spatial positions from a processing perspective. Lakoff and Johnson’s (1980) work on conceptual metaphor has traditionally motivated these investigations, but recent work (Lakens in J Exp Psychol: Learn, Mem Cogn, 38: 726–736, 2012) has suggested that polarity-based perspectives offer an alternative account of response time patterns. We contrasted the predictions of these two theories using a new facial emotion recognition task, in which participants made speeded responses to happy or sad faces on a display, with the spatial location of those faces being manipulated. In three experiments (two-alternative forced choice tasks and a go/no-go task), we found a pattern of responses consistent with a polarity-based account, but inconsistent with key predictions of the conceptual-metaphor account. Overall, congruency effects were observed for positively valenced items, but not for negatively valenced items. These findings demonstrate that polarity effects extend to nonlinguistic stimuli and beyond two-alternative forced choice tasks. We discuss the results in terms of common-coding approaches to task–response mappings.