Two experiments were performed to investigate the principles by which emotional stimuli are classified on the dimensions of valence and arousal. In Experiment 1, a large sample of healthy participants rated emotional stimuli according to both broad dimensions. Hierarchical cluster analyses performed on these ratings revealed that stimuli were clustered according to their semantic content at the beginning of the agglomerative process. Example semantic themes include food, violence, nudes, death, and objects. Importantly, this pattern occurred in a parallel fashion for ratings on both dimensions. In Experiment 2, we investigated if the same semantic clusters were differentiated at the neurophysiological level. Intracerebral EEG was recorded from 18 patients with intractable epilepsy who viewed the same set of stimuli. Not only did electrocortical responses differentiate between these data-defined semantic clusters, they converged with the behavioral measurements to highlight the importance of categories associated with survival and reproduction. These findings provide strong evidence that the semantic content of affective material influences their classification along the broad dimensions of valence and arousal, and this principle of categorization exerts an effect on the evoked emotional response. Future studies should consider data-driven techniques rather than normative ratings to identify more specific, semantically related emotional images.