Recent neuroimaging research has shown that perceptual and conceptual processing share a common, modality-specific neural substrate, while work on modality switching costs suggests that they share some of the same attentional mechanisms. In three experiments, we employed a modality detection task that displayed modality-specific object properties (e.g., unimodal shrill, warm, crimson, or bimodal jagged, fluffy) for extremely short display times and asked participants to judge whether each property corresponded to a particular target modality (e.g., auditory, gustatory, tactile, olfactory, visual). Results show that perceptual and conceptual processing share a tactile disadvantage: people are less accurate in detecting expected information regarding the sense of touch than any other modality. These findings support embodied assertions that the conceptual system uses the perceptual system for the purposes of representation. We suggest that the tactile disadvantage emerges for linguistic stimuli due to the evolutionary adaptation of endogenous attention to incoming sensory stimuli.