A recently discovered electrophysiological response, the social N400, suggests that we use our language system to track how social partners comprehend language. Listeners show an increased N400 response, when themselves not, only a communicative partner experiences a semantic incongruity. Does the N400 reflect purely semantic or mentalistic computations as well? Do we attribute language comprehension to communicative partners using our semantic systems? In five electrophysiological experiments we identified two subcomponents of the social N400. First, we manipulated the presence-absence of an Observer during object naming: the semantic memory system was activated by the presence of a social partner in addition to semantic predictions for the self. Next, we induced a false belief—and a consequent miscomprehension—in the Observer. Participants showed the social N400, over and above the social presence effect, to labels that were incongruent for the Observer, even though they were congruent for them. This effect appeared only if participants received explicit instructions to track the comprehension of the Observer. These findings suggest that the semantic systems of the brain are not merely sensitive to social information and contribute to the attribution of comprehension, but they appear to be mentalistic in nature.