Infants employ sophisticated mechanisms to acquire their first language, including some that rely on taking the perspective of adults as speakers or listeners. When do infants first show awareness of what other people understand? We tested 14-month-old infants in two experiments measuring event-related potentials. In Experiment 1, we established that infants produce the N400 effect, a brain signature of semantic violations, in a live object naming paradigm in the presence of an adult observer. In Experiment 2, we induced false beliefs about the labelled objects in the adult observer to test whether infants keep track of the other person’s comprehension. The results revealed that infants reacted to the semantic incongruity heard by the other as if they encountered it themselves: they exhibited an N400-like response, even though labels were congruous from their perspective. This finding demonstrates that infants track the linguistic understanding of social partners.