The ability to take others’ perspectives on the self has important psychological implications. Yet the logically and developmentally prior question is how children develop the capacity to take others’ perspectives. We discuss the development of joint attention in infancy as a rudimentary form of perspective taking and critique examples of biological and individualistic approaches to the development of joint attention. As an alternative, we present an activity-based relational perspective according to which infants develop the capacity to coordinate attention with others by differentiating the perspectives of self and other from shared activity. Joint attention is then closely related to language development, which makes further social development possible. We argue that the ability to take the perspective of others on the self gives rise to the possibility of language, rationality and culture.