In the sociology of the body, the analysis of physiognomy is a neglected topic. The idea that one can judge the character of another from their facial or bodily characteristics is a pervasive phenomenon. However, its historical and cultural spread does not entail that we inevitably tie it to notions of human essence. This study focuses upon a particular periodic resurgence of physiognomic discourse in the West, at the end of the 18th and the entirety of the 19th century. In contrast to previous arguments, I argue that physiognomic discourse was able to exploit 19th-century phrenology as a conduit for its own perpetuation. I point out that the perception of the other that physiognomy promotes is largely based upon an atemporal view of the body. I suggest that this physiognomic perception remains an entrenched but changeable component in contemporary relations between self and other.