This meta-analysis investigated the association between age-related hearing loss and structural neuroanatomy, specifically changes to gray matter volume. Hearing loss is associated with increased risk of cognitive decline. Hence, understanding the effects of hearing loss in older age on brain health is essential. We reviewed studies which compared older participants with hearing loss (age-related hearing loss: ARHL) to older adults without clinical hearing loss (no-ARHL), on neuroanatomical outcomes, specifically gray matter (GM) volume as measured by magnetic resonance imaging. A total of five studies met the inclusion criteria, three of which were included in an analysis of whole-brain gray matter volume (ARHL group n = 113; no-ARHL group n = 138), and three were included in analyses of lobe-wise gray matter volume (ARHL group n = 139; no-ARHL group n = 162). Effect-size seed-based d mapping software was employed for whole-brain and lobe-wise analysis of gray matter volume. The analysis indicated there was no significant difference between adults with ARHL compared to those with no-ARHL in whole-brain gray matter volume. Due to lacking stereotactic coordinates, the level of gray matter in specific neuroanatomical locations could only be observed at lobe-level. These data indicate that adults with ARHL show increased gray matter atrophy in the temporal lobe only (not in occipital, parietal, or frontal), compared to adults with no-ARHL. The implications for theoretical frameworks of the hearing loss and cognitive decline relationship are discussed in relation to the results.