Listening difficulties (LiD) in people who have normal audiometry are a widespread but poorly understood form of hearing impairment. Recent research suggests that childhood LiD are cognitive rather than auditory in origin. We examined decoding of sentences using a novel combination of behavioral testing and fMRI with 43 typically developing children and 42 age matched (6–13 years old) children with LiD, categorized by caregiver report (ECLiPS). Both groups had clinically normal hearing. For sentence listening tasks, we found no group differences in fMRI brain cortical activation by increasingly complex speech stimuli that progressed in emphasis from phonology to intelligibility to semantics. Using resting state fMRI, we examined the temporal connectivity of cortical auditory and related speech perception networks. We found significant group differences only in cortical connections engaged when processing more complex speech stimuli. The strength of the affected connections was related to the children’s performance on tests of dichotic listening, speech-in-noise, attention, memory and verbal vocabulary. Together, these results support the novel hypothesis that childhood LiD reflects difficulties in language rather than in auditory or phonological processing.