Studies that quantify speech tempo tend to use one of various available rate measures. The relationship between these measures and perceived tempo as elicited through listening experiments remains poorly understood. We assess how canonical and surface syllable and phone rates compare in terms of their mapping to listeners' tempo ratings. Native speakers of English rated short stretches of spontaneous speech for tempo; we modelled ratings for stimulus samples in which correlations between canonical and surface rates were low. Our findings suggest that listeners' ratings map most straightforwardly to canonical rate for syllables, but to surface rates for phones. We find little evidence of global tempo affecting the mappings, and consistent effects of stimulus duration. We discuss implications for the role of phoneme restoration in temporal processing.