A role for conceptual representations in cross-sensory correspondences has been linked to the relative (context-sensitive) mapping of feature values, whereas a role for sensory-perceptual representations has been linked to their absolute (context-insensitive) mapping. Demonstrating the relative nature of the automatic mapping underlying a cross-sensory correspondence therefore offers one way of confirming its conceptual basis. After identifying several prerequisites for relative and absolute mappings, we provide the first compelling demonstration that an automatically induced congruity effect based on a cross-sensory correspondence (i.e., that between haptic size and visual brightness) can be largely contingent on the relative mapping of the two features, thereby implying a conceptual basis for the correspondence. Participants in a speeded classification task were faster to classify a visual stimulus as brighter or darker when this required them to press a hidden response key that, incidentally, was relatively small or big, respectively. Importantly, the same levels of brightness (Experiment 1) and key size (Experiment 2) at different times corresponded to contrasting levels of the other feature depending on the context provided by the alternative stimuli with which they appeared. For example, the same medium key was congruent with a brighter stimulus when paired with a bigger key, but was congruent with a darker stimulus when paired with a smaller key. Reflecting on the broader implications of this finding, it is noted that the involvement of cross-sensory correspondences in some forms of sound symbolism in language also requires the relative coding of stimulus features.