Developmental studies have shown that infants exploit ordinal information to extract and generalize repetition-based rules from a sequence of items. Within the visual modality, this ability is constrained by the spatial layout within which items are delivered given that a left-to-right orientation boosts infants’ rule learning, whereas a right-to-left orientation hinders this ability. Infants’ rule learning operates across different domains and can also be transferred across modalities when learning is triggered by speech. However, no studies have investigated whether the transfer of rule learning occurs across different domains when language is not involved. Using a visual habituation procedure, we tested 7-month-old infants’ ability to extract rule-like patterns from numerical sequences and generalize them to non-numerical sequences of visual shapes and whether this ability is affected by the spatial orientation. Infants were first habituated to left-to-right or right-to-left oriented numerical sequences instantiating an ABB rule and were then tested with the familiar rule instantiated across sequences of single geometrical shapes and a novel (ABA) rule. Results showed a transfer of learning from number to visual shapes for left-to-right oriented sequences but not for right-to-left oriented ones (Experiment 1) even when the direction of the numerical change (increasing vs. decreasing) within the habituation sequences violated a small–left/large–right number–space association (Experiment 2). These results provide the first demonstration that visual rule learning mechanisms in infancy operate at a high level of abstraction and confirm earlier findings that left-to-right oriented directional cues facilitate infants’ representation of order.