Phase-locked multi-terahertz transients map out the full photonic bandstructure of a one-dimensional photonic crystal while a 12-fs control pulse activates ultrastrong interaction on a sub-cycle time scale with quantized electronic transitions in semiconductor quantum wells. We trace the build-up dynamics of a large vacuum Rabi splitting and observe an unexpected asymmetric formation of the upper and lower polariton bands. The pronounced flattening of the photonic bands causes a slow-down of the group velocity by one order of magnitude on the time scale of the oscillation period of light.