We analyze the colors and sizes of 32 quiescent (UVJ-selected) galaxies with strong Balmer absorption (EW(Hδ) ≥ 4 ̊A) at z ∼ 0.8 drawn from DR2 of the LEGA-C survey to test the hypothesis that these galaxies experienced compact, central starbursts before quenching. These recently quenched galaxies, usually referred to as post-starburst galaxies, span a wide range of colors and we find a clear correlation between color and half-light radius, such that bluer galaxies are smaller. We build simple toy models to explain this correlation: a normal star-forming disk plus a central, compact starburst component. Bursts with exponential decay timescale of ∼ 100 Myr that produce ∼ 10% to more than 100% of the pre-existing masses can reproduce the observed correlation. More significant bursts also produce bluer and smaller descendants. Our findings imply that when galaxies shut down star formation rapidly, they generally had experienced compact, starburst events and that the large, observed spread in sizes and colors mostly reflects a variety of burst strengths. Recently quenched galaxies should have younger stellar ages in the centers; multi-wavelength data with high spatial resolution are required to reveal the age gradient. Highly dissipative processes should be responsible for this type of formation history. While determining the mechanisms for individual galaxies is challenging, some recently quenched galaxies show signs of gravitational interactions, suggesting that mergers are likely an important mechanism in triggering the rapid shut-down of star-formation activities at z ∼ 0.8.