Studies of the internal structure of asteroids, which are crucial for understanding their impact history and for hazard mitigation, appear to be in conflict for the S-type asteroids, Eros, Gaspra, and Ida. Spacecraft images and geophysical data show that they are fractured, coherent bodies, whereas models of catastrophic asteroidal impacts, family and satellite formation, and studies of asteroid spin rates, and other diverse properties of asteroids and planetary craters suggest that such asteroids are gravitationally bound aggregates of rubble. These conflicting views may be reconciled if 10–50 km S-type asteroids formed as rubble piles, but were later consolidated into coherent bodies. Many meteorites are breccias that testify to a long history of impact fragmentation and consolidation by alteration, metamorphism, igneous and impact processes. Ordinary chondrites, which are the best analogs for S asteroids, are commonly breccias. Some may have formed in cratering events, but many appear to have formed during disruption and reaccretion of their parent asteroids. Some breccias were lithified during metamorphism, and a few were lithified by injected impact melt, but most are regolith and fragmental breccias that were lithified by mild or moderate shock, like their lunar analogs. Shock experiments show that porous chondritic powders can be consolidated during mild shock by small amounts of silicate melt that glues grains together, and by friction and pressure welding of silicate and metallic Fe,Ni grains. We suggest that the same processes that converted impact debris into meteorite breccias also consolidated asteroidal rubble. Internal voids would be partly filled with regolith by impact-induced seismic shaking. Consolidation of this material beneath large craters would lithify asteroidal rubble to form a more coherent body. Fractures on Ida that were created by antipodal impacts and are concentrated in and near large craters, and small positive gravity anomalies associated with the Psyche and Himeros craters on Eros, are consistent with this concept. Spin data suggest that smaller asteroids 0.6–6 km in size are unconsolidated rubble piles. C-type asteroids, which are more porous than S-types, and their analogs, the volatile-rich carbonaceous chondrites, were probably not lithified by shock.