A split-cache memory architecture is presented which provides efficient architectural support for checkpointing and roll-forward recovery mechanisms in distributed systems. Unlike existing techniques, the approach does not require the use of a discrete stable storage unit or explicit synchronization among the processors. A nonscheduled checkpointing mechanism is presented based on a cache-line replacement policy, instead of the conventionally used periodic checkpoint establishment protocols.