Virtual Machines (VMs) with Just-In-Time (JIT) compilers are traditionally thought to execute programs in two phases: first the warmup phase determines which parts of a program would most benefit from dynamic compilation and JIT compiles them into machine code; after compilation has occurred, the program is said to be at peak performance. When measuring the performance of JIT compiling VMs, data collected during the warmup phase is generally discarded, placing the focus on peak performance. In this paper we first run a number of small, deterministic benchmarks on a variety of well known VMs, before introducing a rigorous statistical model for determining when warmup has occurred. Across 3 benchmarking machines only 43.3–56.5% of (VM, benchmark) pairs conform to the traditional view of warmup and none of the VMs consistently warms up.