We analyze the origin of the intrinsic timing jitter in superconducting nanowire single photon detectors (SNSPDs) in terms of fluctuations in the latency of the detector response, which is determined by the microscopic physics of the photon detection process. We demonstrate that fluctuations in the physical parameters which determine the latency give rise to the intrinsic timing jitter. We develop a general description of latency by introducing the explicit time dependence of the internal detection efficiency. By considering the dynamic Fano fluctuations together with static spatial inhomogeneities, we study the details of the connection between latency and timing jitter. We develop both a simple phenomenological model and a more general microscopic model of detector latency and timing jitter based on the solution of the generalized time-dependent Ginzburg-Landau equations for the 1D hotbelt geometry. While the analytical model is sufficient for qualitative interpretation of recent data, the general approach establishes the framework for a quantitative analysis of detector latency and the fundamental limits of intrinsic timing jitter. These theoretical advances can be used to interpret the results of recent experiments measuring the dependence of detection latency and timing jitter on photon energy to the few-picosecond level.