We develop an analytical approach to the spectral form factor and out-of-time ordered correlators in zero-dimensional Brownian models of quantum chaos. The approach expresses these spectral correlations as part of a closed hierarchy of differential equations that can be formulated for all system sizes and in each of the three standard symmetry classes (unitary, orthogonal, and symplectic, as determined by the presence and nature of time-reversal symmetry). The hierarchy applies exactly, and in the same form, to Dyson's Brownian motion and all systems with stochastically emerging basis invariance, where the model-dependent information is subsumed in a single dynamical timescale whose explicit form we also establish. We further verify this universality numerically for the Brownian Sachdev-Ye-Kitaev model, for which we find perfect agreement with the analytical predictions of the symmetry class determined by the number of fermions. This results in a complete analytical description of the spectral correlations and allows us to identify which correlations are universal in a large class of models. Published by the American Physical Society 2025