Systems with an effectively non-Hermitian Hamiltonian display an enhanced sensitivity to parametric and dynamic perturbations, which arises from the nonorthogonality of their eigenstates. This enhanced sensitivity can be quantified by the phase rigidity, which mathematically corresponds to the eigenvalue condition number and physically also determines the Petermann factor of quantum noise theory. Here, we derive an exact nonperturbative expression for this sensitivity measure that applies to arbitrary eigenvalue configurations. The expression separates spectral correlations from additional geometric data and retains a simple asymptotic behavior close to exceptional points (EPs) of any order, while capturing the role of additional states in the system. This reveals that such states can have a sizable effect even if they are spectrally well separated and identifies the specific matrix whose elements determine this nonperturbative effect. The employed algebraic approach, which follows the eigenvectors-from-eigenvalues school of thought, also provides direct insights into the geometry of the states near an EP. For instance, it can be used to show that the phase rigidity follows a striking equipartition principle in the quasidegenerate subspace of a system.