The quantum-limited linewidth of a laser cavity is enhanced above the Schawlow-Townes value by the Petermann factor K, due to the non-orthogonality of the cavity modes. The average Petermann factor <K> in an ensemble of cavities with chaotic scattering and broken time-reversal symmetry is calculated non-perturbatively using random-matrix theory and the supersymmetry technique, as a function of the decay rate Gamma of the lasing mode and the number of scattering channels N. We find for N>>1 that for typical values of Gamma the average Petermann factor <K> propto N >> 1 is parametrically larger than unity.