A search for nonresonant Higgs boson pair production in the bb¯γγ final state is performed using 140 fb−1 of proton-proton collisions at a centre-of-mass energy of 13 TeV recorded by the ATLAS detector at the CERN Large Hadron Collider. This analysis supersedes and expands upon the previous nonresonant ATLAS results in this final state based on the same data sample. The analysis strategy is optimised to probe anomalous values not only of the Higgs (H) boson self-coupling modifier κλ but also of the quartic HHVV (V = W, Z) coupling modifier κ2V. No significant excess above the expected background from Standard Model processes is observed. An observed upper limit μHH < 4.0 is set at 95% confidence level on the Higgs boson pair production cross-section normalised to its Standard Model prediction. The 95% confidence intervals for the coupling modifiers are −1.4 < κλ < 6.9 and −0.5 < κ2V < 2.7, assuming all other Higgs boson couplings except the one under study are fixed to the Standard Model predictions. The results are interpreted in the Standard Model effective field theory and Higgs effective field theory frameworks in terms of constraints on the couplings of anomalous Higgs boson (self-)interactions.