A search for the nonresonant production of Higgs boson pairs in the HH→bb¯τ+τ− channel is performed using 140 fb−1 of proton-proton collisions at a center-of-mass energy of 13 TeV recorded by the ATLAS detector at the CERN Large Hadron Collider. The analysis strategy is optimized to probe anomalous values of the Higgs boson self-coupling modifier κλ and of the quartic HHVV (V=W,Z) coupling modifier κ2V. No significant excess above the expected background from Standard Model processes is observed. An observed (expected) upper limit μHH