Searches for both resonant and nonresonant Higgs boson pair production are performed in the hh→bbττ, γγWW∗ final states using 20.3  fb−1 of pp collision data at a center-of-mass energy of 8 TeV recorded with the ATLAS detector at the Large Hadron Collider. No evidence of their production is observed and 95% confidence-level upper limits on the production cross sections are set. These results are then combined with the published results of the hh→γγbb, bbbb analyses. An upper limit of 0.69 (0.47) pb on the nonresonant hh production is observed (expected), corresponding to 70 (48) times the SM gg→hh cross section. For production via narrow resonances, cross-section limits of hh production from a heavy Higgs boson decay are set as a function of the heavy Higgs boson mass. The observed (expected) limits range from 2.1 (1.1) pb at 260 GeV to 0.011 (0.018) pb at 1000 GeV. These results are interpreted in the context of two simplified scenarios of the Minimal Supersymmetric Standard Model.