We present spectroscopic observations for a sample of 36 Herschel-SPIRE250-500 μm selected galaxies (HSGs) at 2 <z <5 from theHerschel Multi-tiered Extragalactic Survey. Redshifts are confirmed aspart of a large redshift survey of Herschel-SPIRE-selected sourcescovering ~0.93 deg2 in six extragalactic legacy fields.Observations were taken with the Keck I Low Resolution ImagingSpectrometer and the Keck II DEep Imaging Multi-Object Spectrograph.Precise astrometry, needed for spectroscopic follow-up, is determined byidentification of counterparts at 24 μm or 1.4 GHz using across-identification likelihood matching method. Individual sourceluminosities range from log (L IR/L ⊙) =12.5-13.6 (corresponding to star formation rates (SFRs) 500-9000 M⊙ yr-1, assuming a Salpeter initialmass function), constituting some of the most intrinsically luminous,distant infrared galaxies discovered thus far. We present bothindividual and composite rest-frame ultraviolet spectra and infraredspectral energy distributions. The selection of these HSGs isreproducible and well characterized across large areas of the sky incontrast to most z > 2 HyLIRGs in the literature, which are detectedserendipitously or via tailored surveys searching only for high-zHyLIRGs; therefore, we can place lower limits on the contribution ofHSGs to the cosmic star formation rate density (SFRD) at (7 ± 2)× 10-3 M ⊙yr-1 h 3 Mpc-3 at z ~ 2.5,which is >10% of the estimated total SFRD of the universe fromoptical surveys. The contribution at z ~ 4 has a lower limit of 3× 10-3 M ⊙yr-1 h 3 Mpc-3, gsim20% ofthe estimated total SFRD. This highlights the importance of extremelyinfrared-luminous galaxies with high SFRs to the buildup of stellarmass, even at the earliest epochs.