The largest Herschel extragalactic surveys, H-ATLAS and HerMES, have selected a sample of "ultrared" dusty, star-forming galaxies (DSFGs) with rising SPIRE flux densities ($S_{500} > S_{350} > S_{250}$; so-called "500 $\mu$m-risers") as an efficient way for identifying DSFGs at higher redshift ($z > 4$). In this paper, we present a large Spitzer follow-up program of 300 Herschel ultrared DSFGs. We have obtained high-resolution ALMA, NOEMA, and SMA data for 63 of them, which allow us to securely identify the Spitzer/IRAC counterparts and classify them as gravitationally lensed or unlensed. Within the 63 ultrared sources with high-resolution data, $\sim$65% appear to be unlensed, and $\sim$27% are resolved into multiple components. We focus on analyzing the unlensed sample by directly performing multi-wavelength spectral energy distribution (SED) modeling to derive their physical properties and compare with the more numerous $z \sim 2$ DSFG population. The ultrared sample has a median redshift of 3.3, stellar mass of 3.7 $\times$ 10$^{11}$ $M_{\odot}$, star formation rate (SFR) of 730 $M_{\odot}$yr$^{-1}$, total dust luminosity of 9.0 $\times$ 10$^{12}$ $L_{\odot}$, dust mass of 2.8 $\times$ 10$^9$ $M_{\odot}$, and V-band extinction of 4.0, which are all higher than those of the ALESS DSFGs. Based on the space density, SFR density, and stellar mass density estimates, we conclude that our ultrared sample cannot account for the majority of the star-forming progenitors of the massive, quiescent galaxies found in infrared surveys. Our sample contains the rarer, intrinsically most dusty, luminous and massive galaxies in the early universe that will help us understand the physical drivers of extreme star formation.