In spite of the importance of activity recognition (AR) for intelligent human-computer interaction in emerging smart space applications, state-of-the-art AR technology is not ready or adequate for real-world deployments due to its insufficient accuracy. The accuracy limitation is directly attributed to uncertainties stemming from multiple sources in the AR system. Hence, one of the major goals of AR research is to improve system accuracy by minimizing or managing the uncertainties encountered throughout the AR process. As we cannot manage uncertainties well without measuring them, we must first quantify their impact. Nevertheless, such a quantification process is very challenging given that uncertainties come from diverse and heterogeneous sources. In this article, we propose an approach, which can account for multiple uncertainty sources and assess their impact on AR systems. We introduce several metrics to quantify the various uncertainties and their impact. We then conduct a quantitative impact analysis of uncertainties utilizing data collected from actual smart spaces that we have instrumented. The analysis is intended to serve as groundwork for developing "diagnostic" accuracy measures of AR systems capable of pinpointing the sources of accuracy loss. This is to be contrasted with the currently used accuracy measures. © 2015 ACM 2157-6904/2015/08-ART52 $15.00.