In this article, we describe an approach to computational modeling and autonomous learning of the perception of sensory inputs by individuals. A hierarchical process of summarization of heterogeneous raw data is proposed. At the lower level of the hierarchy, the raw data autonomously form semantically meaningful concepts. Instead of clustering based on visual or audio similarity, the concepts are formed at the second level of the hierarchy based on observed physiological variables (PVs) such as heart rate and skin conductance and are mapped to the emotional state of the individual. Wearable sensors were used in the experiments.