Driven by critical and pressing smart city applications, accessing massive numbers of sensors by cloud-hosted services is becoming an emerging and inevitable situation. Naïvely connecting massive numbers of sensors to the cloud raises major scalability and energy challenges. An architecture embodying distributed optimization is needed to manage the scale and to allow limited energy sensors to last longer in such a dynamic and high-velocity big data system. We developed a multi-tier architecture which we call Cloud, Edge and Beneath (CEB). Based on CEB, we propose an Application Fragment Caching Algorithm (AFCA) which selectively caches application fragments from the cloud to lower layers of CEB to improve cloud scalability. Through experiments, we show and measure the effect of AFCA on cloud scalability. Copyright 2014 ACM.