In this paper we discuss the obdurate problems associated with evaluating the extent to which technological interventions – in particular those based on mobile and ubiquitous technologies – can be judged to have 'improved a sense of community' in their given deployment settings. We report on experiences gained from several deployments of ubiquitous systems that share this design goal, and analyze common issues we observed during real life use of these systems. Based on these we discuss some of the key challenges for evaluating ubiquitous systems of this genre.