In this paper we propose a two-phase methodology for designing datasets that can be used to test and evaluate activity recognition algorithms. The trade offs between time, cost and recognition performance is one challenge. The effectiveness of a dataset, which contrasts the incremental performance gain with the increase in time, efforts, and number and cost of sensors is another challenging area that is often overlooked. Our proposed methodology is iterative and adaptive and addresses issues of sensor use modality and its effect on overall performance. We present our methodology and provide an assessment for its effectiveness using both a simulation model and a real world deployment. © 2010 Springer-Verlag.