Designing technology with sustainability in mind is becoming more and more important, especially considering future scenarios of limited resources where the world’s current lifestyle of wasteful consumption needs to change. But how can researchers believably argue that their solutions are indeed sustainable? How can consumers and technology users reliably acquire, understand, and apply information about environmental sustainability? Those questions are difficult to answer, especially in research domains where the impact on sustainability is not immediately measurable, such as sustainable HCI. The evaluation of sustainability is an ongoing problem that is often glossed over, but we believe the community needs to intensify its efforts to articulate its evaluation methods to other disciplines and external stakeholders. Even if those disciplines and stakeholders understand the importance of designing for sustainability, we need convincing arguments – such as validation through thorough evaluations – to showcase why a specific design solution works in the real world. In this paper, we analyze this problem by highlighting examples of sustainable HCI research in which evaluation of sustainability failed. We also look at previous research that sought to address this issue and discuss how their solutions can be generalized – and when they might fail. While we do not have the final answer, our intention is to start a discussion as to why sustainable HCI research is oftentimes not doing enough to justify the validity of its solutions. We close our paper by suggesting a few examples of what we believe to be potential ways to address those issues and take action to improve the evaluation of sustainability.