The mass adoption of digital technologies raises questions about how they impact people and society. Associations between technology use and negative correlates (e.g., depression and anxiety) remain common. However, pre-registered studies have failed to replicate these findings. Regardless of direction, many designs rely on psychometric scales that claim to define and quantify a construct associated with technology engagement. These often suggest clinical manifestations present as disorders or addictions. Given their importance for research integrity, we consider what these scales might be measuring. Across three studies, we observe that many psychometric scales align with a single, identical construct despite claims they capture something unique. We conclude that many technology measures appear to measure a similar, poorly defined construct that sometimes overlaps with pre-existing measures of well-being. Social scientists should critically consider how they proceed methodologically and conceptually when developing psychometric scales in this domain to ensure research findings sit on solid foundations.