There has been a proliferation of industry-focused cyber security qualifications, which use different techniques to assess the competencies of cyber security professionals and certify them to employers. There is, however, a lingering question about these qualifications: do they effectively assess the competencies of cyber security professionals? 74 cyber security qualifications were analysed to determine how competency assessment is performed in practice, and five distinct techniques were identified together with the frequency of their use within qualifications. These techniques formed the basis of a large-scale survey of the perceptions of 153 industry stakeholders on the effectiveness of individual techniques and their cost-effectiveness as combinations. Despite a perceived low effectiveness of Multiple Choice Examinations, industry qualifications were found to rely on it heavily, often as a sole technique, and few qualifications utilised the cost-effective combinations identified by stakeholders.