Background: Recent research has highlighted the importance of psychological interventions such as cognitive behavioural therapy (CBT) in improving outcomes and promoting recovery for people with experience of psychosis, although a lack of trained therapists means that availability of face-to-face CBT is low. Alternative modes of delivering CBT are being explored, such as telephone and self-help methods, although research to date on whether they can be implemented effectively is limited. Aims: The aims of the present study were to describe and evaluate a new therapy fidelity scale (ROSTA; Recovery Oriented Self-help and Telephone therapy Adherence). This scale was developed to assess fidelity to cognitive behaviour therapy for psychosis (CBTp) focused on improving recovery, with optional subscales for delivery over the telephone and alongside a self-help guide. Method: Experienced CBT therapists rated recorded therapy sessions using the ROSTA scale. The scores were used to assess internal consistency and inter-rater reliability, before being compared to scores from an independent expert rater using an alternative fidelity scale for cognitive therapy in psychosis (the CTS-Psy), to investigate concurrent validity. Results: The ROSTA scale demonstrated excellent internal consistency, inter-rater reliability and validity when evaluated as a whole, although findings were mixed in terms of the individual subscales and items. Conclusions: The ROSTA scale is, on the whole, a reliable and valid tool, which may be useful in training and supervision, a utility that would be further emphasized if the therapeutic intervention it assesses is deemed to be efficacious based on future work.