TY - CONF

T1 - Identification of true RCTs of the effects of educational interventions on asthma-related outcomes in Medline and Embase using abstracts versus full text

AU - Wolf, Fred

AU - Grum, Cyril

AU - Lefebvre, Carol

AU - Bara, Anna

AU - Milan, Stephen James

AU - Jones, Paul

PY - 1995

Y1 - 1995

N2 - Introduction: Electronic searches of bibliographic databases in search of randomized controlled trials (RCTs) may be thought to yield paradoxical results. On the one hand, it is increasingly known that electronic databases miss a significant number of true RCTs that can be identified by hand searching journals. On the other hand, all citations retrieved from an optimally sensitive search are not necessarily true reports of RCTs. One practical, relevant question is the degree to which individual citations may accura-tely be classified as true RCTs on the basis of the database's abstract alone.Objective: To ascertain the degree to which individual Medline (Index Medicus) and Embase (Excerpta Medica) citations may accurately be classified as true RCTs on the basis of the database's abstract alone, or whether a detailed reading of all full text articles, particularly methods sections, is necessary.Methods: A total of 108 citations were retrieved in electronic searches of Medline and Embase databases from 1980-1993, based on the standard Cochrane Collaboration search strategy for RCTs (1994 Cochrane Collaboration Handbook, Chapter V) in combination with "and asthma" and "and education". Each citation was classified on the basis of the abstract alone by two independent raters as (1) a true RCT, (2) not a true RCT, or (3) too ambiguous to classify without additional information. Inter-rater agreement between the two raters was calculated using the Kappa coefficient. The raters then classified each citation into one of the three categories on the basis of the full text version of the article. A Kappa coefficient was again calculated to estimate inter-rater agreement of full text classification. Categories 2 and 3 were then combined and a 2 x 2 table was constructed in order to assess the accuracy of RCT classification by abstracts alone, using full text classification as the "gold standard" for comparative purposes. The sensitivity, specificity, and predictive values of abstract classification were then calculated, as well as true and false positive and negative rates.Results/Discussion: This project is in progress. Results, which will be presented, will permit a direct comparison of estimates of inter-rater agreement in classifying RCTs between using an "abstracts only" approach versus a more intensive "full text" approach. Estimates of the accuracy of abstract classification is particularly important if this approach is used to make initial inclusionary/exclusionary judgments of RCTs for systematic reviews.

AB - Introduction: Electronic searches of bibliographic databases in search of randomized controlled trials (RCTs) may be thought to yield paradoxical results. On the one hand, it is increasingly known that electronic databases miss a significant number of true RCTs that can be identified by hand searching journals. On the other hand, all citations retrieved from an optimally sensitive search are not necessarily true reports of RCTs. One practical, relevant question is the degree to which individual citations may accura-tely be classified as true RCTs on the basis of the database's abstract alone.Objective: To ascertain the degree to which individual Medline (Index Medicus) and Embase (Excerpta Medica) citations may accurately be classified as true RCTs on the basis of the database's abstract alone, or whether a detailed reading of all full text articles, particularly methods sections, is necessary.Methods: A total of 108 citations were retrieved in electronic searches of Medline and Embase databases from 1980-1993, based on the standard Cochrane Collaboration search strategy for RCTs (1994 Cochrane Collaboration Handbook, Chapter V) in combination with "and asthma" and "and education". Each citation was classified on the basis of the abstract alone by two independent raters as (1) a true RCT, (2) not a true RCT, or (3) too ambiguous to classify without additional information. Inter-rater agreement between the two raters was calculated using the Kappa coefficient. The raters then classified each citation into one of the three categories on the basis of the full text version of the article. A Kappa coefficient was again calculated to estimate inter-rater agreement of full text classification. Categories 2 and 3 were then combined and a 2 x 2 table was constructed in order to assess the accuracy of RCT classification by abstracts alone, using full text classification as the "gold standard" for comparative purposes. The sensitivity, specificity, and predictive values of abstract classification were then calculated, as well as true and false positive and negative rates.Results/Discussion: This project is in progress. Results, which will be presented, will permit a direct comparison of estimates of inter-rater agreement in classifying RCTs between using an "abstracts only" approach versus a more intensive "full text" approach. Estimates of the accuracy of abstract classification is particularly important if this approach is used to make initial inclusionary/exclusionary judgments of RCTs for systematic reviews.

M3 - Poster

ER -