As part of the production of written information, patient reader panels provide judgments of their understanding to evaluate the comprehensibility of draft documents. Previous research has suggested i) that there is a limited association, on average, between judgments of understanding and the comprehension demonstrated in tests of understanding and ii) that there is considerable variability between individuals in the direction and magnitude of this association. Unfortunately, while previous research implies, critically, that reader judgments of comprehensibility have limited utility, this research itself is characterized by important limitations that prevent firm conclusions.
This thesis comprises three experimental studies. The study design, method of measurement, and the approach to analysis were motivated by a critical review of previous research. The specification of participant, text and question sample sizes was determined by a novel method of prospective study design analysis, evaluating the accuracy and precision in effect estimation. The robustness of effect estimates are established through the series of empirical replications and in analytical sensitivity checks.
Across the studies, a weakly positive association between perceived and assessed comprehension was found across individuals, on average. Differences in reading ability and background knowledge did not reliably influence metacomprehension accuracy. Further, metacomprehension judgements were similarly predictive of performance on comprehension questions that targeted more versus less semantically central information. In contrast, metacomprehension judgements targeting specific ideas within texts were more predictive of understanding.
The findings of this thesis indicate that metacomprehension judgements are not a gold-standard method of evaluation: judgements show some predictive validity of comprehension outcomes, yet provide little insight into whether critical elements of the documents are sufficiently understood. Overall, whilst situated within an applied context, the present research contributes more widely to the metacomprehension literature, making clear the need for a shift from traditional analytical approaches, in addition to greater theoretical precision.