TY - JOUR

T1 - Drawing on indigenous criteria for more authentic assessment in a specific-purpose language test

T2 - Health professionals interacting with patients

AU - Pill, John

PY - 2016/4/1

Y1 - 2016/4/1

N2 - The indigenous assessment practices (Jacoby & McNamara, 1999) in selected health professions were investigated to inform a review of the scope of assessment in the speaking sub-test of a specific-purpose English language test for health professionals, the Occupational English Test (OET). The assessment criteria in current use on the test represent a generalized view of language and are concerned with Overall Communicative Effectiveness, Fluency, Intelligibility, Appropriateness of Language, and Resources of Grammar and Expression. The research study focused on healthcare consultations between trainee health professionals and patients. Educators and supervisors observed these interactions and subsequently provided feedback on trainees’ performances. The assumption was that, in their comments, educators would give information pertinent to trainees’ acculturation to the expectations and behaviours of the profession, that is, to “what matters” to practitioners. Thematic analysis was undertaken to establish the aspects of performance that matter to health professionals in these contexts. Data for each profession were coded independently. Clear similarities across the professions became apparent as themes emerged. An exploratory conceptual model of what health professionals value in the consultation was developed, comprising three focal areas: foundation, performance and goals of the consultation. Findings from the analysis provided an empirical basis for the generation and definition of two additional, professionally relevant criteria for use in the OET speaking sub-test – Clinician Engagement and Management of Interaction – and of a checklist of performance indicators to be used to train assessors in applying the new criteria. This process of developing, through close analysis of domain experts’ commentary, test criteria that are potentially more authentic to the target language use situation is novel and may be replicated effectively in other specific-purpose language testing contexts.

AB - The indigenous assessment practices (Jacoby & McNamara, 1999) in selected health professions were investigated to inform a review of the scope of assessment in the speaking sub-test of a specific-purpose English language test for health professionals, the Occupational English Test (OET). The assessment criteria in current use on the test represent a generalized view of language and are concerned with Overall Communicative Effectiveness, Fluency, Intelligibility, Appropriateness of Language, and Resources of Grammar and Expression. The research study focused on healthcare consultations between trainee health professionals and patients. Educators and supervisors observed these interactions and subsequently provided feedback on trainees’ performances. The assumption was that, in their comments, educators would give information pertinent to trainees’ acculturation to the expectations and behaviours of the profession, that is, to “what matters” to practitioners. Thematic analysis was undertaken to establish the aspects of performance that matter to health professionals in these contexts. Data for each profession were coded independently. Clear similarities across the professions became apparent as themes emerged. An exploratory conceptual model of what health professionals value in the consultation was developed, comprising three focal areas: foundation, performance and goals of the consultation. Findings from the analysis provided an empirical basis for the generation and definition of two additional, professionally relevant criteria for use in the OET speaking sub-test – Clinician Engagement and Management of Interaction – and of a checklist of performance indicators to be used to train assessors in applying the new criteria. This process of developing, through close analysis of domain experts’ commentary, test criteria that are potentially more authentic to the target language use situation is novel and may be replicated effectively in other specific-purpose language testing contexts.

KW - Health professional–patient interaction

KW - healthcare communication

KW - indigenous assessment criteria

KW - language proficiency

KW - LSP testing

KW - Occupational English Test

U2 - 10.1177/0265532215607400

DO - 10.1177/0265532215607400

M3 - Journal article

VL - 33

SP - 175

EP - 193

JO - Language Testing

JF - Language Testing

SN - 0265-5322

IS - 2

ER -