TY - JOUR

T1 - Extending the scope of speaking assessment criteria in a specific-purpose language test

T2 - Operationalizing a health professional perspective.

AU - O'Hagan, Sally

AU - Pill, John

AU - Zhang, Ying

PY - 2016/4/1

Y1 - 2016/4/1

N2 - Criticism of specific-purpose language (LSP) tests is often directed at their limited ability to represent fully the demands of the target language use situation. Such criticisms extend to the criteria used to assess test performance, which may fail to capture what matters to participants in the domain of interest. This paper reports on the outcomes of an attempt to expand the construct of a specific-purpose test through the inclusion of two new professionally relevant criteria designed to reflect the values of domain experts. The test in question was the speaking component of the Occupational English Test (OET), designed to assess the language proficiency of overseas-trained health professionals applying to practise their profession in Australia.The criteria were developed from analysis of health professionals’ feedback to trainees, a source that reflected what the professionals value, that is, their indigenous assessment criteria. The criteria considered amenable to inclusion in the OET were as follows: (1) Clinician Engagement with the patient and (2) Management of Interaction in the consultation. Seven OET assessors were trained to apply these professionally relevant criteria at a workshop that introduced a checklist derived from the original data analysis as a tool to aid understanding of the new criteria. Following the workshop, assessors rated a total of 300 pre-recorded OET speaking test performances using both new and existing criteria. Statistical analyses of the ratings indicate the extent to which a) the judgements of the language-trained assessors using the new criteria were consistent and b) the new and existing criteria aligned in terms of the construct(s) they represent. Furthermore, feedback from the assessors in the process shows how comfortable and confident they are to represent a health professional perspective.

AB - Criticism of specific-purpose language (LSP) tests is often directed at their limited ability to represent fully the demands of the target language use situation. Such criticisms extend to the criteria used to assess test performance, which may fail to capture what matters to participants in the domain of interest. This paper reports on the outcomes of an attempt to expand the construct of a specific-purpose test through the inclusion of two new professionally relevant criteria designed to reflect the values of domain experts. The test in question was the speaking component of the Occupational English Test (OET), designed to assess the language proficiency of overseas-trained health professionals applying to practise their profession in Australia.The criteria were developed from analysis of health professionals’ feedback to trainees, a source that reflected what the professionals value, that is, their indigenous assessment criteria. The criteria considered amenable to inclusion in the OET were as follows: (1) Clinician Engagement with the patient and (2) Management of Interaction in the consultation. Seven OET assessors were trained to apply these professionally relevant criteria at a workshop that introduced a checklist derived from the original data analysis as a tool to aid understanding of the new criteria. Following the workshop, assessors rated a total of 300 pre-recorded OET speaking test performances using both new and existing criteria. Statistical analyses of the ratings indicate the extent to which a) the judgements of the language-trained assessors using the new criteria were consistent and b) the new and existing criteria aligned in terms of the construct(s) they represent. Furthermore, feedback from the assessors in the process shows how comfortable and confident they are to represent a health professional perspective.

KW - Healthcare communication

KW - language proficiency

KW - LSP testing

KW - Occupational English Test

KW - speaking criteria

U2 - 10.1177/0265532215607920

DO - 10.1177/0265532215607920

M3 - Journal article

VL - 33

SP - 195

EP - 216

JO - Language Testing

JF - Language Testing

SN - 0265-5322

IS - 2

ER -