TY - JOUR

T1 - Using a logic model to evaluate rater training for EAP writing assessment

AU - O'Connell, J.

PY - 2022/11/30

Y1 - 2022/11/30

N2 - Assessment by written exams and coursework is common practice in pre-sessional and preliminary year EAP programmes, but the allocation of marks for written assessment is complex, as is training raters to apply specified assessment standards. This practitioner research uses a Logic Model, a visual diagram commonly used in programme evaluation, to evaluate the rater training procedure for writing assessment in an English-medium university department. This study integrates data from surveys, interviews and workshops with the stakeholders involved in the rater training procedure to develop a Logic Model as part of an ongoing theory of change evaluation. The final product is a Model that reveals the guiding principles of rater training in the department, text that describes the evaluation process, and a measurement plan. This paper showcases how practitioner research can enhance EAP practice by demonstrating how an essential component of EAP assessment, rater training, and the rationale behind it, can be made cogent to the various stakeholders involved in the procedure. This paper offers considerations for EAP practitioners, managers, and testing staff when developing or working with rater training, bridging the gap between EAP and language testing and assessment communities.

AB - Assessment by written exams and coursework is common practice in pre-sessional and preliminary year EAP programmes, but the allocation of marks for written assessment is complex, as is training raters to apply specified assessment standards. This practitioner research uses a Logic Model, a visual diagram commonly used in programme evaluation, to evaluate the rater training procedure for writing assessment in an English-medium university department. This study integrates data from surveys, interviews and workshops with the stakeholders involved in the rater training procedure to develop a Logic Model as part of an ongoing theory of change evaluation. The final product is a Model that reveals the guiding principles of rater training in the department, text that describes the evaluation process, and a measurement plan. This paper showcases how practitioner research can enhance EAP practice by demonstrating how an essential component of EAP assessment, rater training, and the rationale behind it, can be made cogent to the various stakeholders involved in the procedure. This paper offers considerations for EAP practitioners, managers, and testing staff when developing or working with rater training, bridging the gap between EAP and language testing and assessment communities.

KW - Logic model

KW - Rater training

KW - EAP writing assessment

KW - Language testing and assessment

U2 - 10.1016/j.jeap.2022.101160

DO - 10.1016/j.jeap.2022.101160

M3 - Journal article

VL - 60

JO - Journal of English for Academic Purposes

JF - Journal of English for Academic Purposes

SN - 1475-1585

M1 - 101160

ER -