Home > Research > Publications & Outputs > Towards social justice for item writers
View graph of relations

Towards social justice for item writers: Empowering item writers through language assessment literacy training

Research output: Contribution to conference - Without ISBN/ISSN Conference paper

Unpublished

Standard

Towards social justice for item writers: Empowering item writers through language assessment literacy training. / Rossi, Olena; Brunfaut, Tineke.
2020. Paper presented at 41st LTRC (Language Testing Research Colloquium), Atlanta, United States.

Research output: Contribution to conference - Without ISBN/ISSN Conference paper

Harvard

APA

Vancouver

Rossi O, Brunfaut T. Towards social justice for item writers: Empowering item writers through language assessment literacy training. 2020. Paper presented at 41st LTRC (Language Testing Research Colloquium), Atlanta, United States.

Author

Rossi, Olena ; Brunfaut, Tineke. / Towards social justice for item writers : Empowering item writers through language assessment literacy training. Paper presented at 41st LTRC (Language Testing Research Colloquium), Atlanta, United States.

Bibtex

@conference{860090542b9549ffab2abc5f1dc20f9f,
title = "Towards social justice for item writers: Empowering item writers through language assessment literacy training",
abstract = "Item writers play a key role in the language test cycle, as they essentially need to operationalise the construct into actual tasks. Often, however, these assessment professionals receive a rather narrowly-focused training in writing items to a particular set of specifications. Usually, this training is limited to {\textquoteleft}item writing guidelines{\textquoteright} or instruction in mechanical aspects of item writing. In this presentation, we argue that mechanical training is not sufficient for item writers to consistently produce high-quality items. Item writers need to be empowered through more comprehensive assessment literacy training so that they acquire a deeper understanding of not only {\textquoteleft}how{\textquoteright} but also {\textquoteleft}why{\textquoteright} item specifications contain specific requirements and how their work contributes to test validity. Our viewpoint is based on a study in which we explored the effectiveness of a three-month online item writer course that included instruction in writing specific item types as well as in broader language assessment principles. In our talk, we will provide an overview of the course{\textquoteright}s content, and explain how we evaluated the training through a {\textquoteleft}pretest-posttest{\textquoteright} design. More specifically, the 25 novice item writers participating in the study completed three item writing tasks prior to the course as well as after having finished it. The quality of the items they produced were evaluated by expert item reviewers. In addition, pre- and post-course stimulated recall interviews were conducted with the item writers, course feedback was collected at various points during the training, and online group discussions were analysed. In our presentation, we will primarily focus on the interviews, which aimed to gain information on the approach, procedures, and techniques the trainees used while writing items, the difficulties they encountered, and their post-course reflections on the training{\textquoteright}s effect on their item writing skills. Analysis of the interview data revealed that improvements in item quality pre- to post-course were associated with an increased awareness post-course – on behalf of the item writer – of fundamental assessment principles such as authenticity and fairness, a deeper understanding of the test construct, and conscious attempts to improve the validity of items by avoiding bias, construct underrepresentation, and construct-irrelevant variance. The findings suggest that bringing item writers to stage 4 of Pill & Harding{\textquoteright}s (2013) language assessment literacy continuum, i.e. procedural and conceptual literacy, empowers item writers in their work by enabling them to develop higher-quality items, and by making them more interested and more confident in writing language test items. Our study has practical implications for item writer training and, more fundamentally, for the empowerment of item writers and the development of good-quality tests. ",
keywords = "item writing, item writer training, language assessment literacy",
author = "Olena Rossi and Tineke Brunfaut",
year = "2020",
month = mar,
day = "12",
language = "English",
note = "41st LTRC (Language Testing Research Colloquium) ; Conference date: 04-03-2019 Through 08-03-2019",
url = "https://www.iltaonline.com/general/custom.asp?page=LTRC2019Landing",

}

RIS

TY - CONF

T1 - Towards social justice for item writers

T2 - 41st LTRC (Language Testing Research Colloquium)

AU - Rossi, Olena

AU - Brunfaut, Tineke

PY - 2020/3/12

Y1 - 2020/3/12

N2 - Item writers play a key role in the language test cycle, as they essentially need to operationalise the construct into actual tasks. Often, however, these assessment professionals receive a rather narrowly-focused training in writing items to a particular set of specifications. Usually, this training is limited to ‘item writing guidelines’ or instruction in mechanical aspects of item writing. In this presentation, we argue that mechanical training is not sufficient for item writers to consistently produce high-quality items. Item writers need to be empowered through more comprehensive assessment literacy training so that they acquire a deeper understanding of not only ‘how’ but also ‘why’ item specifications contain specific requirements and how their work contributes to test validity. Our viewpoint is based on a study in which we explored the effectiveness of a three-month online item writer course that included instruction in writing specific item types as well as in broader language assessment principles. In our talk, we will provide an overview of the course’s content, and explain how we evaluated the training through a ‘pretest-posttest’ design. More specifically, the 25 novice item writers participating in the study completed three item writing tasks prior to the course as well as after having finished it. The quality of the items they produced were evaluated by expert item reviewers. In addition, pre- and post-course stimulated recall interviews were conducted with the item writers, course feedback was collected at various points during the training, and online group discussions were analysed. In our presentation, we will primarily focus on the interviews, which aimed to gain information on the approach, procedures, and techniques the trainees used while writing items, the difficulties they encountered, and their post-course reflections on the training’s effect on their item writing skills. Analysis of the interview data revealed that improvements in item quality pre- to post-course were associated with an increased awareness post-course – on behalf of the item writer – of fundamental assessment principles such as authenticity and fairness, a deeper understanding of the test construct, and conscious attempts to improve the validity of items by avoiding bias, construct underrepresentation, and construct-irrelevant variance. The findings suggest that bringing item writers to stage 4 of Pill & Harding’s (2013) language assessment literacy continuum, i.e. procedural and conceptual literacy, empowers item writers in their work by enabling them to develop higher-quality items, and by making them more interested and more confident in writing language test items. Our study has practical implications for item writer training and, more fundamentally, for the empowerment of item writers and the development of good-quality tests.

AB - Item writers play a key role in the language test cycle, as they essentially need to operationalise the construct into actual tasks. Often, however, these assessment professionals receive a rather narrowly-focused training in writing items to a particular set of specifications. Usually, this training is limited to ‘item writing guidelines’ or instruction in mechanical aspects of item writing. In this presentation, we argue that mechanical training is not sufficient for item writers to consistently produce high-quality items. Item writers need to be empowered through more comprehensive assessment literacy training so that they acquire a deeper understanding of not only ‘how’ but also ‘why’ item specifications contain specific requirements and how their work contributes to test validity. Our viewpoint is based on a study in which we explored the effectiveness of a three-month online item writer course that included instruction in writing specific item types as well as in broader language assessment principles. In our talk, we will provide an overview of the course’s content, and explain how we evaluated the training through a ‘pretest-posttest’ design. More specifically, the 25 novice item writers participating in the study completed three item writing tasks prior to the course as well as after having finished it. The quality of the items they produced were evaluated by expert item reviewers. In addition, pre- and post-course stimulated recall interviews were conducted with the item writers, course feedback was collected at various points during the training, and online group discussions were analysed. In our presentation, we will primarily focus on the interviews, which aimed to gain information on the approach, procedures, and techniques the trainees used while writing items, the difficulties they encountered, and their post-course reflections on the training’s effect on their item writing skills. Analysis of the interview data revealed that improvements in item quality pre- to post-course were associated with an increased awareness post-course – on behalf of the item writer – of fundamental assessment principles such as authenticity and fairness, a deeper understanding of the test construct, and conscious attempts to improve the validity of items by avoiding bias, construct underrepresentation, and construct-irrelevant variance. The findings suggest that bringing item writers to stage 4 of Pill & Harding’s (2013) language assessment literacy continuum, i.e. procedural and conceptual literacy, empowers item writers in their work by enabling them to develop higher-quality items, and by making them more interested and more confident in writing language test items. Our study has practical implications for item writer training and, more fundamentally, for the empowerment of item writers and the development of good-quality tests.

KW - item writing

KW - item writer training

KW - language assessment literacy

M3 - Conference paper

Y2 - 4 March 2019 through 8 March 2019

ER -