Home > Research > Publications & Outputs > Using Multi-Faceted Rasch Analysis to Examine R...

Research

Links

Text available via DOI:

View graph of relations

Using Multi-Faceted Rasch Analysis to Examine Raters, Prompts, and Rubrics in an Office-Hour Role-Play Task

Research output: Contribution in Book/Report/Proceedings - With ISBN/ISSN › Chapter

Published

Standard

Using Multi-Faceted Rasch Analysis to Examine Raters, Prompts, and Rubrics in an Office-Hour Role-Play Task. / Ren, Haoshan (Sally); Looney, Stephen; Cushing, Sara.
Local Language Testing: Practice across Contexts. ed. / Xun Yan; Slobdanka Dimova; April Ginther. Cham: Springer, 2023. p. 13-33 (Educational Linguistics; Vol. 61).

Research output: Contribution in Book/Report/Proceedings - With ISBN/ISSN › Chapter

Harvard

Ren, H, Looney, S & Cushing, S 2023, Using Multi-Faceted Rasch Analysis to Examine Raters, Prompts, and Rubrics in an Office-Hour Role-Play Task. in X Yan, S Dimova & A Ginther (eds), Local Language Testing: Practice across Contexts. Educational Linguistics, vol. 61, Springer, Cham, pp. 13-33. https://doi.org/10.1007/978-3-031-33541-9_2

APA

Ren, H., Looney, S., & Cushing, S. (2023). Using Multi-Faceted Rasch Analysis to Examine Raters, Prompts, and Rubrics in an Office-Hour Role-Play Task. In X. Yan, S. Dimova, & A. Ginther (Eds.), Local Language Testing: Practice across Contexts (pp. 13-33). (Educational Linguistics; Vol. 61). Springer. https://doi.org/10.1007/978-3-031-33541-9_2

Vancouver

Ren H, Looney S, Cushing S. Using Multi-Faceted Rasch Analysis to Examine Raters, Prompts, and Rubrics in an Office-Hour Role-Play Task. In Yan X, Dimova S, Ginther A, editors, Local Language Testing: Practice across Contexts. Cham: Springer. 2023. p. 13-33. (Educational Linguistics). doi: 10.1007/978-3-031-33541-9_2

Author

Ren, Haoshan (Sally) ; Looney, Stephen ; Cushing, Sara. / Using Multi-Faceted Rasch Analysis to Examine Raters, Prompts, and Rubrics in an Office-Hour Role-Play Task. Local Language Testing: Practice across Contexts. editor / Xun Yan ; Slobdanka Dimova ; April Ginther. Cham : Springer, 2023. pp. 13-33 (Educational Linguistics).

Bibtex

@inbook{fa3e375353fc47d9aaf66567393b4b30,
title = "Using Multi-Faceted Rasch Analysis to Examine Raters, Prompts, and Rubrics in an Office-Hour Role-Play Task",
abstract = "Locally-administered placement tests for International Teaching Assistants (ITAs) impact the quality of ITAs{\textquoteright} graduate studies and the quality of undergraduate education. However, in-house testing programs face challenges in terms of funding, time, and expertise to ensure the reliability and address the validity of their ITA tests. This chapter examines comparability of prompts used in an office-hour roleplay task on an in-house ITA test. The role-play task requires test-takers to respond to one of four office-hour scenarios where the test-taker plays the role of the teacher and one of the two raters plays the role of the undergraduate student. We used Many-Faceted Rasch Measurement (MFRM) to investigate prompt difficulty, rubric consistency, and selected interactions. All raters were self-consistent, but we found significant differences in raters{\textquoteright} relative severities. Further examination reveals that, although raters were instructed to select one of the four prompts randomly, some raters selected certain prompts to the exclusion of others. The rater x prompt interaction shows that raters{\textquoteright} internal consistency is related to selection, while variation in rater severity is an independent effect. This chapter provides insights into practices that connect test analysis, test quality, and test administration in a local context.",
author = "Ren, {Haoshan (Sally)} and Stephen Looney and Sara Cushing",
year = "2023",
month = jul,
day = "19",
doi = "10.1007/978-3-031-33541-9_2",
language = "English",
isbn = "9783031335402",
series = "Educational Linguistics",
publisher = "Springer",
pages = "13--33",
editor = "Xun Yan and Slobdanka Dimova and April Ginther",
booktitle = "Local Language Testing",

}

RIS

TY - CHAP

T1 - Using Multi-Faceted Rasch Analysis to Examine Raters, Prompts, and Rubrics in an Office-Hour Role-Play Task

AU - Ren, Haoshan (Sally)

AU - Looney, Stephen

AU - Cushing, Sara

PY - 2023/7/19

Y1 - 2023/7/19

N2 - Locally-administered placement tests for International Teaching Assistants (ITAs) impact the quality of ITAs’ graduate studies and the quality of undergraduate education. However, in-house testing programs face challenges in terms of funding, time, and expertise to ensure the reliability and address the validity of their ITA tests. This chapter examines comparability of prompts used in an office-hour roleplay task on an in-house ITA test. The role-play task requires test-takers to respond to one of four office-hour scenarios where the test-taker plays the role of the teacher and one of the two raters plays the role of the undergraduate student. We used Many-Faceted Rasch Measurement (MFRM) to investigate prompt difficulty, rubric consistency, and selected interactions. All raters were self-consistent, but we found significant differences in raters’ relative severities. Further examination reveals that, although raters were instructed to select one of the four prompts randomly, some raters selected certain prompts to the exclusion of others. The rater x prompt interaction shows that raters’ internal consistency is related to selection, while variation in rater severity is an independent effect. This chapter provides insights into practices that connect test analysis, test quality, and test administration in a local context.

AB - Locally-administered placement tests for International Teaching Assistants (ITAs) impact the quality of ITAs’ graduate studies and the quality of undergraduate education. However, in-house testing programs face challenges in terms of funding, time, and expertise to ensure the reliability and address the validity of their ITA tests. This chapter examines comparability of prompts used in an office-hour roleplay task on an in-house ITA test. The role-play task requires test-takers to respond to one of four office-hour scenarios where the test-taker plays the role of the teacher and one of the two raters plays the role of the undergraduate student. We used Many-Faceted Rasch Measurement (MFRM) to investigate prompt difficulty, rubric consistency, and selected interactions. All raters were self-consistent, but we found significant differences in raters’ relative severities. Further examination reveals that, although raters were instructed to select one of the four prompts randomly, some raters selected certain prompts to the exclusion of others. The rater x prompt interaction shows that raters’ internal consistency is related to selection, while variation in rater severity is an independent effect. This chapter provides insights into practices that connect test analysis, test quality, and test administration in a local context.

U2 - 10.1007/978-3-031-33541-9_2

DO - 10.1007/978-3-031-33541-9_2

M3 - Chapter

SN - 9783031335402

SN - 9783031335433

T3 - Educational Linguistics

SP - 13

EP - 33

BT - Local Language Testing

A2 - Yan, Xun

A2 - Dimova, Slobdanka

A2 - Ginther, April

PB - Springer

CY - Cham

ER -