Final published version, 644 KB, PDF document
Research output: Book/Report/Proceedings › Other report
Research output: Book/Report/Proceedings › Other report
}
TY - BOOK
T1 - Linking the GEPT listening test to the Common European Framework of Reference
AU - Brunfaut, Tineke
AU - Harding, Luke
PY - 2014
Y1 - 2014
N2 - This document reports on a linking study designed to provide empirical evidence on which to base claims about the connection between the listening test suite of the General English Proficiency Test (GEPT) and the Common European Framework of Reference (CEFR). The investigation was guided by the recommended methods and procedures set out in the manual Relating language examinations to the Common European Framework of Reference for Languages: learning, teaching, assessment (Council of Europe, 2009), and entailed the stages of familiarisation, specification, standardisation and empirical validation. In addition, the project involved an innovative research design marked by a) a pilot stage to evaluate the suitability of two different standard setting methods for a linking study with listening test materials (the basket method and the modified Angoff method), and b) an asynchronous “twin-panel” approach. This methodology allowed for the inclusion and comparison of both insider and outsider perspectives through a panel of six language testing/teaching/standard setting experts without prior knowledge of the GEPT suite and a separate panel of six teachers/researchers/developers with an intimate knowledge of the GEPT suite. Preliminary alignments and cut scores were established by the researchers at the specification stage.Informed by the pilot study, a two-step adapted version of the basket method was developed which requires the standard setter to assign each item to a particular CEFR “basket” and, in addition, to decide whether a just-qualified, a mid, or a high test-taker at the particular CEFR level would already be able to answer the item correctly. After a familiarisation process with the CEFR, the GEPT, and the standard setting procedures, this modified basket method was used by the panels to judge the CEFR level of GEPT listening items. Validation checks were realised through judgement reliability analyses, judgement comparisons across panels, and qualitative analysis of recorded panel discussions.Based on the full linking study – including individual and combined panel data analyses – recommendations are made for the alignment of the GEPT listening suite with the Common European Framework of Reference. In addition, a set of suggestions is formulated in order for the GEPT listening tests to be more easily articulated with the CEFR descriptors, and in so doing to increase the validity of alignment to the framework.
AB - This document reports on a linking study designed to provide empirical evidence on which to base claims about the connection between the listening test suite of the General English Proficiency Test (GEPT) and the Common European Framework of Reference (CEFR). The investigation was guided by the recommended methods and procedures set out in the manual Relating language examinations to the Common European Framework of Reference for Languages: learning, teaching, assessment (Council of Europe, 2009), and entailed the stages of familiarisation, specification, standardisation and empirical validation. In addition, the project involved an innovative research design marked by a) a pilot stage to evaluate the suitability of two different standard setting methods for a linking study with listening test materials (the basket method and the modified Angoff method), and b) an asynchronous “twin-panel” approach. This methodology allowed for the inclusion and comparison of both insider and outsider perspectives through a panel of six language testing/teaching/standard setting experts without prior knowledge of the GEPT suite and a separate panel of six teachers/researchers/developers with an intimate knowledge of the GEPT suite. Preliminary alignments and cut scores were established by the researchers at the specification stage.Informed by the pilot study, a two-step adapted version of the basket method was developed which requires the standard setter to assign each item to a particular CEFR “basket” and, in addition, to decide whether a just-qualified, a mid, or a high test-taker at the particular CEFR level would already be able to answer the item correctly. After a familiarisation process with the CEFR, the GEPT, and the standard setting procedures, this modified basket method was used by the panels to judge the CEFR level of GEPT listening items. Validation checks were realised through judgement reliability analyses, judgement comparisons across panels, and qualitative analysis of recorded panel discussions.Based on the full linking study – including individual and combined panel data analyses – recommendations are made for the alignment of the GEPT listening suite with the Common European Framework of Reference. In addition, a set of suggestions is formulated in order for the GEPT listening tests to be more easily articulated with the CEFR descriptors, and in so doing to increase the validity of alignment to the framework.
KW - Standard setting
KW - Listening
KW - GEPT
M3 - Other report
VL - RG-05
BT - Linking the GEPT listening test to the Common European Framework of Reference
PB - Language Training and Testing Centre
CY - Taiwan
ER -