This paper forms part of a broader project in which we have attempted to “track” a marking guide for open-ended (or “limited production”) listening test items from its inception to its eventual use among assessors. One key issue that has arisen from this ongoing project is assessor decision making while using and applying a marking guide for open-ended test items. In contexts where open-ended items are used, a marking guide is intended to stipulate precisely what kind of response should be accepted as evidence of the ability under test. However, there often remains scope for assessors to apply their own interpretations of the construct in judging responses that fall outside the necessarily limited information provided in a marking guide. This paper explores this issue in relation to the listening component of the Occupational English Test (OET)—a language test for overseas-qualified health professionals wishing to work in Australia.