Recent studies have shown that the involvement of semantic information in visual lexical decision depends on the nature of nonword foils with semantic effects increased as nonwords become more word-like (Evans, Lambon Ralph &Woollams, 2012). Given that most models of lexical decision focus on orthographic information (Coltheart, Rastle, Perry, Langdon & Ziegler, 2001; Grainger & Jacobs, 1996; Seidenberg & McClelland, 1989), the role of semantics and its interactions with vision, orthography, and phonology has been overlooked. We developed a recurrent connectionist model of single word reading including visual, orthographic, phonological, and semantic processing. The model differentiated words from nonwords by integrating measures of polarity across four key processing layers. The contribution of semantics depended on the type of nonword foils. The model was more reliant on semantic information when the nonword foils were pseudowords and pseudohomophones rather than consonant strings. The results support the view that semantic involvement in lexical decision is graded by the difficulty of the decision task.