Numerous studies have reported category differences between animate and inanimate objects in the early visual ERP components. However, contradictory explanations have been suggested for the underlying processes of such category differences. Both low-level physical differences and higher-level category-specific processes have been shown to modulate the visual ERPs. The present research investigates how physical features (such as amplitude spectrum and spatial frequency) and top-down processes (the categorization task that the participants perform) interact and modulate the visual ERPs. We found ERP correlates of categorical representation for animal and inanimate object categories, as well as early, task-related top-down modulation of the visual ERPs. These results indicate that top-down factors can modulate visual processing both at the level of lower-level physical features and at the level of category representations. The results are discussed in terms of shape- and/or category-selective representations and brain areas in the ventral visual pathway, and they are interpreted within the framework of flexible evidence accumulation processes.