The animal vs. non-animal category differences in the visual event-related potentials (ERPs) are widely studied. It has been suggested that top-down factors (e.g. attention) modulate the amplitude of the visual ERPs. In a previous study we found a greater car–bird amplitude difference in N1 when participants had to perform a car vs. bird categorization task than when they had to decide whether the images were in color or greyscale. This suggests that the category differences in the visual N1 reflect interactions between bottom-up and top-down processes. However, N1 category effects have also been associated with both low-level physical differences (e.g. amplitude spectrum) and higher-level discrimination processes. Therefore, it is not clear whether the task affects the physical or the category-specific stages of processing. The aim of the present study was to investigate whether this task-related ERP modulation depends on the amplitude spectrum information (as a physical feature) or occurs irrespectively of low-level physical image characteristics. The stimuli were greyscale images of animals and vehicles with an “X” or an “O” in the background. The images were modified in two versions: they were matched either for luminance, or for luminance and amplitude spectrum. Half of the participants (N = 17) had to perform an animal vs. vehicle categorization task, whereas the other half (N = 17) performed an “X” vs. “O” categorization task. The electroencephalogram was recorded on 128 channels (EGI HydroCel Net) and the P1, N1, P2 and N2 visual components were analysed over the occipital and occipitotemporal areas. We found task-related modulation in the P1 component over the occipital area: a category difference was present only in the animals vs. vehicle categorization task and only when the amplitude spectrum was available. For the P2 and N2, category differences were greater in the animal vs. vehicle categorization task than in the “X” vs. “O” task over both the occipital and occipitotemporal areas. Furthermore, this effect occurred irrespectively of the low-level image modulation. Therefore, we have separated two different top-down mechanisms. The early task-related modulation suggests a top-down process that is selective to physical features. The later task effect occurs irrespectively of the physical differences, which suggest that this late top-down process modulates higher-level visual processes (for instance activation of visual category representations). Overall, these results are in line with recent theories suggesting flexible interactions between bottom-up visual processes and top-down factors such as task demands.