An accumulation of research evidence indicates that hate crimes as a category of offence are more serious than similar but otherwise motivated crimes in respect of the greater post-victimization distress reported by victims. Such evidence has been used by advocates of hate crime laws to justify the imposition of greater penalties in the sentencing of convicted hate crime offenders. However, in focusing on the commonalities of the greater level of post-victimization impacts experienced by hate crime victims as a group, the research evidence to date has obscured the diversity of reactions between victims. Consequently, this article expands the evidence in new directions by illuminating the variation in reported victim impacts. The analysis presented uses data from the Crime Survey for England and Wales on reported racially motivated crime and reveals that not all victims report being emotionally affected by hate crime, not all victims are affected the same way, and some victims of racially motivated crime report less of an emotional impact than some victims of equivalent but otherwise motivated crimes. The research findings are used to reason that in any individual case of hate crime the motivating sentiments of the offender provide an unreliable indicator of the harms inflicted upon the victim. Therefore a blanket uplift in penalty in every case of hate crime which rests upon the offender’s mental state— their prejudice, bigotry, bias, or ‘hate’ — cannot be justified if the justification for sentence uplift is to give offenders their just deserts for the harms they inflict. Instead, the justification must rest upon the culpability of the offender for the harms they may or may not actually inflict. Just as there is variation in victim impacts, there will be variation in offender culpability: discretion and flexibility in the mode of enhanced sentencing is therefore necessary to ensure justice for offenders as well as victims.