Aggressive territoriality can have significant benefits for resource acquisition yet is a costly behaviour. Selection should therefore favour mechanisms that allow individuals to modify their behaviour to maintain and defend their territory whilst minimising costs. One such mechanism between intraspecific territorial competitors is neighbour-stranger discrimination. The familiarity hypothesis suggests that territory holders will respond less aggressively to neighbours they recognise than to strangers they do not recognise. Conversely, in systems where neighbours pose a greater threat to territory loss than strangers, the threat-level hypothesis predicts that neighbours will elicit a greater aggressive response. We tested these opposing hypotheses in territorial farming damselfishes Stegastes diencaeus using a common bottle presentation experiment design to initiate aggressive responses by territory holders to neighbouring and non-neighbour individuals. Neighbours that were smaller in body size than the territory holder elicited the greatest aggressive response, whereas larger neighbours elicited the weakest. The aggressive response elicited by non-neighbours did not vary greatly with body size difference between the stimulus fish and territory holder. We demonstrate that aggressive response in territorial farming damselfishes is influenced by both familiarity and potential threat determined by body size. These findings add to the growing pool of evidence that an understanding of multiple factors is needed to identify the drivers of neighbour-stranger discrimination. Significance statement: Both familiarity and body size may mediate aggressive behaviour yet are not often included in the same study. Using manipulative field experiments, we investigated the interplay between familiarity and body size in shaping patterns of aggressive behaviour in farming damselfishes. We found that territory holders were less aggressive towards neighbours than non-neighbours, but only when they were larger than themselves. Our results showing an interaction between the effects of familiarity and body size on aggressive behaviour may hint at nuances in patterns of neighbour-stranger discrimination, such as dominance relationships.