Social engineering is a popular attack vector among cyber criminals. During such attacks, impostors often attempt to trigger empathy to manipulate victims into taking dangerous actions, for example, sharing their credentials or clicking on malicious email attachments. The objective of this position paper is to initiate a conversation on the tension between positive and negative aspects of empathy in HCI as it pertains to security-relevant behaviors. To this end, we focus on the malicious ways in which empathy can be instrumentalized in social engineering. We describe examples of such empathy-related social engineering attacks, explore potential solutions (including the automated detection of empathy-triggering communication, or of empathetic communication on the part of a potential victim), and discuss technical, social as well as organizational interventions. We highlight research challenges and directions for future work.