The influence of generative Artificial Intelligence (AI) on the propagation and amplification of societal biases, particularly in the context of children’s content creation, is a growing concern. By developing and testing a prototype tool designed to assist children in Digital Storytelling (DST), our research aimed to explore and mitigate the propagation of stereotypes through the use of a character-generating AI tool utilising Stable Diffusion. Despite initial aspirations, the tool demonstrated significant biases inherent in the underlying AI model, leading to the decision against its use by children. The findings we discovered contribute to a broader discourse on the development of ethical AI and its use, advocating for a more responsible and inclusive approach to technological innovation in the context of children’s digital media consumption and creation.