The rather separate growth of two strands of work on personal informatics and AI technologies has prompted increased interest in AI-based personal information technologies. Such work however faces the challenges of user limited mental models of technologies in general and AI ones in particular, and of their ethical concerns. To address such challenges, the paper proposes to build on HCI work on user mental models, sociotechnical infrastructures and ethical design. The paper concludes with a reflection on our findings in these areas and articulates novel directions to address the identified challenges in order to design more ethical AI-based personal informatics systems.