Being an essential component of 5G and beyond, the Internet of Drones (IoD) enables a legitimate operator to employ several drones to carry out distinct missions in restricted airspace under the control of the GCS. Communication with the external user (mobile operator), the central server, and other drones is accomplished via a public wireless medium, which is vulnerable to numerous threats and needs robust authentication. Various authentication schemes have been presented, some of which have design issues, excessive communication, or lack of a delicate balance of security and performance due to modular exponentiation. Therefore, this article proposes a secure authentication scheme based on asymmetric cryptography to address the issues above. The security of the proposed scheme is formally scrutinized through the RoR model, ProVerif. Furthermore, the performance is analyzed through computation, communication, and storage overheads. The comparative analysis shows that the proposed scheme balances security with performance and can be implemented in a real-world IoD environment.