In this paper, the impact of the environment between two communicating orbital angular momentum (OAM) antennas on the orthogonality of the OAM mode is investigated. Orthogonality means one mode's excitation cannot be created by combining other modes. If the orthogonality of the modes is perturbed, inter-mode interference (IMI) occurs between the OAM modes. The question is asked how to improve the environment to minimize IMI. OAM-based communication offers the potential to use orthogonal modes as separate communication channels, but certain environments limit this potential by introducing IMI. It is, therefore, advantageous to investigate the conditions under which IMI occurs. To that aim, the OAM wave transmissions through several environments will be explored numerically using a tool based on the method of moments. It is shown that the orthogonality of the OAM mode requires cyclic symmetry in the environment, and the deviation from this symmetry determines the strength and type of IMI.