This paper introduces a mixed-mode matrix representation of scattering parameters that is suitable for the investigation of wireless communication based on orbital angular momentum (OAM). For this purpose, a unitary transformation matrix is defined, which contains the transmitted and received amplitudes as basis vectors corresponding to OAM-based communication between two antenna arrays. The mixed-mode matrix is then obtained from the corresponding similarity transformation of the standard scattering matrix available, e.g., from numerical simulations or measurements. The mixed-mode matrix allows better physical insight into OAM-based communication by clearly separating all modes and is more efficient than postprocessing of single-antenna results in a system simulator. Mode conversion/isolation and proper mode termination can be easily quantified. Also, it allows generalizing the interpretation of OAM-based communication by focusing on the property of a constant phase difference within each antenna array. The usefulness of this approach is demonstrated using method of moments (MoM) simulations of dipole radiators in various arrangements of a transmitting and a receiving array. Systematic parameter studies that reveal dependencies of OAM-based communication are possible by using the mixed-mode matrix. As an application, communication patterns are studied, in order to obtain the suitable position and relative orientation in space for good mode isolation or communication.