We compare the performance of the extended Kalman filter (EKF), unscented Kalman filter (UKF), and particle filter (PF) for the angle-only filtering problem in 3D using bearing and elevation measurements from a single maneuvering sensor. These nonlinear filtering algorithms use discrete-time dynamic and measurement models. Two types of coordinate systems are considered, Cartesian coordinates and modified spherical coordinates (MSC) for the relative state vector. The paper presents new algorithms using the UKF and PF with the MSC. We also present an improved filter initialization algorithm. Numerical results from Monte Carlo simulations show that the EKF-MSC and UKF-MSC have the best state estimation accuracy among all nonlinear filters considered and have comparable accuracy with modest computational cost.