A long haul optical communication system (LHOCS) is one of the key resources to fulfill the higher capacity requirements in future communication networks. To launch LHOCS, the system mainly faces high order nonlinear effects. The four-wave mixing (FWM) is one of the major nonlinear effects, which limits the transmission distance. Therefore, in this paper, an advanced duo-binary (DB) modulation scheme-based system is evaluated by employing an improved digital signal processing (IDSP) approach at the receiver side to suppress the FWM effect. In addition, an analytical analysis is also performed for the proposed system. To observe the difference between the IDSP and conventional digital signal processing (DSP), the various performance metrics such as bit error rate (BER), Q-factor, and optical signal-to-noise ratio (OSNR) parameters are evaluated. Variable channel spacing along with polarization mode dispersion (PMD) are analyzed at several ranges of input powers and fiber lengths. The analytical and simulation-based calculations exhibit the effectiveness of the proposed model and hence, FWM effect are compensated to achieve 500 km optical fiber propagation range with a BER below 10⁻⁶ .