The existing intelligent fault diagnosis methods of rotor-bearing system mainly focus on vibration analysis under steady operation, which has low adaptability to new scenes. In this paper, a new framework for rotor-bearing system fault diagnosis under varying working conditions is proposed by using a modified convolutional neural network (CNN) with transfer learning. First, infrared thermal images are collected and used to characterize the health condition of rotor-bearing system. Second, modified CNN is developed by introducing stochastic pooling and Leaky rectified linear unit to overcome the training problems in classical CNN. Finally, parameter transfer is used to enable the source modified CNN to adapt to the target domain, which solves the problem of limited available training data in the target domain. The proposed method is applied to analyze thermal images of rotor-bearing system collected under different working conditions. The results show that the proposed method outperforms other cutting edge methods in fault diagnosis of rotor-bearing system.