Object detection is an important problem and has a wide range of applications. In recent years, deep learning based object detection with conventional RGB cameras has made great progress. At the same time, people are more and more aware of the limitations of RGB cameras. The progress of algorithms alone can not fundamentally resolve the challenges of object detection. Unmanned vehicles or mobile robot platforms are often equipped with a variety of sensors in addition to RGB camera, each of which have its own characteristics, and can expand the sensing range of RGB camera from different dimensions. For example, infrared thermal imaging camera and multispectral camera broaden sensing range from spectral dimension, while LiDARs and depth cameras are able to broaden sensing range from the spatial dimension. This paper mainly summarizes the deep learning based object detection methods under the condition of multi-modal sensors, and surveys and categorizes the methods from the perspective of data fusion manner. The datasets of different modality are summarized, and the advantages and disadvantages with different combination of sensors are also discussed in this paper.