This paper presents a novel approach for remaining useful life (RUL) prediction of rotating machinery using hierarchical deep neural networks (DNN). The different health stages are classified by a DNN-based health stage classifier trained by segmented degradation signal. This method builds several RUL predictors based on the health stages of the degradation process. Instead of modeling the entire degradation process (typically including various stages with dramatically different properties) with a single model, the proposed approach builds RUL model for each health stage where more accurate fitting can be obtained. A smoothing operator is applied to obtain the final RUL prediction. The experimental results show that the proposed method can achieve more accurate RUL prediction.