In this study, a hierarchical multi-label classification method called HFFNet-2d is proposed for the automatic classification of scanning electron microscope (SEM) images of metal failure. The method combines the advantages of convolutional neural networks (CNN) and Vision Transformers (ViT) to effectively realize hierarchical feature extraction and classification of SEM images of fracture morphologies, enabling accurate identification of metal failure mechanisms at different scales. The dataset of high-quality SEM images in this work is sourced from reputable materials science publications for its comprehensive coverage of various failure modes and its suitability for training and validating the hierarchical multi-label classification model. The HFFNet-2d model can achieve a high accuracy of 97.71% in the first-level classification and 92.62% in the second-level sub-category identification. This performance surpasses the human experts on the same task. To ensure that the model predictions are sufficiently reliable, a multi-level gradcam algorithm is also introduced for checking the regions of interest of the Hierarchical model at two levels and the comparisons are made with human experts. It is anticipated that the optimization and extension of HFFNet-2d are conducive in diverse material systems and application scenarios to accelerate the intelligent process of material development and failure analysis, ultimately supporting the design of reliable and high-performance engineering materials.