As a well-known urban landscape concept to describe urban space quality, urban street vitality is a subjective human perception of the urban environment but difficult to evaluate directly from the physical space. The study utilized a modern machine learning computer vision algorithm in the urban build environment to simulate the process, which starts with the visual perception of the urban street landscape and ends with the human reaction to street vitality. By analyzing the optimized trained model, we tried to identify urban street vitality’s visual features and evaluate their importance. A region around the Mochou Lake in Nanjing, China, was set as our study area. Seven investigators surveyed the area, recorded their evaluation score on each site’s vitality level with a corresponding picture taken on site. A total of 370 pictures and recorded score pairs from 231 valid survey sites were used to train a convolutional neural network. After optimization, a deep neural network model with 43 layers, including 11 convolutional ones, was created. Heat maps were then used to identify the features which lead to high vitality score outputs. The spatial distributions of different types of feature entities were also analyzed to help identify the spatial effects. The study found that visual features, including human, construction site, shop front, and roadside/walking pavement, are vital ones that correspond to the vitality of the urban street. The consistency of these critical features with traditional urban vitality features indicates the model had learned useful knowledge from the training process. Applying the trained model in urban planning practices can help to improve the city environment for better attraction of residents’ activities and communications.