Digital twin is an integrated multi-physics representation of a complex physical entity. This article constructs the digital twin of the construction crane, proposes a framework for the construction of the tower crane digital twin, and realizes the connection from physical to virtual in the concept of digital twin. The main contributions are divided into three parts: development of tower crane monitoring dataset, tower crane detection and tower crane operation mode recognition. By using labellmg to annotate more than 20,000 tower crane images in 583 tower crane videos, a tower crane image recognition dataset and a tower crane operating mode dataset are established. Yolov5x algorithm is selected in the tower crane detection. Edge extraction is used to improve the quality of the raw dataset and distance-intersection-over union non-maximum suppression is used to replace the traditional non-maximum suppression part in the Yolov5x algorithm to improve the detect accuracy when some tower cranes are overlapping. The final test set detection accuracy rate is 93.85%. After comparing the LSTM and CNN algorithms, 3DResNet algorithm is selected for tower crane operational mode recognition. The raw dataset is augmented by rotating the image by ±10° and ±20°, and the augmented dataset enlarges five times. Using these methods, the final recognition accuracy of tower crane operation mode reaches 87%. These models can be installed on the cctv to monitor the running status of the tower crane in real time and transfer relevant information to the virtual model. The tower crane in the virtual space completes the action of the physical tower crane, thereby realizing the physical-to-virtual mapping in the digital twin.