A digital twin is an integrated multi-physics representation of a complex physical entity. This article develops the physical-to-virtual connection of the digital twin and proposes a framework for the construction of a tower crane digital twin. The main contributions of this paper include development of tower crane monitoring dataset, tower crane detection and tower crane operation mode recognition. By annotating >20,000 tower crane images in 583 tower crane videos, a tower crane image recognition dataset and a tower crane operating mode dataset are established. Yolov5x algorithm is used in the tower crane detection, and the test set detection accuracy is 93.85 %. After comparing the LSTM and CNN algorithms, 3DResNet algorithm is selected for tower crane operational mode recognition. The dataset is augmented by rotating the image and the final recognition accuracy reaches 87 %. These models can be installed on CCTV to monitor operational status of tower crane in real time and transfer relevant information to the virtual model. The tower crane in the virtual space completes the action of the physical tower crane, thereby realizing the physical-to-virtual mapping in the digital twin.