In this article, a cost-effective ultrawideband (UWB) communication system for gesture recognition in a smart home environment is proposed, which uses gesture trajectories and a deep learning model. Most previous studies of gesture recognition using the UWB technology used electromagnetic signals directly, which may bring problems, such as radar clutter, signal coupling, multipath, fading, and interference. However, instead of using UWB’s high-frequency pulse signals, the proposed method only uses gesture trajectories by data positioning. To this end, first, a data set of four gesture activities was created. Then, this data set was trained using a convolutional neural network (CNN) integrated with a squeeze-and-excitation (SE) block, namely, the SE-Conv1D model. Finally, the system was prototyped to interact with appliances in practical smart homes. The experimental data was used to demonstrate the superiority of the SE-Conv1D model in comparison with four baselines: 1) support vector machines; 2) K -nearest neighbor; 3) random forest; and 4) binarized neural networks. Experimental results show that all collected gesture activities are correctly recognized with an overall accuracy of over 95%, among which the proposed SE-Conv1D model achieves the best accuracy of 99.48%. The proposed system is a complete end-to-end sensing system specifically designed for tracking and recognizing human gestures, which is robust against interference and changes in distance or direction. In addition, the proposed system can tackle the device selection problems for smart homes, which means it is reliable for real-world applications.