Chatter is a key factor affecting tool wear, workpiece surface quality and cutting efficiency. When milling thin-walled parts, it is difficult to extract the chatter frequency band due to the time-varying characteristics of the dynamic parameters of the machining system. Variational mode decomposition (VMD) shows good performance in signal processing, but the decomposition result of this algorithm is limited by the influence of initialization parameters. Therefore, this paper proposes a scheme to determine the number of VMD decomposition layers based on the number of Fourier transform frequency peaks. The feasibility of the scheme is verified by the simulation signal and experiment signal. The results show that taking the peak number of the spectrum as the decomposition level of VMD, the spectrum distribution of the decomposed intrinsic mode function (IMF) is clear, and the frequency band containing rich chatter information can be effectively extracted.