An efficient algorithm of the edge detection according to integrating the edge gradient with the average filter is proposed, which can significantly reduce sensitivity of the background subtraction method to noise and illumination. Taking into account the features of the target such as colour, size, etc., a new modified nearest neighbour (NN) algorithm for data association using the target features is designed. A designed interacting multiple model (IMM) filter is utilized to track the maneuvering target motion, i.e. the feature point (called the centroid of the target) motion of the target. The algorithms are validated via an example with natural video sequences. The results show the algorithms are performances and validity for visual tracking. In complex environment, the algorithm can still work well.