This paper proposes a technique for group object motion estimation based on evolving graph networks. The main novelty over alternative group tracking techniques stems from learning the network structure for the group. An algorithm is proposed for automatic graph structure initialisation, incorporation of new nodes and unexisting nodes removal in parallel with the edge update. This evolving graph model is combined with the sequential Monte Carlo framework and its effectiveness is illustrated over a complex scenario for group motion estimation in urban environment. Results with merging, splitting and crossing of the groups are presented with high estimation accuracy.