In this paper, an unsupervised scheme for stereo video matting is presented, where stereo motion analysis is combined to provide an automatic multi-layer clustering scheme of alpha components. With this multi-layer matting scheme, objects in both foreground and background can be extracted for background substitution. The experiment shows that the proposed scheme can have a better performance in term of automatic grouping of alpha components in comparison with min-cut matte grouping.