Currently, a complete system for analyzing the effect of packet loss on a viewer's perception is not available. Given the popularity of digital video and the growing interest in live video streams where channel coding errors cannot be corrected, such a system would give great insight into the problem of video corruption through transmission errors and how they are perceived by the user. In this paper we introduce such a system, where digital video can be corrupted according to established loss patterns and the effect is measured automatically. The corrupted video is then used as input for user tests. Their results are analyzed and compared with the automatically generated. Within this paper we present the complete testing system that makes use of existing software as well as introducing new modules and extensions. With the current configuration the system can test packet loss in H.264 coded video streams and produce a statistical analysis detailing the results. The system is fully modular allowing for future developments such as other types of statistical analysis, different video measurements and new video codecs.