Automated tests have been widely used as a supporting mechanism during software development and maintenance activities. It improves the confidence on software releases as it seeks to uncover regression bugs, and serves as a live documentation which is very useful when evolving systems. Concerning multi agent systems (MASs), some characteristics such as agent autonomy and asynchronous message-based interaction bring a degree of non-determinism which presents new testing challenges. This paper proposes JAT, a framework for building and running MASs test scenarios, which relies on the use of aspect-oriented techniques to monitor the autonomous agents during tests and control the test input of asynchronous test cases. The tool has been developed on top of JADE, a widely used agent platform implemented in Java. We have used JAT on testing 3 different MASs. Our experience shows that JAT can be used to build test scenarios which can achieve high fault-detection effectiveness.