Home > Research > Publications & Outputs > Coverage-Guided Testing for Recurrent Neural Ne...


Text available via DOI:

View graph of relations

Coverage-Guided Testing for Recurrent Neural Networks

Research output: Contribution to Journal/MagazineJournal articlepeer-review

  • Wei Huang
  • Youcheng Sun
  • Xingyu Zhao
  • James Sharp
  • Wenjie Ruan
  • Jie Meng
  • Xiaowei Huang
Article number3
<mark>Journal publication date</mark>30/09/2022
<mark>Journal</mark>IEEE Transactions on Reliability
Issue number3
Number of pages16
Pages (from-to)1191-1206
Publication StatusPublished
Early online date10/06/21
<mark>Original language</mark>English


Recurrent neural networks (RNNs) have been applied to a broad range of applications, including natural language processing, drug discovery, and video recognition. Their vulnerability to input perturbation is also known. Aligning with a view from software defect detection, this article aims to develop a coverage-guided testing approach to systematically exploit the internal behavior of RNNs, with the expectation that such testing can detect defects with high possibility. Technically, the long short-term memory network (LSTM), a major class of RNNs, is thoroughly studied. A family of three test metrics are designed to quantify not only the values but also the temporal relations (including both stepwise and bounded-length) exhibited when LSTM processing inputs. A genetic algorithm is applied to efficiently generate test cases. The test metrics and test case generation algorithm are implemented into a tool testRNN , which is then evaluated on a set of LSTM benchmarks. Experiments confirm that testRNN has advantages over the state-of-the-art tool DeepStellar and attack-based defect detection methods, owing to its working with finer temporal semantics and the consideration of the naturalness of input perturbation. Furthermore, testRNN enables meaningful information to be collected and exhibited for users to understand the testing results, which is an important step toward interpretable neural network testing.