Home > Research > Publications & Outputs > Trust, systems and accidents

Links

Text available via DOI:

View graph of relations

Trust, systems and accidents: Designing complex systems

Research output: Contribution in Book/Report/Proceedings - With ISBN/ISSNChapter

Published
Publication date2003
Host publicationProcs 10TH IEEE Int Conf and Workshop on the Engineering of Computer-Based Systems
PublisherIEEE COMPUTER SOC
Pages31-38
Number of pages8
ISBN (print)0769519172
<mark>Original language</mark>English

Abstract

Trust is easier to destroy than create, hence, trust in overly applied automation may erode following accidents. This paper looks at the crash of a revolutionary supersonic fighter that resulted from over-reliance on protection technology. The protection system has been automated to the extent that it was impossible for the pilot to regain control and convince the system that there was a problem. Complete trust in the safety of the system has thus been translated into a new kind of computer-assisted error, where in the interest of safety the only possible exit strategy is outlawed by the system leading to the potential (or in this case, the actual) destruction of the system it was meant to protect. Overall, trust appears to be an emergent function that takes in safety issues affecting the entire system as well as the relationship between the product, client and developer and the trade-offs that underpin decisions. Operators, working as part of a system, abrogate some of their responsibility by implicitly or explicitly passing on control to computerised systems. While operators function as an integral part of systems, their trust in the ability and safety of the system plays a key role in the ensuing success or failure.