Autonomous incidents: Exploring strategies for monitoring, understanding and learning from failure in autonomous systems (AS)
The project specifically aims to define and map the organisational ‘learning architecture’ that can enable AS safety incidents to be effectively governed and learnt from.
Duration: 2023-2024
Funder: Lloyds Register Foundation and University of York Assuring Autonomy International Programme (AAIP)
Partner: Care Quality Commission
CHILL investigators:
Professor Carl Macrae
Research summary
Background:
A critical part of assuring the safety and social acceptability of autonomous systems (AS) is ensuring that there are robust and trusted processes in place to identify, understand and learn from safety incidents and other adverse events.
One common organisational and regulatory approach to this problem is to implement incident reporting and investigation systems, so that safety incidents can be routinely identified, reported and subjected to rigorous analysis and investigation and system safety can be improved.
However, analysing and learning from AS safety incidents faces a range of challenges including:
- the inherent opacity of some technologies underpinning AS
- the potential for novel and unforeseen failure modes
- the breadth of specialist knowledge and expertise needed to understand AS incidents
- the wide variety of stakeholders and agencies with a responsibility for the safety and regulation of emerging autonomous technologies
This project will explore the strategies and processes that enable reporting and learning from AS safety incidents, focusing on the domain of healthcare where many of these challenges are particularly pronounced.
The project will engage with key national bodies and regulators to identify and define a set of principles, methods and functions that support healthcare organisations and regulators to rigorously analyse and learn from AS incidents.
The project specifically aims to define and map the organisational ‘learning architecture’ that can enable AS safety incidents to be effectively governed and learnt from: that is, the project will identify and characterise a set of practical principles, analytical processes, organisational functions, cultural characteristics and regulatory responsibilities that can support reporting and learning from AS safety incidents.
By working in collaboration with the Care Quality Commission (CQC) and a range of other stakeholders and partners, the project ultimately seeks to better understand the key elements of the social and technical infrastructure required to manage and learn from AS incidents in healthcare.
