Written by Ben Keith
The art of being a professional mariner is about making decisions, whether they pertain to shipping situations, routing and weather, engine configurations, through to threat evaluation and weapon allocation in a military context.
The ‘OODA loop’ (Observe, Orient, Decision, Action) decision-making cycle has been in use by the military for decades. In a human context, assurance of the accuracy of each phase can be achieved through techniques such as observing the process and challenging assumptions. However, when a portion or the entirety of the system-of-systems that delivers the action is automated, assurance of the process becomes difficult, particularly if the OODA loop operates at machine speed.
This article presents an operator’s perspective on the challenges, accompanied by a brief introduction to the technical aspects. It is posited that ensuring the integrity of the Observe, Orient, and Decide functions in a system-of-systems involving AI is vital, and it can be achieved by balancing an objective measure of system trustworthiness with a subjective assessment of users’ confidence in the system.
Decision making – a human context
The OODA loop is a widely known decision-making model originating from the US Air Force. While it was initially employed in military contexts, it has now found applications in various civilian and military fields. The transition from human to machine is seamless, as automation brings the benefits of assimilation of vast quantities of data, rapid assessment, and machine-speed repeatability. However, when operating at machine speed and governed by AI with impenetrable algorithms and overlaid with machine learning, all stages of the decision-making cycle become opaque, with only the ‘action’ being observable.
To place this into context, the OODA loop’s outcome can be illustrated through a human-based scenario like the ‘Rules of the Road’ in maritime contexts. An example is found in the practiced communication of a Royal Navy Officer of the Watch (OOW) during a ‘shipping report:
‘Captain Sir/Ma’am – at red 40, range 1 mile, I have a vessel engaged in fishing [Observation]
we are 30 degrees on their port bow and its bearing is drawing left
with a Closest Point of Approach of 2 cables to port. [Orientation]
It is a Rule 18 situation, and we are the give way vessel. [Decision]
My intention is to alter course 10 degrees to starboard to open the CPA to 5c, it is safe to do so. [Action]
Assurance of that decision-making cycle starts at basic training, and is formalised through qualifications, such as the International Convention for Standards of Training, Certification and Watchkeeping for seafarers (STCW). It is then developed through supervised experience, with defined parameters that the OOW can act within. Assurance that they have oriented themselves based on correct interpretation of all the information would come from questioning the decision and verifying key data – for example, an often repeated phrase on a bridge of a Royal Navy ship is ‘don’t trust the radar – look out of the window’!
There are many softer skills and human factors that also contribute to decision-making assurance, such as the OOW’s levels of sleep, arousal levels, stressors, confidence levels, vocal cues, and past mistakes. Most mariners with experience of Command recount being jolted awake in the middle of the night by something that ‘didn’t feel quite right’ – often called ‘intuition’, but more accurately should be called ‘experience’ – with a rapid re-location to the bridge preventing a more serious incident developing.
It could be assumed that getting to the right ‘Action’ is the point of the OODA loop, which would imply that the journey to get there is immaterial. This holds true only if each incident is treated in isolation, but for an OOW to improve and gain experiential learning (‘machine learning’ in an automated system) throughout their career, the component parts of the OODA loop are just as important. An action which is apparently correct, but based on an incorrect interpretation of the situation is more insidious than an obviously incorrect decision, which can be observed and corrected.
Decision making at machine speed
Substitute AI or ‘autonomy’ into a maritime system-of-systems, particularly in autonomous navigation and collision avoidance, and it is virtually impossible to assure each part of the OODA loop. The decision/action resulting from the OODA loop can be assured through a variety of means, such as repeated testing in live or synthetic environments. Alternatively, if required assurance levels cannot be met, risk can be managed through imposing control, such as limiting operating areas to account for its capabilities or requiring a human to be embarked for supervision.
But how can we assure the ‘Observe’ and ‘Orient’ functions? How do we know why a decision has been made, and therefore assure its repeatability? This assurance becomes increasingly important as the level of automation in a system rises, especially with the incorporation of AI and machine learning.
A vignette from early autonomy trials of a Royal Nany vessel can illustrate the necessity of understanding the entirety of the OODA loop. Whilst transiting an empty Solent on a clear day at speed, the autonomous vessel (jet propelled) came to an instant stop (causing consternation to the human occupants) just before the Coastguard helicopter flew past the bow at 100kts/100 feet. The action of giving way to an unknown, bow-crossing Automatic Identification System (AIS) fitted vessel doing 100kts was probably correct, but it relied on interpretation of an AIS fitted vehicle doing 100kts being on the surface, not airborne.
Trust in automated systems
The key to assuring the safety of an AI-enabled system-of-systems is to build trust in the system and achieve ‘calibrated trust’, which is a balance of:
- User trust – a subjective assessment of how much you trust the system, and;
- System trustworthiness – an objective measure of how much can the system be trusted?
If this is not balanced, there is either too much trust in the system, leading to ‘overtrust’, or a position where the supervisor ‘knows best’, in a position of ‘undertrust’.
History is littered with accidents and incidents relating to both of those situations – ‘overtrust’ resulting in misguided, incorrect or damaging actions taken solely by the system with no human input or supervision, and ‘undertrust’ prompting humans to override system decisions based on flawed judgments of the situation.
‘Calibrated trust’ as a concept can apply to all system-of-systems, including human-centred ones. An example involves a scenario where a RIB recovery operation on a Royal Navy ship, well-planned and executed with an experienced team, spirals into an incident due to the introduction of an inexperienced coxswain. This shifts the balance and leads to ‘overtrust’, endangering the entire ship as the system fails to deliver as expected.
Finding calibrated trust
So how to find ‘calibrated trust’? Frazer-Nash and the University of Bristol have used a systems-engineering approach to design a set of ‘trustworthiness categories’, which can be prioritised for a system based on the Concept of Operations and risk analysis. These categories can be assessed against known and published standards to define a Trust Quality Assessment with performance monitoring and a vital feedback loop.
Work is ongoing in the UK, US and EU to define the set of AI standards and laws against which AI trustworthiness can be benchmarked. Frazer-Nash is working with areas of the military, such as Defence Science and Technology Laboratory (Dstl) to assess applicability in a military context, including Automatic Target Recognition, fire control, and flight control systems.
It is still early days, with technological advances outpacing our ability to regulate and enable. However, there is opportunity for the UK, and in particular the UK maritime industry, to be at the forefront of designing and adopting AI trust standards. This advancement not only enhances technical capability, but also establishes a suitable regulatory regime that maximises innovation opportunity for the UK, while constraining and protecting against nefarious use.
Ben Keith is the Ships Research and Development Business Manager for Frazer-Nash Consultancy, a high-tech, systems-engineering consultancy. He spent 23 years in the Royal Navy serving on the frontline in all classes of ship, in operations from the Middle East through to the Atlantic, as helicopter pilot then Warfare Officer. His career highlight was Captain of the Type 45 destroyer HMS DIAMOND. Post Command, he served in the Royal Navy’s high readiness Maritime Battlestaff, before leaving to work in the emerging technologies of digital transformation and maritime autonomy. Ben is a freeman of the Honourable Company of Master Mariners, and a member of the Society of Maritime Industries Maritime Autonomous Systems Group. He joined Frazer-Nash in 2022, and will be contributing to a panel discussion at Maritime Innovation Week 2023 from 7-9 November.
Register for a free delegate pass at https://yoonity.io/v/maritimeinnovationweek2023
