Co-authored by Dr Raymond Sheh (Adjunct Associate Research Scientist, Johns Hopkins University and Guest Researcher, NIST).

The universe and life is full of chaos. Cutting down on that chaos requires careful ‘checking in’ at all stages of a system’s lifecycle. Even people need regular check-ins to see how they are going, to see if they are doing what they and others expect.

Artificial systems and processes need check-ins too, such as car servicing or home maintenance. Cutting down the chaos requires making sure that they are doing what we want them to do, that they are performing as expected, and that they are doing so in an efficient manner.

So, what do ‘check-ins’ for Artificial Intelligence, or AI, look like? And what are the risks of not keeping on top of them?

First check-in

The first check-in is at the start, even before we select what we want. We need to perform a well-informed requirements analysis on our problem.

For example, when we initially buy a vehicle, we look at what we want to use the vehicle for. Are we going to cruise around with no kids or are we carting kids to and from school and events? Are we going off-road or for short runs? All of these speak to what kind of vehicle we purchase. A Ferrari vs a Wagon, or a 4WD vs an Electric Vehicle.

With AI this means gaining a good understanding of what you want the AI to do, what regulatory and safety requirements may exist, how to measure that it is doing the job right and, perhaps more importantly, how it might be doing the job wrong. For going off-road, we know the vehicle we select is not appropriate if we can’t drive over gravel – immediately discounting the Ferrari from our possible selection. When picking an AI product, it is important to know these requirements and indicators of appropriate, and perhaps just as importantly, inappropriate behaviour.

Finally, we need to be mindful of our acceptable levels of risk, be it in terms of safety, risk of failure/breakdown, and risk of other less consequential things, like the paint peeling in our vehicle example. Perhaps we often drive children around in heavy traffic and could benefit from a car with a higher safety rating and more assistive features. However, this may need to be balanced against resource limitations and reality. The features that provide the best safety may not be available in a vehicle that has other required features, at a price that is affordable. Likewise, making a realistic assessment of requirements that trade-off all these factors is critical in AI.

Second check-in

The next check-in is how to manage, operate, maintain, and track the condition of the AI.

Just as after purchasing a car, consideration of servicing, petrol, cleaning, new tyres, periodic safety inspections, and so on needs to be considered, scheduled, and undertaken; so too does this need to happen with AI. Before purchasing a car, it is advisable to calculate if these ongoing costs are understood and funded and if our acceptance of low probability, high impact risks, such as a blown tyre or flood damage, might require setting aside additional funds or purchasing additional insurance.

With AI again, these ongoing costs need to be considered and traded against the acceptable risks of the AI not performing as desired. Overheads might include ensuring that:

• The AI application remains up to date and patched against vulnerabilities;
• System administrators are trained and up to date;
• Operators and business end users are trained and able to use the system and, crucially, report any deviations from expected behaviour;
• Those managing and monitoring any AI training data sets understand and monitor for any bias risks or ‘dirty’ data;
• The AI system has ongoing maintenance, includes regular checks to determine if the AI is still running as expected and can be explained to the extent necessary, be it for ever-changing legal, regulatory, or “social license to operate” reasons.

Ongoing maintenance is one significant factor that is often forgotten. The ‘I’ in AI often (although not always) implies the ability to “learn” and change, develop and “grow” – a process sometimes called “lifelong learning”. Unlike humans, AI doesn’t “grow up” with a culture and society that imparts ethics or morals that bind its behaviour and shape its interpretation of new information. Rather, AI “ethics” and “morals” are artificial constructs as a product of its development and rules. These are often inherently flawed, especially when the developers of the AI system don’t specifically consider them. Even when they do, these rules and bounds reflect what is known and recognised by the developers at that point in time. If they are unknown, accidentally forgotten, or were not even an ethical consideration at the time of development, they would not be included. Some systems may be able to learn some of this through training data but there is a substantial risk that it does not do so or does so incorrectly.

Failure to consider this can result in AI outputs that are biased or different from expected. Examples of this risk being realised have been in the recent media including this ABC News report on the hidden dangers of AI. Therefore, during AI’s “growth”, checks need to be made to ensure the system has not “grown” in such a way that it is making unethical decisions or recommendations. The reinforcement of human bias and discrimination may be a product of innocent seeds at development and subsequent raw datasets, and these must be managed. Not all AI systems can be monitored and traced in this way and even fewer can have such errors corrected in a rigorous manner.

The risk to an organisation can be significant. If the system makes an error that exposes the organisation to legal risk, and there is no way to correct the system to the satisfaction of the regulators or the community, use of the system may no longer be permitted. If this risk is important in an application, the ability to trace these decisions and correct errors must be a requirement during the development or procurement of the system.

Final check-in

And, like all things, AI systems must come to an end. The end of life for any system is difficult. From ensuring that any dependent business processes can continue, transitioning to a newer system or toolset, to managing historical and current data and information. Returning to the car analogy, when changing cars this is moving all the “old stuff’ like glove box contents from the old car to the new, understanding that the fuel tank is on the opposite side, and other nuances. We can easily decouple from the old car as we understand what the old car does and understand how the new car can carry on those functional requirements. This is where decoupling from AI becomes more difficult.

One of the key differences between a car and an AI system is that a human makes decisions for a car, however AI is designed to assist humans in making decisions. Its purpose is to be embedded into the human decision-making process to alleviate human effort. From the creation of papers, books, and articles by improving decisions on content, to deciding when to turn a circuit on or off or advising on the allocation of limited resources to optimise a particular process. Where AI has become part of the business process, critical business information is, by definition, incorporated into the AI system. This is particularly an issue for AI systems that include Machine Learning (ML) components.

Non-ML decision assistance or autonomous systems make decisions or recommendations based on rules that are set at the time that the system is created or may be traced by understanding the data that has been fed into the system. Of course, such AI systems may still be highly complex and pose a challenge to decipher when transitioning to a new system, just like any other highly complex system.

AI that includes ML, including Deep Learning, and Generative AI systems that create content, such as ChatGPT, adds an additional layer of challenges to overcome when transitioning to a new system. ML systems are designed to change and grow their own algorithms in response to the continuous flow of data to analyse, be it at its initial training or, for systems that perform lifelong learning, throughout its deployment. Eventually (and sometimes quite soon), the decision-making processes of the ML components of the AI system are no longer traceable at a level of abstraction that is meaningful to humans. It is like a superhuman expert who has had initial training in a critical process, and then over time has gained experience and knowledge and now performs the job very well but cannot explain how they do it. This is unsurprising of course – the purpose of the AI system is to do what is hard for humans to do at the desired scale. If it were easy to codify what the AI system is doing, chances are we wouldn’t need to use an AI system.

At end of life this becomes a problem. If decisions have been made with the assistance of the AI output, but understanding of how the AI developed those recommendations or decisions is unknown, is it possible to correctly specify the requirements for a replacement and transfer what has been learned? Can the impact of removing the AI system on further decision making be quantified? This would be the equivalent to buying a new car and, suddenly, not knowing how to get from A to B anymore and being unable to determine if the old route was due to distance, fuel, or time, and yet still needing to maintain that same level of service to customers in a seamless transition.

It is too late to consider such risks when the system is already nearing end of life. If it is anticipated that such end-of-life risks are important, it is vital that the system be developed or procured with the ability to transition to a future system as one of the requirements. This may substantially exclude some potential AI systems, just as the need for child seat restraints may mean excluding 2-seater sports cars.

Need help with check-ins?

The ability to assess and quantify AI risks is relatively new compared with other technology risk considerations such as safety and security.

This is where Anchoram is well placed to assist in assessing and incorporating AI risk into existing risk and assessments. Specifically, we have capability and knowledge in AI risk including within our team who have contributed and influenced the NIST AI Risk Management Framework released in 2022.

Together our team has combined decades of risk experience across all industries including critical infrastructure, mining and government allowing a depth of risk knowledge and demonstrated ability to apply and incorporate new and emerging technology. With an understanding of AI risk, we are able to provide advice on how to access AI opportunities balanced with managed risks to achieve optimal use of AI in an organisation.

Interested to find out more? Reach out to Karen or our team to learn how Anchoram can help your organisation maintain a balanced and managed approach to AI risk.