AI has shifted from “People trying AI” to “People depending on AI regularly”. It is commonplace now that emails and documents are drafted by AI, without adequate human oversight – leading to embarrassing real-life consequences. ChatGPT grew from 100 million active weekly users in November 2023 to a staggering 900 million in 2026.
AI use and dependency is likely to only increase over time, which necessitates awareness of its inherent risks to develop appropriate safeguards.
AI has rapidly become embedded in professional environments, but offering significant efficiency gains, a fabricated AI response (hallucination) or document undermines progress and causes much more harm than good.
Many users mistakenly treat AI like a database, when in fact it is a probabilistic generator. This misunderstanding leads to unfortunate incidents where legal professionals cite case law which does not exist and more recently, the government having to withdraw its own policy document on AI governance, because it cited fictitious sources – embarrassingly drafted by AI. An analogy comes to mind: ‘asking the prisoners how they would like to be guarded’.
There are two types of AI – logic based and Large Language Models (LLM’s). LLM’s learn by idea association and operate through intuitive learning. The engine of modern LLM’s such as ChatGPT, acts through pattern completion instead of truth retrieval and is unequipped with a mechanism to distinguish between what is ‘true’ and what is ‘plausible’. The noted objective of the system is to reach a goal: to provide the most helpful and relevant answer. The dichotomy of course is that if citing a false source or making up a false line or reasoning, it is quite the opposite of helpful or relevant.
Incorrect AI produced output relied on by professionals can (and has) lead to incorrect choices, legal exposure, and reputational damage. These incorrect outputs are occasioned by AI hallucinations – where systems generate fabricated or misleading information. Hallucinations can produce entirely fabricated yet realistic outputs, including fake references and data. This makes them particularly dangerous in decision-making environments where accuracy is critical. In professional contexts, hallucination becomes not just a technical flaw but a governance challenge.
An appropriate Turkish proverb: “When the axe entered the forest, what did the trees say? Look, the handle is one of us”. The mind does not question what comes from a trusted voice – it receives it as truth. Often, trust misplaced in someone familiar can lead to destruction as one fails to recognise the threat until it is too late. This is precisely what occurs when AI reaches a tipping point where outputs shift from being accurate and trusted to fabricated. The question then is why do they lie and when?
According to a research paper published by D & N Restrepo [March 2026], AI hallucinations are not a random glitch but a foreseeable consequence of the technology’s design. Generative AI works by predicting the most likely next word or phrase, not by understanding what is true. The problem is that it can give several correct answers in a row, building trust, before suddenly producing something completely false.
Initially, the model provides harmless repetition and then shifts to valid reasoning. At this point, just when the user perceives the system to be reliable, requiring AI to resolve a complex, novel or unsettled question pushes the model into a region where training data is sparse, leading to a fabricated response. The more complex or uncertain the question, the greater the likelihood of hallucination. So, we know when it will lie.
Users have previously relied on the “black box” defense to avoid accountability. This defense is premised around a misunderstanding of the technology and that users were unaware that AI may hallucinate and create fictitious responses. But as consensus is that AI-generated falsehoods are a foreseeable engineering risk, rather than an unforeseeable lie, the “black box” defense is unlikely to succeed. This perspective paves the way for a more rigorous standard of technological competence and diligence across all industries.
A user is expected to understand, at least to a reasonable degree, how generative AI works, its limitations, and its propensity for fabrication before relying on it. User competence now not only involves the ability to use the system but also requires a practical understanding of how that software can fail. Moreover, the user must still verify the AI generated response even if the initial response appears to be accurate.
With that in mind, the ultimate party responsible still rests on existing professional responsibility. Human judgement remains essential not only for ensuring accuracy and reliability but also serves as a pre-requisite for Professional Indemnity cover. It is therefore of paramount importance that professionals consider the risks associated with relying on AI generated responses.
Santam is an authorised financial services provider (FSP 3416), a licensed non-life insurer and controlling company for its group companies.
