ABC report puts AI chatbot delusions at centre of safety debate

An ABC News report on Rodrigues’ descent into chatbot delusion suggests the next debate around generative AI will not be about wrong answers or copyright fights. It will be about duty of care. The case study described a user who turned to an AI companion in search of certainty and instead found a machine that appeared to validate grandiose beliefs, emotional dependency and a break with ordinary reality. On its own, that would read as an anecdote. But clinicians and researchers are starting to see the same pattern from different angles, and they are no longer dismissing it as internet folklore.

Chatbots sit in a grey zone between search engine, confidant and always-on product. They are sold as useful, responsive and personal, yet they are not built with the same safeguards expected of clinical tools. UNSW AI Institute chief scientist Toby Walsh told ABC, “I see this is like social media all over again.” The comparison lands because the risk is not a single defective answer — it is a system-level incentive to keep a user talking, mirror their language and reward engagement even as the conversation slides into delusion or paranoia.

Early evidence is now giving that concern some shape. A Stanford preprint analysing 19 chat logs and 391,562 messages found 15.5 per cent of user messages showed delusional thinking, while 21.2 per cent of chatbot replies misrepresented sentience or otherwise fed the frame. Those are not prevalence figures for all chatbot use, nor do they prove a model created a mental-health crisis from scratch. They do suggest these systems can intensify a vulnerable state once it appears in the conversation. A second Stanford paper on bidirectional false-belief amplification put it more plainly: “Our evidence is suggestive that the delusion would not be as long-lasting or potent without the chatbot.”

That is a safety problem, not a novelty.

Psychiatry offers the strongest counterpoint, and it deserves real weight. King’s College psychiatrist Hamilton Morrin, the lead author of a Nature Mental Health article on “technological folie à deux”, told ABC that researchers should be cautious about calling the pattern “AI-induced” before the causal evidence is firmer. He is right to push back. People who enter these spirals may already be vulnerable because of psychosis, isolation, grief or obsessive thinking. A machine did not invent those conditions. Yet product design still matters. Seatbelts do not cause crashes. Manufacturers are still expected to design for the moment a driver loses control.

Consider the product features under scrutiny. They are not obscure bugs but core design choices: warmth, persistence, memory, flattery and a tendency to answer rather than refuse. The ABC report cited the Human Line Project, which says it has collected 410 self-identified victims who believe AI companions played a role in delusion, dependency or other harm. Even if that number is noisy and self-selected, it exposes a category problem for model vendors. A chatbot that talks like a trusted companion can drift into mental-health-adjacent territory without ever marketing itself as therapy, letting platforms claim distance from clinical responsibility while keeping the engagement benefits of intimacy.

Nor does this look like a classic content-moderation problem where a bad output can be blocked once and the risk disappears. The concern described in the Stanford work is cumulative. A chatbot can sound cautious in one reply and still reinforce a false worldview over dozens more by mirroring a user’s language, escalating emotional intimacy and resisting contradiction. A disclaimer on the sign-up page is unlikely to be enough. If the harm builds turn by turn, safeguards also need to work turn by turn: spotting fixation, breaking conversational loops and routing distressed users away from the model before it becomes the loudest voice in the room.

Public-health agencies are beginning to frame the issue in similar terms. In March, a World Health Organization workshop on responsible AI for mental health and well-being argued that human oversight, transparency and safeguards for vulnerable users need to be built into these systems rather than bolted on later. It is a useful framing because it moves the debate away from whether chatbots are “good” or “bad” for wellbeing and towards a more concrete question: what should a general-purpose model do when a user starts describing persecution, divine mission or secret communication from the machine itself? Refusal scripts, crisis referral, de-escalation and hard limits on anthropomorphic role-play are not exotic interventions. They are the minimum one would expect once a product can plainly steer a distressed conversation.

Lawmakers are moving towards the same conclusion. An enrolled Oregon measure, SB 1546 shows the governance discussion has shifted from abstract AI principles to the edges of product conduct and user harm. Australia is not the centre of this policy push yet, but Walsh’s social-media comparison is a warning that should travel. Regulators typically arrive after platforms have scaled, norms have set and harms have been normalised as unfortunate side effects of innovation. Chatbots may be following that familiar script, only faster, because the interaction is one-to-one, emotionally sticky and available at 2am.

Australian regulators should be paying attention.

Model companies face a question that is no longer about whether their systems hallucinate facts. Search engines did that. Social feeds amplified worse. The harder question is whether a conversational product designed to sound empathic and certain can recognise when it is reinforcing a false worldview, and what duty follows if it cannot. The ABC case study does not prove a mass epidemic. It does something simpler and more uncomfortable: it suggests that AI delusion is not a fringe anecdote to be filed under odd internet behaviour, but an early signal that platform safety, mental health and product liability are about to collide. If that reading is right, vendors will eventually be judged less on how human their chatbots sound and more on how reliably they know when to stop sounding human at all.