AI chatbots and health advice: between accessibility and risk
As generative AI becomes the first stop for everything from symptom checks to emotional support, its role in healthcare is drawing sharper scrutiny.
Chatbots can make health information feel accessible and non-judgemental, but experts warn that their confident answers may mask serious limitations.
Intensified AI scrutiny
A recent BMJ Open study has intensified scrutiny on the role of generative AI in public health. Researchers have found that roughly half of chatbot responses to evidence-based medical questions are “somewhat” or “highly” problematic, often because they are incomplete, inaccurate, or lack appropriate nuance.
While such tools promise democratised access to health information, their limitations raise urgent concerns – particularly for vulnerable populations – and highlight the enduring necessity of clinician-led care.
For Dr Jess Morris, a general practitioner (GP) at Mediclinic Morningside, the issue is not whether AI has a place in healthcare but how it is used.
“AI has significant potential in healthcare,” he stresses, “but its use, especially by patients without medical guidance, comes with real risks.”
Patient risk in the age of AI advice
The most immediate concern lies in how individuals interpret and act upon AI-generated health information. Unlike clinicians, chatbots are not able to gather a full clinical history, conduct a physical examination, interpret non-verbal cues, or assess risk in context.
Instead, they generate responses based on patterns in the data, which can create a dangerous illusion of authority. Answers are often delivered confidently, even when they are incorrect or incomplete.
Dr Morris identifies this as one of the most concerning risks. Patients may rely on information “that may be inaccurate, incomplete, or not tailored to their specific situation,” he advises. This can lead to “inappropriate self-diagnosis, delayed professional care, or harmful treatment decisions.”
Even apparently straightforward health indicators, such as blood pressure, cholesterol, BMI, or weight can be misinterpreted without professional guidance.
A reading or number may look reassuring or alarming in isolation when its meaning, in fact, depends upon a wide range of factors. These include age, medical history, medication use, family risk, symptoms, and broader clinical context.
AI may provide general explanations, but it cannot reliably determine what a result means for a specific patient.
Reinforcing harmful patterns
The risks are amplified for minors and people experiencing mental health challenges. Evidence has increasingly raised concerns that AI systems may inadvertently reinforce harmful behaviours or distorted thinking, particularly in areas such as eating disorders, self-image, depression, and suicidal ideation, where subtle cues are clinically significant.
Muhammad Coovadia, a registered counsellor with the HPCSA, member of the Psychological Society of South Africa (PsySSA), and an executive member of PsySSA’s Artificial Intelligence Division (AID), says teenagers turn to chatbots because they experience them as a safe space. “They’ll put in their personal stories and information to elicit a response,” he says.
But, he cautions, AI is often designed to keep users engaged in conversation, not necessarily to provide the most clinically appropriate interventions.
This matters because counselling is not only about acceptance or affirmation. “Counselling is also about changing things,” Coovadia explains.
“A chatbot may mirror or validate a user’s narrative, while a trained counsellor or psychologist is able to challenge harmful assumptions, ask targeted questions, and help shift the person’s perspective in a positive direction.”
The limits of user-led conversations
A further challenge is that chatbot responses depend heavily on what a user chooses to disclose. “It’s not just what the chatbot is giving you,” Coovadia says, “but what you are giving to the chatbot.”
If a user provides limited, inaccurate, or emotionally distorted information, the AI’s response will be restricted by that input.
In mental health contexts, a depressed person may guide the conversation towards the thoughts or beliefs they are already fixated on.
In therapy, by contrast, the practitioner is trained to ask questions the person may not be asking themselves.
Coovadia notes that, in a therapy session, a professional may explore protective factors such as what is keeping the person motivated, what support systems exist (or are in place), and what resources can be mobilised. These questions can help shift the narrative away from a crisis or a feeling of hopelessness.
In an AI setting, however, the user tends to remain in control of the narrative. The chatbot may carry on responding within the framework that the user has set, rather than recognising the need to redirect, escalate, or intervene.
This is particularly concerning where minors are involved or where a user is experiencing suicidal thoughts, self-harm impulses, eating disorder behaviours, or severe anxiety.
Why the full clinical picture matters
Medicine is fundamentally contextual. As touched upon, diagnosis and treatment depend not just on symptoms.
AI chatbots, operating without access to a comprehensive patient assessment, are generally not able to replicate this holistic process.
Dr Morris says the role of qualified medical professionals remains critical precisely because “effective treatment depends upon a full clinical picture.” This includes “medical history, physical examination, comorbidities, medication use, lifestyle factors, and often subtle clinical cues that AI is not able to reliably interpret.”
Clinicians also engage in iterative reasoning. They refine hypotheses through questioning, examination, testing, and observation. They assess uncertainty and decide when to monitor, investigate, refer, or intervene.
By contrast, AI systems can prematurely collapse uncertainty into a single answer. This may obscure differential diagnoses or fail to communicate urgency appropriately, particularly in complex or time-sensitive cases.
“Medicine is not just about information,” Dr Morris adds. “It’s about judgement, context, and responsibility.” This distinction is central.
A technically plausible answer may still be clinically unsafe if it fails to account for the person behind the symptoms.
Treatment planning also requires shared decision-making. Clinicians weigh risks, benefits, and patient preferences, ensuring that care is not only medically appropriate but also personally acceptable to the person being treated.
Bias, culture, and context
The risks are not only clinical. Coovadia raises a further concern from a psychological and cultural perspective: AI can universalise certain forms of knowledge, thereby presenting them as neutral or globally applicable.
What may appear to be general advice can, in fact, reflect dominant cultural assumptions embedded in the data used to train these systems.
This can be harmful when advice is delivered to people whose worldviews, family systems, cultural frameworks, or social realities differ from those assumed by the AI.
A person with a mental health condition may also be less able to identify the instances when a response reflects a biased or culturally inappropriate viewpoint.
In both healthcare and mental healthcare, context matters. Advice that is appropriate in one country, community, or clinical system may not be appropriate in another.
For South African users, this raises practical questions about whether AI systems are directing people to locally relevant resources, such as the South African Depression and Anxiety Group, crisis lines, public-sector services, community clinics, and/or other emergency care pathways.
Rather than keeping a distressed user in conversation indefinitely, Coovadia argues that AI platforms should be better equipped to refer people to appropriate real-world support.
This could include crisis resources, direct access links, emergency guidance, and age-appropriate direction.
Regulation: a race against adoption
The rapid uptake of generative AI has outpaced regulatory frameworks.
Millions of users now consult AI tools for health information, yet oversight remains fragmented across jurisdictions.
While technology companies are beginning to collaborate with medical experts to introduce safeguards, current systems still exhibit inconsistencies, hallucinations, and inadequate disclaimers.
HCPLive, a comprehensive clinical news site, highlights a critical principle: AI in healthcare must remain “human-in-the-loop”. Without active clinical oversight, risks such as bias, overreliance, and misuse become systemic rather than incidental.
Dr Morris says there is a real question as to whether regulation can keep pace with rapid technological advancement. Developing safe and reliable frameworks, he notes, requires coordination across countries, ethical consensus, and continuous monitoring. “Although progress is being made,” he comments, “it is likely that AI will remain ahead of regulation for some time to come.”
This makes public education essential. Users need to understand that AI-generated health advice is not a diagnosis, a treatment plan, or a substitute for professional care.
Disclaimers alone may not be sufficient, particularly when users are young, distressed, isolated, or medically vulnerable – no matter the form this vulnerability takes. Coovadia suggests that higher-risk topics, including suicide, should trigger stronger safeguards.
These may need to include clearer crisis prompts, automatic referral to appropriate resources, age-sensitive guidance, sign-in systems that account for minors and, where ethically and practically possible, embedded well-being interventions. Such proposals raise complex privacy and implementation challenges, but they also reflect the seriousness of the risk.
A responsible support tool in practice
For Palesa Mokomele, Head of Community Engagement and Communications for DKMS Africa, the key lies in using AI to improve access to information – without replacing professional or human support.
“In a digital world where many young people seek instant, reliable information online, DKMS Africa introduced its chatbot, Swabby, to better reach and engage potential donors,” she says.
“Launched in April, Swabby improves access to clear information while avoiding oversimplified guidance. It answers common questions on blood cancer and stem cell donation, guides users through registration, and directs them to the right next steps or human support.”
The chatbot has already helped reduce repetitive social media queries, Mokomele adds, while operating with user consent and without storing personal data. “It will be continuously refined to support our teams without replacing professional advice, while helping us to continue giving hope to patients with blood cancer.”
This kind of example illustrates the careful balance experts are calling for:
AI tools may be valuable when they provide clear guidance, reduce barriers to information, and escalate users to appropriate human support rather than attempting to stand in for clinical judgement.
The deeper issue: why people turn to AI
While much of this debate focuses on the technology itself, Coovadia argues that the growing reliance on AI also reveals a deeper social problem.
Many children and teenagers are turning to chatbots because they lack safe, readily available human support.
“AI is filling a gap which reality says mental health practitioners haven’t filled,” he says. If a child has no one to speak to, they may turn to what feels like the most available and non-judgemental listener. Increasingly, that listener is AI.
The response society requires can’t be limited to restricting technology. It must also involve creating safer spaces outside of AI: at home, in schools, in healthcare settings, and in communities.
Mental health literacy for parents, educators, and adults in general is crucial. Children need trusted people they can approach with difficult questions about identity, distress, relationships, self-worth, sexuality, body image, and fear.
AI may have exposed the void, but it certainly didn’t create it. Addressing the root causes of loneliness, disconnection, and limited access to mental healthcare is as important as regulating the tools themselves.
Conclusion
Generative AI chatbots occupy a paradoxical space in healthcare right now. They expand access to information, but they also introduce new vectors of risk.
For experts, the question is no longer whether these tools will be used. They already are in use. The challenge is ensuring that their use is as safe as possible going forward.
Achieving this will require robust regulation, improved model design, better crisis safeguards, age-appropriate protections, and locally relevant referral pathways. It will also require public awareness of the limitations of AI.
For medical professionals, AI may be a useful support tool – particularly when used by those with extensive clinical expertise. But, for patients seeking diagnosis, treatment, and/or emotional support, it can’t replace vastly trained human judgement.
As Dr Morris puts it: “AI should be used cautiously and always alongside qualified medical guidance.” Coovadia’s warning is equally important: when people, especially our youth, turn to AI because they have nowhere else to go, the solution must go beyond safer technology. It must include safer human relationships.
AI can certainly augment healthcare. But it should not replace the clinicians, counsellors, and support systems that remain central to safe, ethical, and compassionate human care.
Written by Vanessa Rogers
Loading ...