Q&A: AI hallucinations in healthcare

Q: What are artificial intelligence (AI) hallucinations, and why do they occur?

A: AI hallucinations refer to instances where an AI system produces inaccurate or invented content without any basis in the source data. For example, an AI tool might cite a clinical guideline that doesn’t exist or misinterpret a provider’s documentation when composing an appeal letter. AI tools can even hallucinate while citing legitimate sources. While these errors may appear minor, they can have significant consequences in the highly regulated world of healthcare reimbursement.

AI hallucinations occur because the large language models that power these tools generate text by predicting patterns based on vast amounts of training data—not by fact-checking against authoritative sources. AI does not “understand” the information it ingests; it merely uses the data to predict the word that statistically is most likely to come next. When the model encounters gaps in context or ambiguous prompts, it may “fill in the blanks” with plausible-sounding but incorrect information.

In healthcare settings, security restrictions can amplify this issue. Many organizations block external AI tools or limit access to real-time clinical databases to comply with HIPAA and cybersecurity policies. This means the AI often works with incomplete or outdated context, increasing the likelihood of hallucinations. In short, some hallucinations may stem not only from the model’s design, but also from restricted data environments created for patient privacy and security.

Essentially, hallucinations are a byproduct of how generative AI works—it prioritizes linguistic coherence over factual accuracy. This makes human validation essential, especially in compliance-driven fields like healthcare.

Editor’s note: This Q&A was excerpted from “Trust but verify: The hidden risk of AI hallucinations in appeals,” a CDI Journal article written by Karen R. Lane, MSN.ed, CCDS, CCDS-O, CDIP, RN. Opinions expressed are those of the author and do not necessarily reflect those of ACDIS, HCPro, or any of its subsidiaries.