A language model predicts plausible text. When context is insufficient, it may invent details or figures and present them as fluently as accurate content. Grounding lowers the likelihood but never eliminates it.
The new risk agents introduce
A chatbot's error remains within the conversation. An agent may write the fabrication into records, tickets, and downstream documents, which other systems then treat as authoritative input before any person reviews it.
Contributing factors
- The model has no external ground truth to check against, and its internal uncertainty signals are weak, poorly calibrated, and rarely exposed to the user.
- Ambiguous or incomplete prompts, retrieval gaps, and outdated knowledge bases lead the model to supply plausible inventions.
- Fluent, confident phrasing conceals errors and reduces reviewer scrutiny.
Examples
- A customer support assistant cites a refund policy clause that does not exist, and the incorrect guidance is copied into an internal procedure document.
- A clinical documentation agent invents a past procedure when summarizing a patient record, and the fabricated history is saved to the chart.
- A coding assistant references a library function that was never defined, and the call enters a build before review identifies it.