Answer

This behavior is called hallucination — when an AI model generates text that sounds authoritative but is factually incorrect or entirely made up. It's one of the most important limitations to understand about modern LLMs.

LLMs don't retrieve facts from a database. They predict the next most likely token based on patterns learned during training. This means the model is always generating an answer, not looking one up. When a question falls outside its training data or into an ambiguous area, the model doesn't know to say "I don't know" — it just keeps predicting plausible-sounding text.

Confidence and correctness are separate things in a language model. The same mechanism that makes an LLM fluent and readable also makes it capable of producing wrong answers with total conviction. This is why grounding — connecting models to verified sources or tools — has become such an important area of AI development.