Why does an AI model make up an answer?
Because the way models are trained and graded rewards guessing. That is the conclusion of Why Language Models Hallucinate, a paper by Adam Tauman Kalai, Ofir Nachum and Edwin Zhang (OpenAI) with Santosh S. Vempala (Georgia Tech), dated 4 September 2025 — all four affiliations are printed under the title on the first page. Their analogy is a student in an exam: if leaving the answer blank scores zero and a plausible guess might score, guessing is the rational move. In their words, models "guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty", and this persists "due to the way most evaluations are graded".
In other words: the model is not lying out of malice and it is not broken. It is doing what it was taught to do. The practical consequence for your business is that no provider can promise you their AI will never get something wrong. What can be done is to build the system so that when it is unsure, it stops and hands over.
So can it be prevented completely?
No. It can be greatly reduced, and it can be fenced in. A properly built customer service assistant does not work like the generic chat app on your phone: it does not answer from memory. It answers with your data in front of it — your catalogue, your opening hours, your prices, your terms — and with a clear instruction that anything not in there does not get answered. The difference is not the model. It is how it is built around the model.
We say exactly that on our 24/7 customer service page: what it does not know, it does not invent. And that is a design decision, not a marketing line: you have to choose between an assistant that always has something to say and one that sometimes says "a person will look at this". The second one is worse in a demo and much better in real life.
What does answering "with your data" mean, and why does it change the risk?
It is the difference between asking someone from memory and letting them check the manual first. The technical name is grounding: before answering, the system looks something up in a source of yours — the catalogue, the diary, the FAQs — and only uses what it finds. Three consequences you notice:
- If the fact is not there, there is no answer. The assistant hands over instead of filling the gap.
- If the fact changes, the answer changes. Update a price in your system and the assistant says the new one, with nothing reprogrammed.
- The answer is auditable. You can check where it came from, which turns a complaint into something reviewable instead of an argument.
What does a well-built assistant do when it does not know something?
It says so and passes the conversation to a person, with everything said already logged. That is exactly what is running at Dartel Solutions, a security company: there, an invented detail about a piece of equipment or an installation is not a customer service slip, it is a technical problem. So the system is built to hand over rather than to guess.
The general rule, for any sector: the assistant can answer what is written down and collect what a person needs to finish the job. What it must never do is commit to a price, a deadline or a diagnosis it cannot see in a real piece of data.
What should you ask whoever builds your assistant?
- Where do the answers come from? If they cannot name the source — a catalogue, a document, a sheet — it is answering from memory.
- What does it do when it does not know? The good answer is "it says so and alerts a person". The bad one is "it always finds something to say".
- What is it forbidden to say? There should be a list: prices that are not published, deadlines, medical or legal advice, promises.
- How is it tested? Ask to see the set of questions they tested it with, trick questions included.
- What happens to the conversations? Where they are stored, for how long, and who can read them.
- Can I read what it has answered? With no log, there is no way to know whether it works.
The six questions worth asking before hiring any agency, not just for this, are in how to choose an AI automation agency.
How do you prove an assistant is not making things up?
With questions you know it cannot answer. This is the part of the work nobody sees, and the part that separates a serious assistant from a pretty demo:
- Off-catalogue questions. A product you do not sell, a service you do not offer. It has to say it does not have it.
- Trick price questions. A discount nobody authorised. It cannot improvise one.
- Deadline questions. "Can I have it tomorrow?" when the system cannot actually see your diary.
- Questions in another language. Plenty of faults only show up when the language changes.
- The same question ten times. If the answer drifts between attempts, the system is not reading a fact — it is improvising.
And after that, every so often, somebody reads real conversations. An automated check tells you the flow is working; only reading it tells you whether the answers are any good.
Who is legally responsible for what it says?
You are, as far as your customers' data is concerned. The Spanish data protection authority publishes an introductory guide on bringing AI-based processing into line with the GDPR — from February 2020 and still marked "under review" on its cover — covering what has to be in place around information to the individual, legal basis and automated decisions, and it has also published, more recently, guidance on agentic AI for systems that act on their own. If an assistant is talking to your customers, those pages come before the signature, not after.
And if it still answers something wrong?
It happens, and it has to be planned for. What marks out a sensibly built system is that the fault gets caught and fixed: the conversation is logged, you can see which piece of data caused it, the source gets corrected, and that question joins the test set so it does not come back. What cannot be done is promising it will never happen. Anyone who promises that has either never built one, or has not read what the research says about why it happens.
Frequently asked questions
Can you guarantee the AI will never make something up?
No, and anyone who guarantees it is not telling you the whole truth. What can be done is to fence it in: answer only from your data, forbid committing to prices or deadlines it cannot see, and hand over to a person when the fact is missing.
Why does a language model invent answers?
Because normal training and grading reward giving a plausible answer over admitting a doubt, much like an exam where a blank answer scores zero. That is the argument in Why Language Models Hallucinate, by researchers at OpenAI and Georgia Tech, published in September 2025.
What does the assistant do when it does not know something?
It says so and passes the conversation to a person, with everything said already logged, so the team picks it up with context instead of starting from scratch.
Could it quote a customer the wrong price?
It should not be able to: any price it gives has to come from a source of yours, and anything not published does not get answered. If an assistant improvises a discount in testing, it is badly built.
How do you check that it answers well?
With a set of trick questions before it goes live — a product that does not exist, an unauthorised discount, a deadline it cannot see, another language — and by reading real conversations every so often. An automated check tells you the flow works; only reading it tells you the answers are good.
Who is legally responsible for what the assistant says?
The business using it remains responsible for its customers’ data. The Spanish data protection authority publishes an introductory guide — from February 2020 and still marked under review — on bringing AI-based processing into line with the GDPR, and it is worth reading before anything is connected to a customer database.
Zerolagia builds assistants that answer from the real data of the business and hand over when they do not know. Free website check here.
Talk to us: info@zerolagia.com · +34 612 48 50 87 (WhatsApp).
You might also like