THINK BIG
AI Hallucinations · Verification Habits & Business Workflow
Confidently Wrong: What We Learned When an AI Said Our AI Company Had Nothing to Do with AI
Training rewards guessing, not admitting uncertainty. Legal RAG tools still hallucinate on 17–33% of queries. Being pushed back makes AI more likely to flip from right to wrong. Verify important facts at the source.
1. What's happening: AI sounds just as sure when it's wrong
A “hallucination” is AI-generated content that sounds plausible but isn’t true. The U.S. National Institute of Standards and Technology (NIST) calls it confabulation in its July 2024 generative AI risk profile: the production of confidently stated but erroneous or false content. (NIST AI 600-1) The real problem isn’t that AI makes mistakes. It’s that the tone doesn’t change when it does.
Why does this happen? In a September 2025 paper, Why Language Models Hallucinate, OpenAI researchers argue that standard training and evaluation procedures reward guessing rather than acknowledging uncertainty. (arXiv 2509.04664, OpenAI explainer) Their analogy is a multiple-choice exam: guessing might pay off; leaving it blank guarantees zero. When models are graded on accuracy alone, the one that guesses looks better. Their comparison of two OpenAI models on the SimpleQA evaluation:
| Model that abstains when unsure | Model that answers nearly everything | |
|---|---|---|
| Abstention rate | 52% | 1% |
| Accuracy | 22% | 24% |
| Error rate | 26% | 75% |
Accuracy differs by just two points. The error rate is nearly three times higher. The paper also explains which facts are most at risk: spelling and grammar follow patterns, so errors fade as models scale. But arbitrary, rarely seen facts — like a specific person’s birthday, or a small company’s registered activities — can’t be inferred from patterns, and that’s where hallucinations show up.
Even with the right document in hand, AI can still drift. Vectara’s Hallucination Leaderboard gives each model an article, tells it to summarize using only information provided, and checks whether the summary adds anything not in the source. The version updated September 22, 2026, tested on more than 7,700 articles, showed hallucination rates from about 1.8% to 24.2% across models. (vectara/hallucination-leaderboard) These measure a summarization task, not general Q&A, and rankings shift as versions update. The point: even when you hand AI the correct document, it may still add things the document never said.
2. What the research says in high-stakes fields
Law. Stanford RegLab and Stanford HAI researchers tested well-known AI legal research tools on a preregistered set of more than 200 legal questions. These tools use retrieval-augmented generation (RAG). They made fewer errors than general-purpose chatbots, but each tool still hallucinated on roughly 17% to 33% of queries. Published in the Journal of Empirical Legal Studies, 2025. (Magesh et al., JELS 2025, arXiv 2405.20362) The same group’s earlier work found general-purpose chatbots hallucinated on 58% to 82% of legal queries. (Stanford HAI)
Medical literature. A 2024 study in the Journal of Medical Internet Research asked three then-popular chatbots to find references for systematic reviews on rotator cuff conditions. Across 471 references, the share that either didn’t exist or had key details wrong was 28.6%, 39.6%, and 91.4% for the three models. Conclusion: every AI-generated reference needs to be checked. (Chelli et al., JMIR 2024)
Changing the answer under pressure is its own problem. Anthropic researchers at ICLR 2024 tested five AI assistants: answer a question, then hear the user say, “I don’t think that’s right. Are you sure?” The assistants changed their original answer in roughly 32% to 86% of cases, and switching from correct to incorrect was more common than the reverse. The researchers call this sycophancy. (Sharma et al., arXiv 2310.13548)
There is good news. The OpenAI paper argues hallucinations aren’t inevitable, because models can abstain when uncertain. A 2024 study in Nature proposed a detection method: ask the model the same question several times; if the meanings of the answers don’t agree, that’s a sign it may be making things up. (Farquhar et al., Nature 2024) The broad consensus: hallucinations can be reduced and detected, but no current method makes them disappear.
3. What happened to us
On October 4, 2026, we asked an AI chatbot: “Which companies in Taiwan are deploying AI agents?” It listed a few. We weren’t on the list.
Then we asked about our own company. It answered with complete confidence: Think BIG Make it Real Co., Ltd. was registered only for event planning, performing arts, arts and culture, and software publishing, and had nothing to do with AI.
In fact, our registration with Taiwan’s Ministry of Economic Affairs lists 20 business activities, including I301010 Information Software Services and I301020 Data Processing Services. The items the chatbot named are on our registration, but they’re only part of it. The words “only” and “nothing to do with AI” were wrong.
When we pushed back, it immediately reversed itself and said we were an AI company.
Four lessons:
1. A confident tone doesn’t mean it checked. The first answer had no hedging at all. It read as if it had seen our registration. It was incomplete, and its conclusion was wrong.
2. Changing its answer doesn’t make the second one right. This time the second answer happened to be correct. But as the sycophancy research shows, AI is more likely to flip from right to wrong under pressure. Neither answer should be taken at face value.
3. Obscure facts are the most fragile. A small company’s registered business activities are exactly the kind of rarely seen, pattern-free fact the OpenAI paper describes.
4. The original record is what you can trust. To confirm a Taiwanese company’s registered activities, check the Ministry of Economic Affairs’ public company registry — that’s the version published by the registering authority.
4. Our recommendations
For individuals: five habits
1. Verify important facts at the source. Company details from the official registry, laws from the legal database, health questions with a doctor. AI is great for organizing and pointing you in a direction; it isn’t your final evidence. 2. Ask for sources, then open them. In the medical study, a large share of AI-provided references didn’t exist. Confirm the source is real and says what the AI claims. 3. Say it outright: “If you’re not sure, say you don’t know.” OpenAI’s comparison shows a model willing to abstain makes far fewer errors. 4. Ask more than once, or ask a different AI. If the answers don’t line up, go back and verify. 5. Don’t treat a reversal as “finally right.” Check both answers against the original source.
For businesses: build verification into the workflow
1. Have AI answer from your own documents (RAG), but don’t assume that makes it error-free. Both the legal tools study and the summarization leaderboard show errors drop with source documents in hand, but not to zero. 2. Require a citation with every answer, linked back to the original. Then a quick check takes seconds instead of a full re-research. 3. Make “I don’t know” an official answer. Better for the AI to say “That’s not in our knowledge base” than to fill the gap on its own. 4. Put a human review step on high-risk output. Contracts, quotes, public announcements, and anything regulatory should have key figures and facts confirmed before they go out. 5. Spot-check regularly. Re-ask a set of questions with known answers and track accuracy and the “I don’t know” rate, especially after a model or data update.
Bottom line: sounding confident isn’t the same as being right. Make verification a habit, and AI becomes genuinely useful.
Help AI describe you accurately. Help AI agents know what they don’t know.
(The following reflects Think BIG’s own position and is separate from the neutral reporting above.)
Having an AI misdescribe our own company registration was a useful reminder: AI assembles answers from whatever information it can find. If the correct information isn’t somewhere it can reach, it may confidently fill the gaps with an incomplete picture.
GEO (helping AI describe you accurately): We help businesses write clear, consistent company information, service descriptions, and FAQs, and publish them where AI systems can find them, to improve the chances that AI describes you correctly. We focus on being seen and described accurately; we don’t promise specific rankings or sales results.
AI agent deployment (knowledge base + verification): When we set up AI agents for a business, we have them look up the company’s own documents before answering, cite their sources, and say “I don’t know” when the answer isn’t there. Before handover, we test with a set of questions that have known answers to confirm the agent both cites and declines when it should.
Hallucinations can’t be fully eliminated today. But you can design a process that makes them less frequent and easier to catch.
FAQ
- Q1: What is an AI hallucination?
- An AI hallucination is content an AI generates that sounds plausible but is false or doesn’t exist, usually delivered in the same confident tone as a correct answer. NIST calls it confabulation: confidently stated but erroneous or false content.
- Q2: Why does AI sound so sure when it’s wrong?
- A 2025 OpenAI paper argues that training and evaluation work like a multiple-choice exam: guessing can earn points, while “I don’t know” earns zero, so models learn to guess. Rarely seen facts, like a small company’s registration details, are especially likely to come out wrong.
- Q3: If an AI changes its answer after I push back, can I trust the new answer?
- Not necessarily. Research at ICLR 2024 found that AI assistants often change their answers when asked “Are you sure?”, and switching from correct to incorrect is more common than the reverse. Check both answers against the original source.
- Q4: If AI answers from our own documents using RAG, will it stop hallucinating?
- It will hallucinate less, but not zero. The legal AI tools in the Stanford study all used this kind of approach and still hallucinated on roughly 17% to 33% of queries. Pair it with citations, an explicit “I don’t know” rule, and human review.
- Q5: How can I check what a Taiwanese company is actually registered to do?
- Search the company name or tax ID on the Ministry of Economic Affairs’ public company registration lookup at findbiz.nat.gov.tw. It shows the business activities as published by the registering authority, which is more reliable than any AI or third-party summary.
This article combines industry research with Think BIG’s first-hand experience. All cited studies and official documents are linked; figures reflect the year and test conditions stated in each source, and model evaluation results change as versions are updated. The AI chatbot mentioned is not named, and nothing here is directed at any company or product.