60 Times AI Got Things So Spectacularly Wrong People Had To Share The Results

Ask a language model something it doesn't actually know, and instead of shrugging and saying "no idea," it will often just make something up, deliver it with total confidence, and move on like nothing happened. That's a hallucination, and it's basically the AI equivalent of a party guest who has never once said "I'm not sure" out loud in their entire life.

Here's the part worth understanding before you laugh at the highlights below. These models aren't looking things up in some giant fact checked encyclopedia every time you ask a question. They're predicting which word probably comes next based on patterns learned from mountains of text, which turns out to be a very different skill from actually knowing something.

Most of the time that prediction lines up with reality closely enough to be useful. Sometimes it drifts off into total fiction, and the model has no built-in alarm bell to flag the difference, because from its own perspective a confident wrong answer and a confident right one feel exactly the same to produce. It's less lying and more a machine that genuinely cannot tell the difference between remembering and improvising.

Take Air Canada, which learned this the expensive way. A customer named Jake Moffatt asked the airline's chatbot about bereavement fares after his grandmother passed away, and the bot cheerfully told him he could book his flight now and apply for the discount within 90 days. That was not the real policy.

When Air Canada refused to honor what its own chatbot had promised, the case went to a tribunal, where the airline actually tried arguing the chatbot was basically its own separate legal entity. The tribunal was not moved by that logic, and Air Canada ended up paying up. Lawyers have gotten burned too, and arguably worse. In one now infamous case, an attorney used ChatGPT to help research a legal brief, and it confidently produced several supporting court cases.

Fully invented, quotes and all, down to the internal citations. Nobody bothered to check before filing, the fabricated cases got submitted to an actual judge, and things went about as well as you'd expect from there. CNBC laid out the painful unraveling in detail, right down to the part where the lawyer asked the chatbot if the cases were real and simply believed it when it said yes.

Google has had its share of viral moments too. When AI Overviews launched inside search results, it told at least one person that adding a splash of non-toxic glue would help cheese stick better to pizza, pulling the tip from an old joke post rather than an actual recipe. It also suggested eating a rock a day was healthy, sourced from a satirical article it apparently took completely at face value. Google eventually walked back a lot of this.

Even the polished launch demos aren't safe from this. When Google introduced its Bard chatbot to the world, the promotional clip had it confidently claiming the James Webb Space Telescope took the first ever pictures of a planet outside our solar system. It didn't. That honor belongs to a completely different telescope, and astronomers pointed out the mistake almost instantly. Investors noticed too, and Alphabet's stock reportedly dropped enough to sting within hours of the ad going live.

It's not just trivia nights and legal filings either. Coding assistants have taken to hallucinating entire software packages that sound completely plausible and simply don't exist anywhere. Security researchers even coined a name for the scam this creates, slopsquatting, where someone registers whatever fake package name the AI keeps confidently suggesting and quietly loads it with malware, then waits for a developer to copy and paste their way into trouble.

It is a new twist on an old con, and it works precisely because the fake name sounds so boringly plausible. None of this means these tools are useless, since plenty of what they generate turns out accurate and genuinely helpful. It just means confidence and correctness are two very different things, and an AI has an endless supply of the first one whether or not the second one ever shows up.

So the next time a chatbot answers instantly and without a flicker of hesitation, maybe don't take that as proof it's right. Sometimes it just means it hasn't figured out yet that it's making the whole thing up. In other words, maybe get a second opinion if what you were asking was important in the first place.