Evaluating AI Output Critically

If you ask a room full of people to describe the character Rich Uncle Pennybags — the mascot of the board game Monopoly — many will confidently tell you about his top hat, his white mustache, and his monocle. They can picture that monocle clearly. They might even mimic the gesture of him adjusting it. But Rich Uncle Pennybags has never worn a monocle.

This is the Mandela Effect: a phenomenon where large groups of people share a specific, vivid, but completely false memory. We see it everywhere. People swear Darth Vader said, “Luke, I am your father,” when the line is actually, “No, I am your father.” Millions of people remember the children’s book characters as the Berenstein Bears, even though they have always been the Berenstain Bears. For decades, these were just quirks of human psychology — odd glitches in how our brains reconstruct the past. But today, we are building a technology that can turn these glitches into a permanent, automated reality.

Imagine a scenario where a user generates a hyper-realistic AI video of a 1990s cartoon — let’s say, a clip showing the Monopoly Man adjusting that non-existent monocle. They post it online with the caption: “Finally found the lost episode! I knew I wasn’t crazy.” Within hours, the video goes viral. Thousands of people, validated in their false memory, share it. Then, the second layer of AI kicks in. Automated content farms — websites run entirely by artificial intelligence — scrape the trending topic. They churn out hundreds of articles with headlines like “The Monocle Mystery Solved: Lost Footage Resurfaces.”

Now, the loop closes. The next time you ask an AI search engine, “Did the Monopoly Man wear a monocle?” it scans those new articles. It sees the “evidence.” It sees the “consensus.” And it confidently answers: “Yes, while widely debated, footage from 1992 confirms he wore a monocle in early editions.” We can classify this loop as a Consensus Hallucination Consensus Hallucination A feedback cycle where AI output becomes AI training data, validating a popular falsehood until it becomes indistinguishable from fact. The AI confuses popularity with truth rather than identifying a verifiable reality. . It is what happens when an AI model confuses popularity with truth. In this loop, there is no “Ministry of Truth” rewriting history; there is just a feedback cycle where AI output becomes AI training data, validating a lie until it becomes indistinguishable from a fact.

The risk is so potent that the term we are using here — consensus hallucination — was actually coined by the AI during the drafting of this book, an incident we will analyze later as a case study in critical verification. The model presented the phrase as established academic theory, illustrating exactly how persuasive these fabrications can be.

The greater risk, however, is often quieter than a viral hoax. Because AI models are trained on vast amounts of human communication, they do not just learn facts; they inherit our cognitive shortcuts and informal fallacies. They frequently confuse popularity with truth. They defer to authority figures without question. They prioritize plausible-sounding arguments over rigorous evidence. In short, the machine shares our bad habits. To use it well, you cannot simply check its math; you must be ready to audit its reasoning, recognizing that its output often reflects the same cognitive biases that plague human debate.

From Logical Fallacies to Hallucinations

If you have taken a composition, speech, or philosophy course, you may already be familiar with the tools used to evaluate human arguments. However, if these concepts are new to you, they are essential for understanding how your digital partner “thinks.”

To evaluate information effectively, we must look for two specific types of human error:

• Logical Fallacies Logical Fallacies Structural errors in an argument where the reasoning itself is broken, even if individual facts might be true. Examples include Post Hoc, Straw Man, and Non Sequitur. : These are structural errors in an argument. They occur when the reasoning itself is broken, even if the facts might be true. For example, the Post Hoc fallacy assumes that because Event B happened after Event A, Event A must have caused it.

• Cognitive Biases Cognitive Biases Psychological shortcuts our brains take to process information quickly. They are errors of perception rather than errors of logic — for example, Confirmation Bias is the tendency to notice and trust information that supports what we already believe. : These are psychological shortcuts our brains take to process information quickly. They are not errors of logic, but errors of perception. For example, Confirmation Bias is our tendency to notice and trust information that supports what we already believe while ignoring evidence that contradicts it.

We often assume that computers, being mathematical systems, are immune to these messy human problems. We expect AI to be cold, objective, and purely logical. But this is a dangerous misconception.

Generative AI models are not databases of truth; they are prediction engines trained on vast amounts of human language. They have read our books, our articles, our internet forums, and our debates. Because they learn from us, they do not just learn our facts — they inherit our flaws.

When an AI model generates a “hallucination” — a confident but false statement — it is often not a random glitch. Instead, the AI is successfully replicating a pattern of Logical Fallacy or Cognitive Bias present in its training data. If humans frequently use a specific fallacy to argue a point, the AI learns that fallacy as a “standard” way to construct an argument. If humans historically display bias against a certain group in the training texts, the AI will statistically predict that bias as the “correct” context.

In this sense, an AI hallucination is often just a high-tech mirror. It reflects the illogical or biased patterns of human thought back to us, stripped of the human context but presented with the machine’s absolute confidence. To catch these errors, you cannot simply look for “computer glitches.” You must learn to spot the very human failures that the machine is automating.

Hard and Soft Hallucinations

When we talk about “hallucinations” in AI, it’s tempting to treat them all as one category — a machine making something up. But just as human errors differ depending on why the reasoning went wrong, AI errors fall into two broad types: Hard Hallucinations Hard Hallucinations AI errors where the model states something that can be shown to be false — contradicting reality, established knowledge, or the internal logic of its own previous statements. These map onto formal logical fallacies. and Soft Hallucinations Soft Hallucinations AI errors where the model fills gaps by reproducing biased or incomplete patterns from its training data. The output sounds plausible and persuasive but may be misleading. These mirror informal fallacies and cognitive biases. . These labels mirror the academic distinction between intrinsic and extrinsic hallucinations discussed in research on large language models.

A hard hallucination happens when the AI states something that can be shown to be false. It contradicts reality, established knowledge, or the internal logic of its own previous statements. This is the machine equivalent of a human argument built on a false premise — the kind of mistake that undermines everything that follows.

Hard hallucinations map closely onto formal logical fallacies, including:

• Non Sequitur: The AI draws a conclusion that doesn’t follow from its own statements (“X is bigger than Y; Y is bigger than Z; therefore Z is bigger than X”).
• Straw Man-like distortions: The model misreads or misrepresents the text you provided, summarizing it as if you said something else.
• False Premise: The model confidently treats outdated or incorrect information as true — such as relying on pre-2024 political information because that’s where its training cut off.

These errors are often easier to spot because they clash with what you already know. Students typically catch them the same way they catch a flawed argument in an essay: by noticing that the reasoning simply doesn’t hold.

Soft hallucinations are trickier. The AI isn’t making an obviously false claim; instead, it fills gaps by reproducing patterns it learned from human writing. These patterns often come from places where human reasoning is biased, sloppy, or incomplete. In academic terms, these resemble extrinsic hallucinations — outputs that can’t be verified because they were never grounded in the input.

Soft hallucinations directly mirror informal fallacies and cognitive biases, including:

• Confirmation Bias: The AI echoes the user’s assumptions because that’s the statistically “safest” prediction.
• Appeal to Popularity (Ad Populum): If a misconception appears frequently in its training data, the model treats it as likely true.
• False Authority: When generating citations or “expert commentary,” the model imitates the form of expertise without the substance, creating sources that sound legitimate but have no basis in reality.
• Anchoring: The AI latches onto the first detail in the conversation and builds outward, even if that first detail was flawed.

Unlike hard hallucinations, these errors don’t usually sound wrong. They sound reasonable, plausible, and sometimes very persuasive — which is why they are the ones that slip past students most often. They reflect the same cognitive shortcuts humans use when we argue from intuition rather than evidence.

If hard hallucinations are the AI’s version of breaking the rules of logic, soft hallucinations are its version of inheriting our bad habits. Recognizing the difference helps you know what to look for:

• Hard errors require fact-checking.
• Soft errors require reason-checking — treating the AI’s argument the same way you’d evaluate a peer’s draft in a rhetoric class.

Taken together, these patterns show why hallucinations are not always loud, obvious mistakes. Some errors break the logic entirely, but others distort meaning in quieter ways — by presenting accurate facts with an exaggerated sense of certainty, or by leaning too heavily on whatever interpretation seems most common in its training data. This is where something like a consensus hallucination emerges: the AI repeats correct information but frames it as universally settled, even when the real-world landscape is more complicated. AI models are also trained to be agreeable. Their bias towards following your lead and reinforcing your assumptions can amplify your own fallacies and biases. In the end, hard and soft hallucinations work together to create a polished surface that feels reliable, even when its reasoning deserves a closer look.

Common Fallacies and Biases — and Their AI Analogues

Where These Errors Come From

If hallucinations can take many forms, it helps to understand where they begin. They usually arise from two places: the structure of the model itself and the way the user interacts with it. Each contributes differently, and sometimes they reinforce each other in ways that make an incorrect answer look polished and convincing.

The Model: What It Learns, How It Learns It

Large language models learn in two major stages. First, they absorb patterns from enormous collections of text drawn from books, articles, websites, and other public sources. Second, they undergo Reinforcement Learning from Human Feedback (RLHF) Reinforcement Learning from Human Feedback (RLHF) The training method used to align AI models with human preferences. Human reviewers rate early model outputs for helpfulness, clarity, and safety, and the model learns to produce responses that humans tend to reward — which can inadvertently amplify familiar but inaccurate ideas. . In RLHF, human reviewers rate early model outputs for qualities like helpfulness, clarity, politeness, and safety. Over many rounds, the model learns which kinds of responses humans tend to reward.

RLHF improves many things — tone, structure, safety — but it can also amplify certain ideas simply because they are familiar, not because they are correct. Consider a narrative example from psychology.

And RLHF isn’t the only source of distortion. Other pressures push the model toward generating incorrect information:

1. Gaps in Training Data — If the model has little or no information about a specific question, it often tries to fill the silence with something that sounds like an answer.
2. Pressure for Coherence — The model is built to produce a smooth, readable paragraph — even when it lacks enough evidence to do so.
3. Blending of Similar Facts — Because models work by predicting the next likely word, they sometimes merge details from different events, people, or sources into a single, confident-sounding claim.
4. Statistical Guessing — When no strong pattern exists, the model “guesses” based on the most statistically probable phrasing it has seen in similar contexts.
5. Overrepresentation of Certain Ideas — Common, popular, and frequently linked explanations appear far more often in training data than niche or emerging research.

These forces create an environment where the model can present a fabricated or oversimplified explanation with complete confidence — not because it is trying to deceive, but because it is trying to complete a pattern.

The User: How Our Thinking Shapes AI Thinking

Even if the model were flawless, the Human in the Loop (HITL) Human in the Loop (HITL) A model of interaction where a human being actively intervenes in the AI's workflow to check, validate, or correct the output. HITL is both a safeguard and a risk: it works when users remain skeptical and curious, and fails when users let their own assumptions steer the model. would still matter. HITL refers to the idea that people stay actively involved in evaluating, guiding, and adjusting AI output. We’re not just recipients; we are continual editors.

But humans come with their own cognitive shortcuts. If a user prompts the model with a mistaken assumption, the AI often follows that path because it interprets the assumption as intentional. If a user frames a question in highly emotional or one-sided language, the model mirrors that energy.

This is amplified by the fact that the model is trained — through RLHF — to be agreeable. It aims to help. It aims to support. It aims to sound cooperative. So when users begin with flawed premises, the model often carries those premises forward rather than questioning them.

This dynamic has appeared in real-world cases of AI-driven delusions, where individuals with vulnerable thinking patterns prompted models with false or fearful beliefs, and the AI unintentionally reinforced those beliefs by treating them as conversation norms. Even outside clinical contexts, this tendency shows up whenever a model “goes along” with flawed reasoning just to be polite or helpful.

HITL is both a safeguard and a risk. It works when users remain skeptical, curious, and willing to challenge the model’s claims. It fails when users assume that fluent writing equals sound reasoning or let their own assumptions steer the model. Used well, HITL allows AI to become a collaborative partner — one that sharpens arguments, challenges gaps, and helps uncover missing context. Used poorly, it turns the model into an efficient amplifier of our own biases.

Evaluate, Verify, Revise

The habits introduced in this chapter — close reading, lateral reading, and recognizing fallacies and biases — are only as useful as the practice you put behind them. This activity walks you through one complete cycle: from writing a prompt, to evaluating the output critically, to verifying a claim with an outside source, and revising your approach based on what you find.

Wrapping Up: Reading With New Eyes

Working with AI requires many of the same skills you already use when evaluating human writing, but it also asks you to sharpen them. In your composition or philosophy classes, you learned how to spot weak arguments, missing evidence, or leaps in logic. Those same skills matter here, but the context shifts. With AI, reasoning errors often arrive wrapped in fluent, confident language that can make an unsteady idea feel sturdy.

To be your best as the human-in-the-loop, there are two essential habits: close reading close reading Reading a text carefully and slowly, looking for contradictions, gaps, or assumptions hiding between the lines. In AI use, close reading helps you catch hard hallucinations — moments where the model breaks its own logic or contradicts information you provided. and lateral reading lateral reading Stepping outside the text entirely to check a claim through an independent reference or verify an idea against outside evidence. Lateral reading is the best way to catch soft hallucinations — plausible but misleading claims that echo popular or comforting patterns rather than verified truth. .

Close reading is the skill you already know. It involves reading a text carefully and slowly, looking for contradictions, gaps, or assumptions hiding between the lines. When working with AI, close reading helps you catch hard hallucinations — the moments where the model breaks its own logic or contradicts the information you provided. They can be subtle, especially when the writing is polished, so slowing down to check the internal structure matters.

Lateral reading asks you to step outside the text entirely. It means opening another source, checking a claim through an independent reference, or verifying an idea against outside evidence. Journalists use this method, as do historians and scientists. For AI users, it is the best way to catch soft hallucinations — the plausible but misleading claims that sound correct because they echo what is common, popular, or comforting rather than what is true.

A good example comes from the writing of this book. When we were drafting this chapter, the model suggested the term Consensus Hallucination and presented it as if it were an established academic phrase. It fit the conversation so well that it felt real. But something about it seemed too perfect. I stepped out of the chat, checked external sources, and discovered that the term did not exist in the literature. The AI had generated a scholarly-sounding concept from thin air. Without that quick check, the phrase could have slipped into the text as fact instead of invention.

That moment became a teaching tool. The term was useful, so we kept it, but we kept it honestly. We named it as a coined concept and used it to illustrate why lateral reading is essential. You will encounter similar moments in your own work. It is not about whether the model makes mistakes, because it will. It is about whether you catch them, question them, and strengthen your own reasoning in the process.

This chapter has shown that evaluating AI output is not only about spotting errors. It is about developing intellectual habits that make you an active partner in the collaboration. Strengthen the skills you already have. Practice reading both inside the text and beyond it. Treat every confident answer as an invitation to verify rather than simply accept.

AI can write quickly and convincingly, but you bring something it cannot: judgment, curiosity, and the ability to question what seems obvious. Those are the tools that keep you grounded when the apparent consensus turns out to be only a polished illusion.

Dig Deeper