Skip to main content

“Three Ways AI Can Make Things Up. How True But Irrelevant Can Be Harder to Correct Than Pure Nonsense.”

· 5 min read
Chad Ratashak
Chad Ratashak
Owner, Midwest Frontier AI Consulting LLC

More Than One Type of Hallucination​

ChatGPT sometimes makes things up. For example, ChatGPT famously made up fictional court cases that were cited by attorneys for the plaintiff in Mata v. Avianca. But totally made up things should be easy to spot if you search for the sources. It’s when there’s a kernel of truth that large language model (LLM) hallucinations can waste the most time for lawyers and judges or small businesses and their customers.

  1. A “Pure Hallucination” is something made up completely with no basis in fact.
  2. A “Hallucinated Summary” has a footnote or other citation referencing a real source, but the LLM’s description of what that source says has little if anything to do with the source.
  3. An “Irrelevant Reference” is when an LLM cites a real sources and summarizes it fairly correctly, but the citation itself is not relevant to the purpose of the citation. This might be because the information is outdated, because the point only tangentially refers to the same topic, or for other reasons.
info

These examples were derived by actually reading the sources and were not written by LLMs. All of the written content on our website and social media is human-written, unless it is an example of AI-output that is clearly labelled.

danger

AI can help people summarize or rephrase content they know well. But Midwest Frontier AI Consulting strongly encourages AI users not to rely on AI-generated overviews of content they are not already familiar with precisely because of the subtler forms of AI hallucinations described below.

Scenario 1: You Got Your Chocolate In My Case Law​

  • Pure Hallucination: ** The LLM says: “Wonka v. Slugworth clearly states that chocolate recipes are not intellectual property.” ** In reality: No such case exists.

  • Hallucinated Summary: ** The LLM says: “NESTLE USA v. DOE clearly states that chocolate recipes are not intellectual property.” ** In reality: The case involves a chocolate company but is not about intellectual property rights.

  • Irrelevant Reference:

    • The LLM Says: ‘HERSHEY CREAMERY v. HERSHEY CHOCOLATE involved two parties that both owned trademarks to “HERSHEY’S” for ice cream and chocolate, respectively. This supports our assertion that chocolate recipes are not intellectual property.’
    • In reality: The facts of the case do not support the conclusion.

Scenario 2: Every Day is Pizza Day (Google AI Overview)​

“Three Ways Customers Learn About Your Business from Google AI (and what you can do about it)”

  • Pure Hallucinations: based on nothing.

    • The AI summary says: “The pizza place offers a large pizza for the price of a small pizza.”
    • In reality: No such deal ever existed.

  • Hallucinated Summary:

    • The AI summary says: “The pizza place offers a large pizza for the price of a small pizza.”
    • In reality: The pizza place offers a sale for a certain amount off that applies to both large and small pizza.

  • Irrelevant Reference:

    • The AI summary says: “The pizza place offers a large pizza for the price of a small pizza today.
    • In reality: The pizza place offers a large pizza for the price of a small pizza on Wednesdays only, and today is not Wednesday.

Scenario 3: EX PARTE LEE warning, but no show cause order​

A Texas state court case from 2023 noted that case citations were fake or irrelevant, and an extensive footnote indicated that this may have been due to AI misuse, referencing Mata v. Avianca.:

However, in [LEE’s] “Argument” section, where appropriate citations must be included, Lee cites to five cases to support the two sub-arguments to his issue. Only three of those five cases are published. None of the three published cases cited actually exist in the Southwest Reporter.1 Each citation provides the reader a jump-cite into the body of a different case that has nothing to do with the propositions cited by Lee. Two of the citations take the reader to cases from Missouri. As the State points out, even Texas cases with the same names as those cited by Lee do not correspond with the propositions relied upon. EX PARTE LEE, Chief Justice Tom Gray, Court of Appeals of Texas, Waco, No. 10-22-00281-CR, Decided: July 19, 2023.

APPELLANT’S BRIEF

  • Pure Hallucination: Appellant’s brief appears to have cited two cases out of five that did not exist, which if prepared with AI as the court noted was possible would have been pure hallucinations.
  • Hallucinated Summary: There were Texas cases with the same names as those cited by appellant’s brief but those cases did not correspond to the propositions relied upon. If prepared with AI, at least the Texas cases (and possibly also the Missouri cases) would be hallucinated summaries.
  • Irrelevant Reference: Appellant’s brief cited two Missouri cases. Even if the citations had been summarized correctly, they would not have been from Texas state court.
tip

It is not enough to check that a cited case exists. It is not even enough to skim the case to make sure it seems like it is addressing the right topic. If you are using LLMs for research, you must still actually read the sources you are citing. Because ultimately, you are the one citing the sources, not the AI.

A modified version of this post focusing on the legal cases only was cross-posted from Substack as “2. Ex Parte Lee (Texas) Not My First Rodeo”.

Footnotes​

  1. Arkansas, Kentucky, Missouri, Tennessee, and Texas. ↩