14 posts tagged with "Law"

Mata v. Avianca: The First ChatGPT Misuse Case​

The case Mata v. Avianca was a personal injury lawsuit against an airline in the U.S. District Court for the Southern District of New York (SDNY). However, the reason it became a landmark legal case was not the lawsuit itself, but the sanctions issued against the plaintiff’s lawyers for citing fake legal cases made up by ChatGPT. At least that was the popular version of the story emphasized by some reports. The reality, according to the judge’s opinion related to the sanctions, is that the penalty was about the attorneys doubling down on their misuse of AI in an attempt to conceal it. They had several opportunities to admit their fault and come clean (page 2, Mata v. Avianca, Inc., No. 1:2022cv01461 - Document 54 (S.D.N.Y. 2023)).

Take this New York Times headline “A Man Sued Avianca Airline. His Lawyer Used ChatGPT,” May 27, 2023. This article, written before the sanctions hearing in June 2023, focused on the ChatGPT-gone-wrong angle. By contrast, Sarah Isgur of the Advisory Opinions podcast had a very good breakdown noting the attorney’s responsibility and the back-and-forth that preceded the sanctions (episode “Excessive Fines and Strange Bedfellows,” May 31, 2023). However, in that podcast episode the hosts questioned the utility of ChatGPT for legal research and said “that is what Lexis and Westlaw are for” but as of 2025 both tools have added AI features including use of OpenAI’s GPT large language models (LLMs).[^1]

Hallucinating cases about airlines.

Why Care? Our Firm Doesn’t Use AI​

Before I get into the details of the case, I want to point out that only one attorney directly used AI. It was his first time using ChatGPT. But another attorney and the law firm also got in trouble. It only takes one person using AI without proper training and without an AI policy to harm the firm. It seems that one of the drivers for AI use was access to other federal research tools was too expensive or unavailable, a problem that may be more common for solo firms and smaller firms.

Partner of Levidow, Levidow & Oberman: “We regret what's occurred. We practice primarily in state court, and Fast Case has been enough. There was a billing error and we did not have Federal access.” Matthew Russell Lee’s Newsletter Substack

You might say, “Fine! We just won’t use AI then.” Do you have a written policy stating that? Do you really not use AI? I have two simple questions:

1. Do you have Microsoft Office? (then you probably have Office 365 Copilot)
2. Do you search for things on Google? (then you probably see the AI Overview) If the answer to either is yes (extremely likely), are you taking measures to avoid using these AI features? If not, how can you say you don’t use AI? Simply put, avoiding AI is not the default option. It requires conscious effort to avoid the features being added to existing software, from word processors to specialty legal research tools.

Overview of Fake Citations​

The lawyers submitted hallucinated cases including the court and judges who supposedly issued them, hallucinated docket numbers and made up dates.

Lack of Guessing Penalties: The Source and Solution to Hallucination?​

Language models like GPT-5 “are optimized to be good test-takers, and guessing when uncertain improves test performance” Why Language Models Hallucinate This is the key to AI hallucinations, according to a new research paper from OpenAI, the maker of ChatGPT, published on September 4, 2025. I think this explanation has merit, although it doesn't seem to explain when large language models (LLMs) have access to sources with the correct answers and incorrectly summarize them.

The most interesting point to me in the paper is their call for changing how AI benchmarks score different AI models to penalize wrong guesses. This reminded of how for most multiple-choice tests in school, you should choose any random answer rather than leave the answer blank. If the answers are ABCD, you have a 25% chance of getting the answer right and you always have a positive expected value, because you either get one point or zero. Zero for a wrong answer is the same as zero for no answer. However, Advanced Placement (AP) tests used to give negative points for wrong answers. When I went to find a source for my recollection about AP test scoring, I learned that this policy had changed shortly after I graduated high school. (“AP creates penalties for not guessing,” July 2010). So it appears that penalizing guessing is just as unpopular with human benchmarks as AI benchmarks. I, for one, am in favor of wrong-guess penalties for both.

False Cognates​

In foreign languages, there are cognates, words that are the same or similar and mean the same thing. Think "house" in English and "Haus" in German. Then there are false cognates that seem similar but mean very different things. For example, "Gift" in German means “poison.”

In generative artificial intelligence (GenAI), certain popular terms overlap with terminology in other fields. Fish don’t know they’re swimming in water. Likewise, GenAI specialists often interact with people in other fields without realizing their use of terms familiar to themselves are causing confusion because of different meanings in another field.

False Cognates in Generative AI​

Some common terms that might cause confusion include:

• In general: “local” meaning from a nearby area v. “local” meaning an AI model can run on your own computer.
• Chemistry, Economics, Acting, Publishing, Real Estate: AI agents clashes with several fields’ terms, including:
  • “chemical agents.”
  • an economic “agent” as in “principle-agent problem.”
  • an “agent” representing actors or writers.
  • a Realtor or similar agent.
• Law:
  • Master of Laws (LLM) degree clashes with large language model (LLM).
  • “inference" of fact v. the process of running the AI model.
• Finance:
  • anti-money laundering (AML) is similar, especially verbally, to artificial intelligence/machine learning (AI/ML).
  • “model” (in the context of model risk management) v. “model” (like “GPT-5” or “Gemini Flash 2.5”).
  • “token” as in cryptocurrency v. the unit of meaning in an LLM

Professor’s Lament​

I’m writing this to explain prompt engineering, but that’s too vague. What I’m specifically responding to is a former college professor after he wrote earlier this month:

Wait, so 'learning to write sophisticated prompts' is now a class, and the title of the course >is 'Prompt Engineering'? Is it too late to stop this?

So Prof. X (you know who you are) I’m going to try to convince you—and any other skeptics reading—that prompt engineering is a real skill with meaningful implications for AI. There are three things I want to address:

1. I get why you’d roll your eyes at it.
2. There may be things you like about prompt engineering.
3. Failure to understand prompt engineering and prompt injection risks creates real-world security risks.

The Reaction Against Slop​

There is already too much AI slop. Facebook is particularly full of slop images that get thousands or millions of likes from people who seemingly don’t realize they are interacting with AI-generated content. But the problem is in every corner of the internet. You can even find examples out in the real world if you look careful, especially in ads and posters. So when you hear “prompt engineering” but mentally translate it to “slopmonger,” I get why you have such a strong negative reaction.

I’m against slop. I hate slop. I do not want my kids to grow up in a word overrun by slop. You can look up John Oliver’s recent rant against slop, but I personally prefer Simon Willison’s 2024 statement here:

I’m a big proponent of LLMs as tools for personal productivity, and as software platforms for building interesting applications that can interact with human language.

But I’m increasingly of the opinion that sharing unreviewed content that has been artificially generated with other people is rude.

Slop is the ideal name for this anti-pattern. […] One of the things I love about this is that it’s helpful for defining my own position on AI ethics. I’m happy to use LLMs for all sorts of purposes, but I’m not going to use them to produce slop. I attach my name and stake my credibility on the things that I publish.

Hacking with Poetry and Foreign Prose​

Back in 2023, a Swiss AI security firm called Lakera released a game called Gandalf AI involved seven levels of increasing difficulty trying to get a large language model (LLM) chatbot “Gandalf” to tell you a secret password. As the levels got more difficult, prompts required more ingenuity. Successful strategies included convincing the LLM that it was telling a fictional story or saying that the password was needed for some emergency.

For the hardest levels, the most successful prompts asked the LLM to write poetry or translations into a foreign language. In doing so, the LLM leaked information about the password that evaded scrutiny. Surely a champion of the humanities like yourself can appreciate the irony that poetry and foreign language education can now be considered essential ingredients in a computer-related industry.