2 posts tagged with "Research"

TL;DR​

• N.D./S.D. can mean “Northern District” and “Southern District” for federal cases, overlapping with “North Dakota” and “South Dakota” postal abbreviations; other states have possible issues here like MD and OK
  • “CA” (Court of Appeals? California?). I addressed this in my “Writing Help” game.
  • “SC” is a triple threat: South Carolina, Supreme Court, or Superior Court?
• I looked at the data. Yes, there were some human errors, but it is clear that Charlotin has put in the work to create and maintain this valuable resource. That’s why I think people find it worthwhile to send him cases. He’s provided something of value and put in the time. Anyone who looks closely at the data can see that. People who churn out slop reports, on the other hand, are wasting your time. If it wasn’t worth their time to write, why would it be worth your time to read?
• Once you account for relative population, the map is much more of a dog bites man story. To me, the more interesting part is the specifics of each case and the citation graph of cases that are frequently cited by later cases in these disciplinary decisions.
  • For example, I thought this may have been a mistakenly coded jurisdiction but it was not: Lowery Wilkinson Lowery, LLC, et al. v. State of Illinois, et al. (E.D. Oklahoma 2025). Instead, it’s an interesting attempt by the plaintiff to apply the “Indian Country” McGirt ruling and the plaintiff also misattributed that U.S. Supreme Court ruling to the Eastern District of Oklahoma Judge.
  • “There is no one righteous, not even one.” After data cleanup, New Hampshire alone seemingly had no hallucination cases, but I found both that New Hampshire had had its own hallucination case around July 2025 per reporting last fall, and that a New Hampshire attorney had been responsible for hallucinations (actually from the client) in a case in the Vermont Supreme Court.

“N.D.” and “S.D.”: Did Not (Usually) Mean the Dakotas​

Damien Charlotin maintains a well-known public database of AI hallucination cases around the world. It has even been cited in a legal case here in Iowa. I’ve written about his work before, including his point about low false-positive rates potentially creating automation bias, lulling users of legal AI tools into a false sense of security. Recently I saw a post on LinkedIn, in which Charlotin, reacting to a report by Straight Arrow News on a study by “Laine AI,” criticized the claim that AI-driven legal hallucinations are especially concentrated in North Dakota and South Dakota. Charlotin pointed out that this was most likely due to a misinterpretation of “N.D.” and “S.D.”—common abbreviations in federal cases indicating “Northern District” and “Southern District.”

The June 19, 2026 article noted how surprising the result was, with the headline: “Where does the most AI legal slop come from? Not from the states with the most lawyers.” Laine AI, according to the article, stated that North Dakota courts had 109 cases with AI errors, South Dakota had 82, and California had 59. Supposedly, this was from Charlotin’s website, but as I often advise clients, this is not a good use of LLMs: pivot tables already exist!

After I manually reviewed and coded the states for 1,145 USA cases and cleaned up data from Charlotin’s website downloaded on June 21, 2026, I found that the actual count was: California 121, North Dakota 6, and South Dakota 1. California had some “E.D. Cal” and similar, as well as two Ninth Circuit cases that were appeals from California cases, a case that was “CA SC” (Court of Appeals, South Carolina? California Superior or Supreme Court?) which was actually California; and one “CA 8th Circuit (Bankruptcy)” a federal Eighth Circuit case that was an appeal from District of North Dakota Bankruptcy.

None of these minor ambiguities would explain the large discrepancy in counts. My review of the numbers suggests that Charlotin was correct about the likely cause of the hallucinations.

You can read Charlotin’s own account of the mix-up on LinkedIn.

That being said, 6 cases in North Dakota relative to its population size and number of attorneys does seem to be somewhat disproportionate. There, I’d argue that almost every state is probably undercounting/not catching AI hallucinations, or the hallucinations have been caught, but not reported and added to the database. In fact, this exercise led me to find some information about New Hampshire as the only (apparent) state without hallucination cases.

So, Midwest’s honor defended. Let’s move on to some other lessons.

It’s not just “garbage in, garbage out”​

Sometimes you’ll hear about “garbage in, garbage out” (GIGO). As it is applied to practical deployment of generative AI in businesses and other organizations, there’s a common refrain that “the AI is only as good as the data you give it.”

I’d somewhat reframe that to say “the data quality limits how well your AI can perform.” Even with good data—company policies, solid accounting spreadsheets, detailed written reports—the LLMs can still hallucinate. LLMs can make ungrounded statements even when they have access to the correct answers, they can misunderstand details, and they can improperly combine text, among other failure modes.

Charlotin’s dataset is carefully maintained as far as I have seen, and I’ve looked through a lot of it. His database appeared to be the Laine study’s sole source of data. The Laine summary stats quoted in the aforementioned report, on the other hand, suggest AI hallucinations. These uses of AI: “AI can write this research paper,” “AI can write that quarterly report for the board,” “AI dashboards give the CEO unmediated access to perfect visibility into the organization” are actually organizationally destructive uses of LLMs. They are truthiness machines, but once you look under the hood at the data, you can tell that the summaries bear no relationship to the underlying data. Since there is no relationship, correcting the data is pointless. That’s the difference between a human-curated database—like Charlotin’s—and a slopped together dashboard or report or slide deck. Of course humans make mistakes too, and human-curated databases will have rough edges or errors, but it is worthwhile to look at them closely and help fix them because somebody who cares about the truth is on the other end.

Abbreviations Are Not Always OK​

North Dakota and South Dakota are not the only examples. There is a whole class of collisions between legal shorthand and two-letter state postal codes. Here are the ones I’m aware of:

Literal String	What it really means	Collides	Impact
‘N.D.’	Northern District	North Dakota	Low population, many federal northern districts, dramatic false spike
‘S.D.’	Southern District	South Dakota	Low population, many federal southern districts, dramatic false spike
‘M.D.’	Middle District (e.g., M.D. Fla.)	Maryland	More populous, fewer middle districts, mixed effect
‘Ct.’	Court (Sup. Ct., Ct. App., Dist. Ct.)	Connecticut	possible, but less frequently seen “in the wild”
‘CA’	Court of Appeals	California	Large population, hides in the noise
‘S.C.’	Supreme Court or Superior Court	South Carolina	Mid-spectrum

Supreme Court or Supremest Court?​

‘S.C.’ for Supreme Court raises a deeper problem. You still don’t know which court is highest in the given state without domain familiarity, because state court naming is not uniform across jurisdictions and “Supreme Court” is not the highest court everywhere:

• In New York, the “Supreme Court” is the trial court and Court of Appeals is the highest.
• Texas and Oklahoma each have two highest courts: Supreme Court (civil) and Court of Criminal Appeals (criminal).
• Maryland changed the naming to be more familiar in December 2022.

If there are other examples, please let me know. But my main point is that you can’t just assume based on the name.

Divided Islands​

When parsing strings for the name of a state, “Hawaii” or “Hawai’i” can be written with or without the apostrophe, which can also be written as ʻokina (Unicode U+02BB), a real letter in the Hawaiian alphabet. This is actually an example of an area where appropriate use of LLMs can help, because it can be exasperating for regular expressions, especially with an unpaired apostrophe.

Look at the Data and Learn About the Domain​

Anyone who has spent time with both LLMs and legal citations should recognize that “N.D.” could have multiple, conflicting meanings. Also, you should assume every map is secretly a map of population until proven otherwise. So when “North Dakota” has far more AI misuse than California, look at the rows in your spreadsheet first.

My own background is in financial-crimes investigation, OSINT, and GIS. I have hit similar failures with AI, law, and geography before, as I wrote about in December 2025 (NOTE: correction, I did this in December, but I wrote about it in January). I rebuilt my hallucination-case map with Claude Code, and at first it produced a patchwork quilt because it matched county names across states without using the state as part of the key, and there are a lot of Polk Counties. The output was obviously wrong to me because I looked at the output, and I also knew why it failed, because I’m familiar with the domain. Claude Code also tried to fabricate an Eastern and Western District of South Carolina, which is a more specific hallucination that would require domain knowledge (which I had developed), to recognize that the output was incorrect.

Nice Try, New Hampshire​

After manually reviewing and cleaning/hand-coding the state data, the only state that (apparently) had no attorneys with hallucinated cases was New Hampshire.

However, I found that a New Hampshire attorney included a fake quotation from a real case in a Vermont Supreme Court divorce case provided by his client. This touches on multiple topics I want to write about, including: a) that LLMs can hallucinate quotations, so checking that a case merely exists is not sufficient; b) that client-provided AI-generated fake citations are a problem for attorneys who “don’t use AI,” so they need a reality check about Shadow AI use; and c) family law is coming up frequently for AI hallucinations (e.g., early Iowa misuse and a recent Nebraska Supreme Court case).

It wasn’t adding up. A New Hampshire attorney representing a divorce client filed a legal brief that quoted from a previous court case. But when Vermont Supreme Court justices went to the case he referenced, the quote was nowhere to be found. In a November hearing, they asked the attorney where the quote came from. “Your honor, my client used an AI, um, helper” said the attorney. Justice William Cohen, now retired, responded: “The secondary source was AI? And you didn’t identify it?” “I’m not familiar with what’s involved with it and so forth,” the attorney said. He claimed that his client offered to write the brief using artificial intelligence. The quote came from “AI GPT or something like that,” he said later on. “I didn’t use it exactly but it’s a common one, I believe.” After the hearing, justices on the state’s highest court chastised the attorney for his mistakes in a court filing, requiring him to file a copy of the write-up in all of his pending cases in Vermont Superior Court. Valley News

Later, I found a more direct example of a New Hampshire case. The “Windham case” with Judge Lisa English, reported in October 2025 involving a NH/MA attorney. I unfortunately could not track down the name of the case, despite having the attorney’s name and the judge.

The Laine report produced unsupported conclusions about supposedly excessive AI misuses in North Dakota and South Dakota. However, my analysis of the data allowed me to find the one missing gap and show that there is no state without any AI hallucination cases.

Charlotin’s Map on July 2, 2026​

As of today, there is no state with 0 hallucination cases on the map of USA cases on Charlotin’s website.

Indian Country Jurisdiction: Plaintiff also claims that Judge White wrote the McGirt opinion (he did not)​

Because I was manually reviewing the cases for state coding, I thought that a case with “State of Illinois” as Defendant, but in the Eastern District of Oklahoma, may have been mistakenly coded. It was not. Instead, it was interesting case and a great example of why you should look at the data and not just let AI summarize everything in the aggregate for you.

Lowery Wilkinson Lowery, LLC, et al. v. State of Illinois, et al. (E.D. Oklahoma 2025).

Multiple times, the plaintiffs attempted to raise several jurisdictional arguments under McGirt that are so inadequate and clearly unresearched that any reasonable attorney would know they lack merit. Plaintiff repeatedly asserts that if this court fails to direct bar disciplinary proceedings in Illinois, that it is effectively “overturning” McGirt v. Oklahoma, 140 S.Ct. 2452 (2020), a case where the United States Supreme Court held that the entirety of the Eastern District of Oklahoma is “Indian Country” for the purposes of the Major Crimes Act. Dkt. No. 74 at ¶44. Plaintiff also claims that Judge White wrote the McGirt opinion (he did not), indicating that counsel has not made any attempt to research the case. Id. In addition, plaintiffs further allege that “the Defendants accused Plaintiff Lowery of a violation of the Major Crimes Act while on an Indian reservation land under McGirt … Therefore, if the Court wants to exercise jurisdiction under McGirt it would be Mandatory jurisdiction.” Lowery III, Dkt. No. 22 at p. 5. OMNIBUS ORDER

Takeaways​

• Even if an LLM feels like it gives you an overview of the data, look at the data. There’s fun stuff in there and you’ll catch more hallucinations.
• But you should also ask if you should even be using LLMs for a given data analysis task. Pivot tables and SUM functions still exist.
• Your starting assumption should be that every map is a population map.
• Damien’s database is legit.
• Come at the Midwest, better not miss.

The main concern of the paper (and what I gather to be a focus area for years to come) is the cognitive impact of using generative AI for legal tasks. This requires both short-term and long-term studies; the authors are careful to note that negative effects may appear in longer term use of AI, but that in this particular study, the group that used AI throughout outperformed the group that only had access at the end.

This particular study is a short-term, randomized controlled trial (RCT), such as you might see more frequently in medicine (indeed, one author speculated that the use of that term may have caused public access issues for the article initially on X/Twitter).

I did not personally know Daniel Schwarcz, one of the University of Minnesota Law Professors, prior to reaching out to provide feedback on this study. But I knew of him and have a high opinion of his earlier work on cybersecurity and law. For example, Schwarcz had co-authored an excellent paper, “How Privilege Undermines Cybersecurity,” published in the Harvard Journal of Law & Technology in Spring 2023, which my group in the Dept. of Homeland Security’s Public-Private Analytic Exchange Program (AEP) read and cited in our work on Ransomware Attacks on Critical Infrastructure.

Study Design​

University of Minnesota Law School professors Nick Bednar, David Cleveland, Allan Erbsen, and Daniel Schwarcz ran a randomized controlled trial involving approximately 100 2L and 3L students. The study was published April 5, 2026 on SSRN: Artificial Intelligence and Human Legal Reasoning. Strictly speaking, the comparison was not an “AI v. Not AI” group as much as an “AI from the outset” group v. an “AI as a final editor only” group.

This experiment used Google’s Gemini 2.5 Pro; I will discuss this specifically in a later section; this experiment is relevant, even for attorneys not using Gemini directly, because Gemini is the LLM behind Google AI Overviews and it is a popular API model in legal AI tools (e.g., Westlaw’s CoCounsel based on references to “Thomson Reuters AI third-party partners, such as OpenAI and Google…”).

Participants completed four sequential tasks:

1. Synthesis Task (AI for AI group):

The synthesis task was designed to test whether using AI can help lawyers synthesize legal sources addressing unfamiliar subjects. We cast each participant in the role of a law firm associate who received an email from a partner asking them to summarize a legal rule based exclusively on five supplied sources. The partner explained that: “Your memo should outline the elements of the rule and any exceptions, providing a framework for how a court would approach the legal question. In other words, don’t merely summarize the sources; synthesize them into a summary of the rule that indicates how the elements fit together.”60 Participants had up to 75 minutes to read the packet and complete the memo.61 The control group was instructed not to use AI for this task, while the AI-exposed group was instructed to use Gemini 2.5 Pro62 “to assist you in writing the assignment.”

2. Comprehension Task (closed book, no AI for either group): six moderately difficult multiple choice questions in ten minutes without access to either the source packet or AI.
3. Application Task (access to their prior synthesis memo, no AI for either group):

That task presented participants with a follow-up email from the partner who had assigned them the synthesis task instructing them to write a memo applying their knowledge from the synthesis task to a new set of facts……“identify strengths and weaknesses in the client’s position, recommend arguments that the client should make, and rebut counterarguments.” Participants had up to 60 minutes to complete the memo.

4. Revision memo (access to prior application memo, AI for both groups): revise the second memo with Gemini in 20 minutes.

Specifically, “[e]ach task related to a problem involving servitudes that burden personal property.”

What They Found​

AI helped on the synthesis task: as expected, students who used AI produced better memos and finished faster on the initial task when they had access to AI.

Early AI use did not diminish comprehension

contrary to our preregistered hypothesis, AI exposure at this initial stage did not diminish downstream comprehension of the underlying legal principles. To the contrary, participants who used AI on the synthesis task outperformed the control group on the later application task even when neither group had access to AI.

The full AI group outperformed the control (AI at the end) group. But the authors note that with long-term use, skills may atrophy. They warn that everyone (especially new lawyers) may lose or fail to develop skills if they don't learn

"to sit with a hard question, to trace an argument through its premises, to recognize when doctrine is uncertain and when it is settled" and learn to explain that reasoning to clients and to judges and lawyers, rather than delegating to AI.

"Leveling effect": AI use by high-performing individuals may degrade work product, while improving the work of the lowest performer. It didn't change the overall ranking, but made people more average. The experiment involved new information, not areas of expertise; however, the authors’ advice was to typically use AI where you are able to check the work because of your expertise.

What This Means and Connection to My CLE Advice​

Like the authors, I was surprised by the results. However, the way the AI was used generally fits with the advice I give in my CLE on how to use AI responsibly.

Summarization as Triage, Not Replacement for Reading​

In my CLE “Generative Artificial Intelligence Risks and Uses for Law Firms,” I note that you can use AI to summarize documents for triage when you have limited time. The students in the synthesis scenario only had 75 minutes to read and synthesize 5 documents.

However, I also warn that reading an AI’s summary of a document is not the same as having read the document, especially due to hallucinations. Hallucinations are not limited to making up fake case citations; they may also include fake quotations and improper summarization of the holdings of cases, for example.

Experts Should Check AI Output​

I note throughout my CLE and the public talks that I give, such as my recent talk with ACAMS DC—you can watch a recording here, that the person most familiar with material should check the AI output.

For reports, this means if AI is used to write an executive summary, it should be a draft executive summary that is then reviewed and edited by the original author of the longer piece. It should not be a summary generated by a lazy reader who can’t be bothered to read the full document. The former can root out hallucinations and distortions of the writer’s intent. However, I would warn that even this use could shape the writer’s focus by making them highlight parts of the paper other than what they would have chosen had they written the executive summary from scratch.

Experts should check the output. AI can be very wordy and very persuasive. It can write things that appear to be correct. The user should not be learning something new when they are reviewing AI output, because the AI may persuade them. I warn in my CLE that sometimes AI is so persuasive, its confident hallucinations may even cause you to second-guess yourself about something you know well.

Warning About “Writing Help” at the End and Don’t Grab AI as a Last Resort​

The authors recommend:

“A final principle suggested by our experimental results is that lawyers should avoid using AI to complete legal tasks under artificially tight time constraints or when cognitively fatigued.”

I frequently note that attorneys should learn about generative AI risks and responsible use now, even if they do not currently use it. Don’t wait until you feel like time pressure is forcing you to bail yourself out with AI:

• Did not go well for an attorney in Colorado who used AI and blamed it on an intern in 2023; result: temporary suspension.
• Did not go well for attorneys in New York in 2023 or an attorney in Iowa in 2025 who used AI in lieu of access to paid legal databases; result: monetary sanctions.
• Did not go well for an AUSA with a 30-year career in North Carolina who reportedly used AI to “catch up” on a filing in 2026; result: resigned from office.

Empirically, the study found that using AI at the end under tight time pressure led “in many cases [to] a modest deterioration.” I warn that even using AI just for “writing help” at the end of a task can be risky in my CLE courses, especially in “AI Gone Wrong in the Midwest.” This has caused problems for attorneys and experts witnesses. You can play an interactive demonstration of that risk and read more about it in The AI "Writing Help" Trap.

Gemini and Outdated Models in Academic Studies​

This study used Gemini 2.5 Pro, but a more current Gemini model since late Fall 2025 was Gemini 3 Pro, and now there is Gemini 3.1 Pro (preview). All formal academic research seems to suffer from the problem of referencing outdated GenAI models by the time of publication.

This is not the fault of the researchers. The pace of academic publishing is simply too slow for the pace of release of major model updates. I am not the first to point this problem out, and others, e.g., Ethan Mollick, have commented on it frequently.

The most important reason I mention this is that people will cite a paper claiming “AI fails at X task” and cite a recently published academic paper, but the paper itself almost certainly doesn’t have recent models. If you look closely, it might mention “o3” or “Claude Sonnet 4” or “Gemini 2.5,” which does not invalidate the study. However, it may be demonstrably false that “AI can’t do X” today with a frontier model, if you simple logged in and tried it for yourself. Lesson: do not rely on academic studies alone for claims about the limits of what AI can do.

For this reason, I think there is a lot more value in studies like the UMN study looking at AI-human interaction. They teach us how AI works in practice and how human users respond to AI use.

Context Window​

Gemini was the first major LLM to have an extremely long context window: one million tokens.¹ Gemini’s long context window has made it popular for processing large numbers of documents, which might make it seem well-suited for legal purposes.

Where Gemini Has Performed Well​

In addition, Gemini is a very capable model in some tests, although it fails on others. For example, Gemini has been the model most capable of identifying coded allusions to sovereign citizen legal ideology among the models I've tested.² It also currently has the best image editing model in my opinion, and Google has SynthID for provenance verification.³

Gemini 3 was also the first LLM to pass one of my personal benchmarks.⁴

Where Gemini Has Performed Poorly​

However, Gemini also powers the Google AI Overview and Google AI Mode, which have access to Google Search results. Despite access to the best search engine in the world, the LLM-generated summaries can still hallucinate false descriptions of nonexistent cases based on nothing but the bare case citation. Even ungrounded false information that contradicts the Google Search results with the correct answer, e.g., an OSC chastising an attorney for citing the nonexistent case. This Doppelgänger Hallucination problem has not gone away with Gemini 3.

On an AI research task in the Gemini app, Gemini 2.5 performed worse than ChatGPT. When I ran the test again, Gemini 3 still failed the test, and tried to persuade me that it had provided a comprehensive answer. I have been warning consistently that AI models are not just “getting better,” but rather they are getting “more capable.” One way this manifests itself is in LLMs becoming more persuasive about covering for their own faults.

Even when Gemini performs well, its performance can be “jagged”: impressive on one detail, hallucinating on the next in the same task.⁵

Bad Data Training Policy​

However, I would warn attorneys to carefully consider what they put into the consumer version of Gemini if using it for work purposes, such as the non-public discussion of client position and possible strategy described in the Application Task in this experiment. Google’s Gemini has one of the worst data training opt-outs for major U.S. labs if you are using the consumer version.

Conclusion​

I always try to keep up with the latest LLM research, so that I am offering fresh and accurate advice to my clients. That being said, I am also not chasing the latest fad and trend. I focus on core principles around accuracy, end-to-end processes, and realistic understanding of how human users actually relate to their AI tools. That’s why I really like this study design and encourage the authors to continue in projects like this. It is also why I stand by the advice I provided in my CLE, despite this field of GenAI changing so dramatically every couple months.

How Midwest Frontier AI Consulting Can Help​

That reading informs the governance advice I give to law firms: if you'd like to put it to work for yours, schedule an introductory call.

You can take my CLE on demand on CLE Hero. Find out more details here.

Footnotes​

1. Tokens are words, parts of words, or even single characters: the unit LLMs use when processing text. One million tokens in English is roughly 750,000 words, longer than War and Peace. ↩

2. I first encountered this issue when writing about the case Thomas v. Pangburn (S.D. Georgia) but have found additional examples for subsequent tests, which I have not written about publicly. ↩

3. I will be writing about SynthID more in a forthcoming series. ↩

4. Specifically, a test involving an obscure Moroccan Arabic word. I discuss the utility of personal, non-public benchmarks in my CLE. ↩

5. When fact-checking a U.S. federal court district map, Gemini correctly flagged the oft-overlooked inclusion of Yellowstone National Park in the District of Wyoming and Tenth Circuit (rather than the adjacent Montana/Idaho districts in the Ninth). That subtle catch was impressive, but in the same response it hallucinated a second "error" that was clearly not an error. ↩