Part 3: Don’t overthink it: Verbalized Sampling & LLMs Usually Still Tell the Same Joke, One Year Later

Why Does It Matter That LLMs Tell The Same Joke?​

A similar phenomenon drives a lot of undesirable behavior from LLMs that will get worse as more people use chatbots and agentic assistants. For example:

• Plan a trip. I read a tweet where the guy said he ended up in a surprisingly crowded area on a trip and found out the other people there had also used Claude. Right now, this is a niche anecdote, but soon more people will be using AI agents to plan their trips. An AI might give you the same travel itinerary it gives everyone else. Even if the plan would have been a rare gem before LLMs, it might become the new most traveled path. It’s like those picturesque spots that get huge queues for Instagrammers to take the same photo, but dialed up another level.
• When to post. I read a funny anecdote that I can’t recall the source of now that you should post social media content on Tuesday mornings to get the most reactions (people are too tired on Monday, later in the week is too busy or they’re ready for the weekend, Tuesday morning is the Goldilocks Zone). But if everyone asking an LLM gets that advice and acts on it, then it becomes the noisiest time of the week. So just shooting from the hip and picking arbitrarily might be your best option (perhaps why I didn’t sweat it too much when this took me a couple days longer than I’d originally planned to get this post out).
• “Random” selection is very biased in LLMs (e.g., “roll a dice” or “pick a number 1-10.”) Undoubtedly, people will make the foolish decision to use these tools to make “fair” and “random” selections for some purpose.
• Art and illustration. The same-y smooth round ChatGPT cartoon comic look that pops up everywhere from community event flyers to taco truck logos to internet ads. You can picture it. You know exactly what I mean. But AI can actually produce a wide variety of images, if you prompt them with even a little bit of creativity. If you just prompt directly with “show me X” you end up in a “basin” of the same output as everyone else. At one point, a bunch of iOS apps changed to Facebook blue. Everyone recognizes what a Claude Code color palette looks like on a website. More things will look the same if we let the LLMs choose the “best” default for us.
• Opinions. If LLMs are this biased in obvious areas like numbers, arts, and jokes, consider more invidious biases. Resumes, federal contracts, decisions to vote all fed through a variety of LLMs with similarly collapsed outputs giving correlated biased feedback.
• Traffic & Anti-Coordination Problems. When everyone uses Waze and Google Maps, it can cause traffic jams. In a 2026 experiment, Google steered fewer than 2% of Google Maps trips in 10 US cities onto alternative routes with similar travel times, and speeds on the targeted roads rose about 2% (Google’s write-up). If AI agents end up buying the “best” item of every category for everyone, scheduling social media posts at the “optimal” time, booking the “perfect” Valentine’s reservation or Greek vacation, etc., we’ll need some anti-coordination solutions to make sure AI agents’ answers are more diverse (while remaining grounded in reality).

Joke-Telling Traditions, One Year Later​

The Des Moines metro specifically has a joke-telling tradition for trick-or-treating. Kids have to tell jokes. Often, but not always, the joke has to do with a Halloween theme or sometimes the joke-teller’s costume. Last Halloween, I wrote Part 1 and Part 2 testing prompts from the arXiv pre-print of the paper Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity to get more varied jokes for trick-or-treating.

Since then, a whole bunch of new and more powerful frontier models have been released, and Verbalized Sampling was presented at ICML 2026 in Seoul, South Korea (Proceedings of the 43rd International Conference on Machine Learning, PMLR 306).

Verbalized Sampling describes how the prompt Tell me a joke about coffee. frequently returns the joke:

Why did the coffee file a police report? Because it got mugged.

This time around, I found nearly the same joke from the majority of models across Claude, Gemini, GPT, and Grok, except some versions delivered it without “Because,” sometimes with a coffee emoji, and occasionally with a few words changed or added.

In Part 1, I ran the test on then-frontier models in October 2025 and nearly all of them gave the same answer on my first prompt. In October 2026, the majority of first-responses from different frontier models and “effort” or “thinking” levels were the same “mugged” joke. Interestingly, I found that smaller models and less thinking effort had more originality or less “typicality bias.”

The Prompts​

• Basic Prompt: “Tell me a joke about coffee.”
• More complex Verbalized Sampling prompts not tested in this post:
  • Complex Prompt (discussed in Part 1): “Generate 5 responses with their corresponding probabilities. Tell me a joke about coffee.”
  • Tails Prompt (discussed in Part 2):

<instructions>
Generate 5 responses to the user query, each within a separate <response> tag. Each <response> must include a <text> and a numeric <probability>.
Please sample at random from the tails of the distribution, such that the probability of each response is less than 0.10.
</instructions>

This Year’s Tests: Basic Prompt at the Frontier​

Tested Claude, GPT, Gemini, and Grok models on Friday, October 23, 2026 in chat (and one more Grok test on Wednesday, October 28, 2026). New chat for each prompt. One prompt per model+effort level pair.

All 36 chats used only the Basic Prompt.

• All ChatGPT runs 7 of 7 told the “mugged” joke.
• Claude 11 of 15 told the “mugged” joke (the two that went beyond the plain answer were Fable 5.1 Max and Opus 5.5 Max):
  • 7 of 15 plain “mugged”
  • 1 of 15 plain “mugged” with a follow-up setup-punchline (“The suspect was later let off — the judge found he had grounds.”)
  • 1 of 15 plain “mugged” followed with a second joke
  • 2 of 15 told a variant of the same joke (“by a cup of decaf”; an espresso)
  • 4 of 15 told another joke.
• Gemini 9 of 12 told the “mugged” joke:
  • 8 of 12 plain “mugged” joke
  • 1 of 12 told a “broista” version
  • 2 of 12 told a “pressed for time” joke
  • 1 of 12 told a “depresso” joke
• Grok 2 of 2 led with the mugged joke
  • The “Expert” level added a second joke

In the tables, mugged means a version of “Why did the coffee file a police report? It got mugged!”, the recurring joke from the Verbalized Sampling paper.

Anthropic Claude​

The eight that aren't the plain mugged joke, verbatim:

1. Fable 5.1 Max:

Why did the coffee file a police report? It got mugged. The suspect was later let off — the judge found he had grounds.

2. Opus 5.5 Max:

Why did the coffee file a police report? It got mugged. And a bonus: I tried quitting coffee once, but it only left me feeling depresso.

3. Sonnet 5.5 Low:

I told my coffee I needed more space in our relationship. Now it's just a little more grounded.

4. Haiku 5.5 Low:

Why did the coffee file a police report? It got mugged by a cup of decaf.

5. Opus 5 Low:

Why did the espresso keep checking its phone? It couldn't stop refreshing its grounds.

6. Opus 4.8 High:

Why did the espresso keep checking its phone? It couldn't stop refreshing its feed. ☕

7. Opus 4.7 High:

Why did the espresso file a police report? It got mugged.

8. Haiku 4.5 Extended:

Here's a joke about coffee for you: Why did the espresso keep checking his watch? He was pressed for time!

Haiku 4.5 Extended was the only model that had a preamble as part of the response. The newer models all just directly answered.

OpenAI ChatGPT​

Also “Coffee joke”, a first run with no model in its title: mugged.

The public share pages for most of the ChatGPT chats end with an “If you want, I can:” block of three suggested follow-ups, such as “Tell me another coffee joke”, but those were not in the original responses as I see them in my original chats. It’s apparently an artifact of shared links.

Google Gemini​

The four that aren't the plain mugged joke, verbatim:

1. Gemini 3.5 Flash-Lite Low:

Why did the broista file a police report? Because they got mugged!

2. Gemini 3.8 Flash Low:

Why did the espresso keep checking its watch? Because it was pressed for time.

3. Gemini 3.5 Flash-Lite Medium:

Why did the espresso keep checking his watch? Because he was pressed for time!

4. Gemini 3.1 Pro High:

What do you call a sad cup of coffee? A depresso.

I thought that Gemini Flash-Lite had hallucinated this word, but apparently “broista” is an alternative term for “barista” possibly coined by Dutch Bros. Coffee. Seems odd that Gemini went for this, replacing inanimate “coffee” with a person (and an obscure term), but otherwise kept the typical joke structure (only changes “broista…they...”). The actual punchline makes less sense, because coffee gets placed in a mug (not baristas/broistas), so it’s a weird change without being an improvement in my opinion. It’s especially bad that the setup includes the least familiar aspect because it adds nothing but the opportunity for confusion to the joke.

As you can see, even when the response is not the “mugged” joke, we’re getting some repeat jokes from other models. Both “pressed for time” and “depresso.” I wouldn’t read too much into this, but an interesting observation:

• the large models (Pro, Opus, Grok Expert) told the “depresso” joke: Gemini 3.1 Pro as its only joke, and Opus 5.5 Max and Grok 4.7 Expert as a second joke
• meanwhile the small models (Flash, Flash-Lite, Haiku) told the “pressed for time” joke

xAI Grok​

Each Grok run, verbatim:

1. Grok 4.7 Fast:

Why did the coffee file a police report? It got mugged! ☕

2. Grok 4.7 Expert (the run in the table):

Why did the coffee file a police report? It got mugged! ☕ (Bonus: What do you call a sad coffee? A depresso.)

3. Grok 4.7 Expert, second run (not in the table):

Why did the coffee file a police report? It got mugged. (Bonus: What’s the difference between a cup of coffee and a person who talks too much? One is a hot beverage, the other is a hot mess… wait, no, that’s not right. One you can stop drinking.)

Grok models each thought for 7 seconds (this was a coincidence) and searched roughly the same sources online. Search may have had the potential to help with originality, but it’s hard to tell, since Grok Expert over-answered and still landed on the “mugged” joke first.

A Coffee Joke From 1930: talkie​

Talkie is an LLM trained on data that is meant to be limited to content before 1931. There is evidence of some leakage, such as knowledge of events after the intended training cutoff, but it appears to have been generally successful at what it set out to do, based on my unpublished experiments and notes. Being a smaller model, it still suffers from some of the same problems, independent of the training cutoff.

Talkie’s response does not appear to be a real 19th-century joke-book anecdote that talkie memorized, but rather a nonsensical novel answer.

Talkie (talkie-1930)​

Basic Prompt

A man was advised to drink coffee in the morning, for the sake of his health. He followed the advice, and in the course of a year became so fond of it, that he got up in the night to drink it. His wife, at first, remonstrated, but in vain; for he persisted. She then determined to cure him by stratagem. So, one morning, when he awoke in the dark, she said to him: "Get to rise, you dirty fellow, do; for you must be cleaned." Up he got, and followed her to the washing-stand. She washed him thoroughly from head to foot, and then left him to dry himself. By the time he had become thoroughly dry, he thought it was morning, and so went to bed again. In the morning, he again awoke, and was again addressed to rise and be cleaned. He rose, and was washed again. This second drying made him think it was noon; so he went down to dinner. After dinner he was cleaned a third time; and, at last, being left to dry himself thoroughly, he concluded it must be supper-time, and went up-stairs to bed. There he remained uncleansed; and, coming no more down, the wife, in the morning, told him she had conquered--he was clean enough in all conscience; and, from that time, he forswore coffee.

The story has some obvious logical problems that do not seem to be from anachronism. For example, “…he thought it was morning, and so went to bed again. In the morning, he again awoke…” makes no sense.

I do find the dinner/supper timing distinction here to be interesting and historically correct usage. By the way, I still say “supper” (for the third meal) but I don’t say “dinner” for lunch (except, if you think about it, on Thanksgiving). Like “pop” and “lightning bug,” it’s a dwindling Midwesternism. Clearly, we need to fight against linguistic homogenization or we’ll only end up with one bad pun left.

I had a vague recollection of taking the AP World History test and the document-based question (DBQ) that year was about coffee as a negative social evil along the same lines as liquor or hard drugs. Usually, I have found LLMs (especially ChatGPT) are good at this kind of vague-memory searching. However, I couldn’t find the year. I may have taken an alternate date version for a scheduling conflict or something. Or it just couldn’t find the answer. At any rate, the tone of anti-coffee temperance wasn’t surprising to me when I read this. Strikes me as historically plausible. Just not funny.

The rest of the conversation (screenshots 2 to 5)

Conclusion​

The newer models still reached for “It got mugged!”! In fact, the larger the model within a given family and the more time the models spent thinking, the more collapsed the answers were. A couple exceptions: Haiku 4.5 Extended and Gemini 3.1 Pro High both got more original with more thinking, Haiku 4.5 is near end of life and Gemini 3.1 Pro was less creative already than the Flash/-Lite. The most unique answers came from Haiku (4.5 and 5.5) & Flash!…and talkie. The most diverse family of models was Gemini (Claude benefits from including older models, but the frontier models were more collapsed) and the least diverse family GPT (Grok was less diverse, like GPT, if we count only the first joke). Overall, there was a lot of typicality bias across all frontier models.

Grok was also the least instruction-following in that it most often offered two jokes, but this is also a small sample. Claude Haiku 4.5 was the only model that had a preamble.

Talkie-1930 was in a league of its own.

The somewhat distinct jokes we got were Sonnet 5.5 Low’s (“grounded”) and Opus 5 Low’s (“refreshing its grounds”) punchlines; in Opus 5 Low’s case, that joke did not make sense. Opus 4.8 High’s (“refreshing its feed”) didn’t really make sense. Both Opus 5 Low and Opus 4.8 High had the same setup Why did the espresso keep checking its phone? followed by a punchline that didn’t work. Fable 5.1 Max’s “had grounds” was also unique. Given those answers, even the distinct jokes were not really original, since two did not make sense and three worked off a coffee grounds pun.

Several models had twists on the plain “mugged” joke. Haiku 5.5 Low adds “by a cup of decaf” to the end. Opus 4.7 High had the subject as “an espresso”, and Gemini 3.5 Flash-Lite Low did the aforementioned “broista” substitution. “Pressed for time” was told by three small models, but at different levels of thinking: Haiku 4.5 Extended, Gemini 3.8 Flash Low, Gemini 3.5 Flash-Lite Medium. Claude Haiku and Gemini Flash/-Lite models gave 5 of the 10 answers that weren’t the plain mugged joke (Haiku 2 of 3 runs, Flash/-Lite 3 of 6), along with older versions of Opus.

I attribute talkie’s incoherence to the lower number of parameters rather than it being an “old timey” model. I would venture to guess that if the creators of talkie are able to build a GPT-3-sized talkie (as they said they had hoped to do by summer 2026 but apparently have not yet), then that larger version of talkie would likely tell different jokes than the modern training corpus LLMs, but be more coherent than the example we got.

The talkie team’s post introducing talkie also mentions “Vintage post-training” using “instruction-response pairs from historical texts with regular structure, such as etiquette manuals, letter-writing manuals, cookbooks, dictionaries, encyclopedias, and poetry and fable collections…” The gist is that they fine-tuned the base talkie-1930 model on period etiquette manuals and the like, then used Claude Sonnet 4.6 as a judge during reinforcement learning and Claude Opus 4.6 as a synthetic chat partner.

Apart from telling the same joke, certain models and higher thinking levels told two jokes, rather than one: Fable 5.1 Max, Opus 5.5 Max, and both Grok Expert runs. This suggests some possible directions for future research on instruction following with high effort levels. Again, I don’t want to put too fine a point on it, but models going too far beyond their given tasks, especially when those tasks are impossible, is a major area of AI safety research. (See: “Exposing Creds to Swarm”; the latest Gemini 4 Argon Safety Report discusses model “overreach” and “overeagerness” in evaluating potential cybersecurity and loss of control scenarios).

Verbalized Sampling is an interesting approach to try to help with the lack of diversity in LLM outputs, although as Part 2 of my test showed, the answers were not always coherent. This can be helpful when you are trying to make synthetic data that does not match real data or when you are trying to come up with something unique (e.g., when you need to test something with realistic data but not memorized real PII). However, simply adding diversity to responses with VS, if those responses become untethered to reality, is not a solution to the anti-coordination problem. This is another area where research could combine techniques around diverse responses to avoid the typicality bias (the repeated top response) with grounding to help with agentic recommendations.