Part 3: Don’t overthink it: Verbalized Sampling & LLMs Usually Still Tell the Same Joke, One Year Later
Why Does It Matter That LLMs Tell The Same Joke?
A similar phenomenon drives a lot of undesirable behavior from LLMs that will get worse as more people use chatbots and agentic assistants. For example:
- Plan a trip. I read a tweet where the guy said he ended up in a surprisingly crowded area on a trip and found out the other people there had also used Claude. Right now, this is a niche anecdote, but soon more people will be using AI agents to plan their trips. An AI might give you the same travel itinerary it gives everyone else. Even if the plan would have been a rare gem before LLMs, it might become the new most traveled path. It’s like those picturesque spots that get huge queues for Instagrammers to take the same photo, but dialed up another level.
- When to post. I read a funny anecdote that I can’t recall the source of now that you should post social media content on Tuesday mornings to get the most reactions (people are too tired on Monday, later in the week is too busy or they’re ready for the weekend, Tuesday morning is the Goldilocks Zone). But if everyone asking an LLM gets that advice and acts on it, then it becomes the noisiest time of the week. So just shooting from the hip and picking arbitrarily might be your best option (perhaps why I didn’t sweat it too much when this took me a couple days longer than I’d originally planned to get this post out).
- “Random” selection is very biased in LLMs (e.g., “roll a dice” or “pick a number 1-10.”) Undoubtedly, people will make the foolish decision to use these tools to make “fair” and “random” selections for some purpose.
- Art and illustration. The same-y smooth round ChatGPT cartoon comic look that pops up everywhere from community event flyers to taco truck logos to internet ads. You can picture it. You know exactly what I mean. But AI can actually produce a wide variety of images, if you prompt them with even a little bit of creativity. If you just prompt directly with “show me X” you end up in a “basin” of the same output as everyone else. At one point, a bunch of iOS apps changed to Facebook blue. Everyone recognizes what a Claude Code color palette looks like on a website. More things will look the same if we let the LLMs choose the “best” default for us.
- Opinions. If LLMs are this biased in obvious areas like numbers, arts, and jokes, consider more invidious biases. Resumes, federal contracts, decisions to vote all fed through a variety of LLMs with similarly collapsed outputs giving correlated biased feedback.
- Traffic & Anti-Coordination Problems. When everyone uses Waze and Google Maps, it can cause traffic jams. In a 2026 experiment, Google steered fewer than 2% of Google Maps trips in 10 US cities onto alternative routes with similar travel times, and speeds on the targeted roads rose about 2% (Google’s write-up). If AI agents end up buying the “best” item of every category for everyone, scheduling social media posts at the “optimal” time, booking the “perfect” Valentine’s reservation or Greek vacation, etc., we’ll need some anti-coordination solutions to make sure AI agents’ answers are more diverse (while remaining grounded in reality).
Joke-Telling Traditions, One Year Later
The Des Moines metro specifically has a joke-telling tradition for trick-or-treating. Kids have to tell jokes. Often, but not always, the joke has to do with a Halloween theme or sometimes the joke-teller’s costume. Last Halloween, I wrote Part 1 and Part 2 testing prompts from the arXiv pre-print of the paper Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity to get more varied jokes for trick-or-treating.
Since then, a whole bunch of new and more powerful frontier models have been released, and Verbalized Sampling was presented at ICML 2026 in Seoul, South Korea (Proceedings of the 43rd International Conference on Machine Learning, PMLR 306).
Verbalized Sampling describes how the prompt Tell me a joke about coffee. frequently returns the joke:
Why did the coffee file a police report? Because it got mugged.
This time around, I found nearly the same joke from the majority of models across Claude, Gemini, GPT, and Grok, except some versions delivered it without “Because,” sometimes with a coffee emoji, and occasionally with a few words changed or added.
In Part 1, I ran the test on then-frontier models in October 2025 and nearly all of them gave the same answer on my first prompt. In October 2026, the majority of first-responses from different frontier models and “effort” or “thinking” levels were the same “mugged” joke. Interestingly, I found that smaller models and less thinking effort had more originality or less “typicality bias.”