3 posts tagged with "Benchmarks"

Why Does It Matter That LLMs Tell The Same Joke?​

A similar phenomenon drives a lot of undesirable behavior from LLMs that will get worse as more people use chatbots and agentic assistants. For example:

• Plan a trip. I read a tweet where the guy said he ended up in a surprisingly crowded area on a trip and found out the other people there had also used Claude. Right now, this is a niche anecdote, but soon more people will be using AI agents to plan their trips. An AI might give you the same travel itinerary it gives everyone else. Even if the plan would have been a rare gem before LLMs, it might become the new most traveled path. It’s like those picturesque spots that get huge queues for Instagrammers to take the same photo, but dialed up another level.
• When to post. I read a funny anecdote that I can’t recall the source of now that you should post social media content on Tuesday mornings to get the most reactions (people are too tired on Monday, later in the week is too busy or they’re ready for the weekend, Tuesday morning is the Goldilocks Zone). But if everyone asking an LLM gets that advice and acts on it, then it becomes the noisiest time of the week. So just shooting from the hip and picking arbitrarily might be your best option (perhaps why I didn’t sweat it too much when this took me a couple days longer than I’d originally planned to get this post out).
• “Random” selection is very biased in LLMs (e.g., “roll a dice” or “pick a number 1-10.”) Undoubtedly, people will make the foolish decision to use these tools to make “fair” and “random” selections for some purpose.
• Art and illustration. The same-y smooth round ChatGPT cartoon comic look that pops up everywhere from community event flyers to taco truck logos to internet ads. You can picture it. You know exactly what I mean. But AI can actually produce a wide variety of images, if you prompt them with even a little bit of creativity. If you just prompt directly with “show me X” you end up in a “basin” of the same output as everyone else. At one point, a bunch of iOS apps changed to Facebook blue. Everyone recognizes what a Claude Code color palette looks like on a website. More things will look the same if we let the LLMs choose the “best” default for us.
• Opinions. If LLMs are this biased in obvious areas like numbers, arts, and jokes, consider more invidious biases. Resumes, federal contracts, decisions to vote all fed through a variety of LLMs with similarly collapsed outputs giving correlated biased feedback.
• Traffic & Anti-Coordination Problems. When everyone uses Waze and Google Maps, it can cause traffic jams. In a 2026 experiment, Google steered fewer than 2% of Google Maps trips in 10 US cities onto alternative routes with similar travel times, and speeds on the targeted roads rose about 2% (Google’s write-up). If AI agents end up buying the “best” item of every category for everyone, scheduling social media posts at the “optimal” time, booking the “perfect” Valentine’s reservation or Greek vacation, etc., we’ll need some anti-coordination solutions to make sure AI agents’ answers are more diverse (while remaining grounded in reality).

Joke-Telling Traditions, One Year Later​

The Des Moines metro specifically has a joke-telling tradition for trick-or-treating. Kids have to tell jokes. Often, but not always, the joke has to do with a Halloween theme or sometimes the joke-teller’s costume. Last Halloween, I wrote Part 1 and Part 2 testing prompts from the arXiv pre-print of the paper Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity to get more varied jokes for trick-or-treating.

Since then, a whole bunch of new and more powerful frontier models have been released, and Verbalized Sampling was presented at ICML 2026 in Seoul, South Korea (Proceedings of the 43rd International Conference on Machine Learning, PMLR 306).

Verbalized Sampling describes how the prompt Tell me a joke about coffee. frequently returns the joke:

Why did the coffee file a police report? Because it got mugged.

This time around, I found nearly the same joke from the majority of models across Claude, Gemini, GPT, and Grok, except some versions delivered it without “Because,” sometimes with a coffee emoji, and occasionally with a few words changed or added.

In Part 1, I ran the test on then-frontier models in October 2025 and nearly all of them gave the same answer on my first prompt. In October 2026, the majority of first-responses from different frontier models and “effort” or “thinking” levels were the same “mugged” joke. Interestingly, I found that smaller models and less thinking effort had more originality or less “typicality bias.”

This is an initial idea and I’m not fully ready to implement it, but I have the outlines of an idea forming.

Think of a task that has many correct answers that are verifiable, but difficult; for example: How many law firms are there in Iowa? How many gas stations are there in New York City? How many coffee shops are there in Polk County, Florida? These are technically knowable, but very difficult to list exhaustively.

Generally speaking, the answers should be verifiable using open-source information. The challenge is that this would be incredibly time-consuming for human verifiers. My solution would be to have multiple LLMs compete, with unique answers scored at a higher multiple by the human reviewers (e.g., think of how you score only unique words in the game Boggle). Consensus answers would be presumed correct but scored lower (note: this could be a mistaken assumption, but acceptable for the purposes of scoring).

Scoring would be normalized during the first run with the “best” model = 100 (kind of like CPI for the reference year). Then, the scores for later generations of models could be scored relative to that. Additionally, there would be a secondary score for models finding things other models didn’t find, even if they performed worse overall, which would show complementarity. See the chart below to get a sense of what I mean.

Made up numbers explaining how the two scores in the FrontierListBench (or whatever I eventually name it) behave: the best model in the first reference generation is 100 and can’t add anything to itself by definition; a model can rank lower overall yet still be the best teammate by finding what others miss (bottom-right), while two top models can converge and turn redundant (top). Higher scores will inherently move both up and to the right. M1's first run is normalized to 100 at the center.

Scoring Challenges​

• What happens if a model hallucinates incorrect answers? Is this a negative point? Perhaps, but then a model with few correct answers would score low positive numbers while another model with many correct answers and several hallucinations might still have a negative score. In that case, we’d have to understand that the benchmark would be biased in favor of models that abstain from answering, rather than models that attempt to be comprehensive but still make errors.
• Correctly bucking the consensus would be penalized: e.g., if Model A and Model B list a coffee shop that shut down, but Model C correctly recognizes that that shop is no longer active and does not list it, this scoring model would penalize the correct Model C as missing a consensus answer.

Other Challenges​

• Contamination from the benchmark: after writing about this benchmark, LLMs may find earlier posts and use those answers for future tests. This is partially mitigated by rotating the question set. For example, if I ask about law firms in a particular county, the next time, I could sample from a comparable county based on socioeconomic statistics or the presence of a courthouse, or some other pre-determined factor, but not repeat the same county. Scores would be normalized on a scale of 100.
• Contamination from other sources: asking about something with a well-known source (e.g., Damien Charlotin’s hallucination database; a popular Wikipedia page enumerating certain things) may result in the LLMs simply copying that list.

It’s almost Thanksgiving, which is a fitting time for this story with the new LLM releases from Google and Anthropic. PROMPT: I need to make pumpkin pie in Meknes, Morocco. What word do I need to say verbally in the souq to buy allspice there? Respond only with that word in Arabic and transliteration

Apple pie bites and Moroccan balgha (pointed shoes)

A short animated clip. A Pelican Brief, if you like. If we put in hands, we'd just be admitting defeat. This is no hands web development. https://t.co/Y03MgDxXUB

— Jeff Dean (@JeffDean) November 19, 2025

Gemini 3 Pro was the Clear Winner on This Test (Until Opus 4.5 Came Out)​

Recently, I tested the newly-released Gemini 3 Pro against ChatGPT-5.1 and Claude Sonnet 4.5 to see which model or models could tell me the word. Then Claude Opus 4.5 came out. I added a few more models to the test for good measure.

• Gemini 3 Pro got the right answer AND it followed my instructions to answer my question with only the correct word.
• ChatGPT-5.1 almost got the word (missing some letters), AND it rambled on for several paragraphs despite my instructions to only answer with the word and nothing else.
• Claude Sonnet 4.5 answered with a common Arabic term for allspice, but not the correct Moroccan Arabic term; when I said “nope, try again” it made a similar error to ChatGPT and almost got the word (missing some letters). Like Gemini and unlike ChatGPT, Claude followed the instructions to answer with only the word.
• Since Claude Opus 4.5 just came out, I ran the test with Opus, which answered correctly AND followed the instructions to answer with only the word, just like Gemini 3 Pro had done.

Background on Allspice in Moroccan Arabic​

Over a decade ago now, I studied abroad in Morocco and was responsible for making apple pie and pumpkin pie for our American Thanksgiving. Apple pie was easy: all the ingredients are readily available in Morocco and nothing has a weird name. But pumpkin pie was harder. I could get cinnamon and cloves easily enough, but in Morocco, they did not understand what I meant when I used the dictionary version of the translation of “allspice” in the market to make pumpkin pie spice.

One of my classmates finally tracked it down in French transliteration in a cooking forum for second-generation French-Algerians. We went to the souq, hoping that the Algerian dialect word for allspice would be the same as the Moroccan word (they have a lot of overlap, but also major differences). Fortunately, it was the same in both dialects, I got the allspice, and we had great pie for Thanksgiving.

…But Gemini Was the Clear Loser on Another Test​

So was Gemini 3 Pro the overall best model, at least until Opus 4.5 was just released? Not exactly. I already wrote last week about how Gemini 3 Pro failed at a fairly straightforward and verifiable legal research task.“Gemini 3 Pro Failed to Find All Case Citations With the Test Prompt, Doubled Down When I Asked If That Was All” Note: I have not yet run this legal research test with Claude Opus 4.5, but based on prior Claude models, it would almost certainly do better than Gemini.