Skip to content
Back to Understanding models

Understanding models · Chapter 7 of 8

Comparing models

Benchmarks and leaderboards help you compare models, but margins of error, effort settings, cost, leaked tests, maxed-out tests, the test program and who ran it all change what a score means.

15 minutes0 of 4 goals done

Goals

  • Read a leaderboard with its intervals and find the ties.To do
  • Compare one model at five effort settings with a switch to another model.To do
  • Match four benchmark traps to examples.To do
  • Pass the check.To do

Step 1 of 4: The read

Benchmarks and leaderboards

A benchmark is a fixed set of test tasks with a way to score the answers. A leaderboard ranks models by such scores. The visual uses two real ones. On Arena, people send a prompt, read two answers from models they cannot see, and vote for the one they prefer; millions of votes become a score for each model. Artificial Analysis, an independent company, runs nine benchmarks itself and combines them into one number, its Intelligence Index.

Every score has a margin

A score from votes, or from a sample of tasks, is an estimate. Arena writes it as a score ± a margin: the true score most likely lies in that range, called a 95% interval. Fewer votes give a wider range. When two models' ranges overlap, the board cannot tell them apart, so treat them as tied, whatever their rank.

Settings and cost

Many models have an effort setting, such as low, medium, high or max: how much the model may think before it answers, as in chapter 3. More effort often scores higher, but the model writes more tokens, so each task costs more and takes longer. Artificial Analysis also shows a cost per task: what running the model on one of its tasks cost, in US dollars. A score means little without its setting and its cost.

Traps

Contamination: the test's questions and answers leaked into the training data, so a model may remember them instead of solving them. Saturation: the top models all score near the maximum, so the test no longer separates them. The harness is the program that runs the test, with its instructions, tools and retries; a better harness can raise the score of the same model. A maker's own score is run its own way and can differ from an independent one. Version drift: tests change between versions, so a score needs its version and its date.

In the visual

The Arena and Artificial Analysis figures are real, as published, in a snapshot dated 2026-11-03. The chart's cost axis is a log scale: $0.10 to $1 takes the same width as $1 to $10, so cheap and costly models both fit. The examples in the last step are made up.

Step 2 of 4: Try it: Reading benchmark scores

Reading benchmark scores

Find the ties on a real leaderboard, climb one model's effort ladder, then match four traps to made-up examples.

Mode:RealSimulatedReal: the leaderboard and index figures are as Arena and Artificial Analysis published them, in a snapshot dated 2026-11-03; ties and gaps are worked out from them in this page. Simulated: the four examples in the last step are made up for this chapter and name no real model or test.

Step 3 of 4: Check

Check

Question 1 of 3In an example, model A scores 1,310 ± 8 on a leaderboard and model B scores 1,302 ± 9. What can you say?

Next: Question 2 of 3

Question 2 of 3A report says model X scores 5 points higher than model Y on an index. What should you ask before you switch to X?
Question 3 of 3A blog post puts model A's score on the current version of a test next to model B's score on the version from a year ago. A's number is higher. What can you conclude?

Step 4 of 4: What you can do now

What you can do now

  • Read a leaderboard's intervals and say which models it cannot tell apart.
  • Compare models at named effort settings, together with their cost per task.
  • Spot contamination, saturation, harness effects and a maker's own scores in a claim.
  • Ask for the version, the date and who ran the test before trusting a score.