noun · AI practice
Benchmark
also called leaderboard test
A standard exam for AI models. Everyone takes the same test, so you can compare scores, but the test may not match your job.
A standardized set of tasks and scoring used to compare the performance of different models.
A fixed set of tasks with agreed scoring, run against many models so their results can be compared. Scores depend on prompts and settings, and any test that leaks into training data stops measuring what it was meant to.
Real-life example
You pick a model because it tops a popular leaderboard. On your own task, summarizing messy support tickets, it does no better than a cheaper model. The benchmark measured something real, but not the thing you needed.
Where the analogy breaks
-
A high score does not mean it is best for your task. A benchmark measures its own questions, which may be far from your work.
-
Scores can be inflated. If the test questions appeared in training data, or the model was tuned to the test, the number overstates ability.
-
Comparisons need the same setup. Different prompts, settings or scoring rules can change a model's rank.