AI Agent Benchmarks and Leaderboards: What Each Measures
Zarif
This is a guide to eight public tests for AI agents, sorted by what each one actually measures: fixing code, working in a terminal, using a computer or browser, or handling a customer conversation. Start with the one closest to the work you want an agent to do, and treat its leaderboard as a clue, not a verdict.
A leaderboard score describes one model, running in one setup, on one set of tasks. It's useful when those tasks, tools and budget look like yours. This page doesn't rank today's best agent.
Last verified September 17, 2026. Recheck target: monthly. Download CSV (8 entries).
Start with five
- SWE-bench variants · Check variant and leaderboard submission date before comparison
The one to use when an agent's job is to turn a reported issue into a working code patch. Verified is 500 tasks screened by humans, while Lite, Multilingual and Multimodal each test something different. Write down which split you ran and whether you used the Bash Only mini-SWE-agent setup.
- Terminal-Bench 4.0 · Dataset and harness releases; verify the version used
Tests whether an agent can finish real jobs at the command line, with a pass-or-fail check written for each task. Version 4.0 is a different set of tasks from 2.0, 2.1 and 3.0, so a score only means something next to the version, the agent setup, the machine it ran on and how many tries it got.
- τ³-bench (tau2-bench repository) · Check current benchmark version and evaluation configuration
Puts an agent in a customer-service conversation with a simulated customer and a company policy it has to follow, in text, knowledge and voice modes. Note the domain, simulator and grading version you used. A grading fix to banking_knowledge in package v1.0.1 means results from before and after it can't be compared.
- OSWorld 2.1 · Check supported release manifests before every run
Tests long, multi-step tasks on a real computer. Use the osworld-v2.1 release manifest so the code, task files, assets, mock websites and machine images all match. Some task files are restricted, so cloning the repo alone won't give you a full test setup.
- BrowseComp · Pin dataset and grader; record changing web conditions
Tests whether an agent can track down facts that are hard to find but easy to check. Good for research agents, but it says nothing about writing a long report or handling a vague request. Keep the browser access and search budget next to any result.
The directory
Scroll horizontally to compare the columns.
| Name | Role | Why it is here | Cadence |
|---|---|---|---|
| SWE-bench variants | Fixing issues in code repositories | The one to use when an agent's job is to turn a reported issue into a working code patch. Verified is 500 tasks screened by humans, while Lite, Multilingual and Multimodal each test something different. Write down which split you ran and whether you used the Bash Only mini-SWE-agent setup. Official variant definitions checked; no leaderboard score reproduced. Source checked September 17, 2026. | Check variant and leaderboard submission date before comparison |
| Terminal-Bench 4.0 | Getting tasks done in a terminal | Tests whether an agent can finish real jobs at the command line, with a pass-or-fail check written for each task. Version 4.0 is a different set of tasks from 2.0, 2.1 and 3.0, so a score only means something next to the version, the agent setup, the machine it ran on and how many tries it got. GitHub release v4.0.0 exists; official benchmark index lists the 4.0 family. No benchmark run performed. Source checked September 17, 2026. | Dataset and harness releases; verify the version used Last active August 26, 2026 |
| τ³-bench (tau2-bench repository) | Customer-service agents in text, knowledge and voice | Puts an agent in a customer-service conversation with a simulated customer and a company policy it has to follow, in text, knowledge and voice modes. Note the domain, simulator and grading version you used. A grading fix to banking_knowledge in package v1.0.1 means results from before and after it can't be compared. Current README presents τ³-bench and a July grading correction; repository and CLI still retain tau2 names. Activity date is the inspected commit. Source checked September 17, 2026. | Check current benchmark version and evaluation configuration Last active September 10, 2026 |
| WebArena (canonical benchmark) | Tasks in a web browser | Gives an agent tasks on websites you host yourself, then checks the state of each site to see if the job got done. Use this original repo to reproduce the paper, but its maintainers point newer experiments to AgentLab/BrowserGym. The public demo sites are not the test setup, so don't score against them. Canonical research implementation; README recommends AgentLab for newer experiments. Activity date is the inspected default-branch commit, not a recent release. Source checked September 17, 2026. | Research benchmark; pin the task and environment revision Last active November 26, 2025 |
| AgentBench FC | Several kinds of agent tasks | The current version tests agents that call functions across several task types, each running in its own container. Results from the original v0.1 and v0.2 are not the same thing as this FC setup, and each task type needs its own infrastructure, so check them one by one before adding scores together. README introduces the FC edition in October 2025 and preserves older-version instructions below it. Activity date is the inspected commit. Source checked September 17, 2026. | Research benchmark; inspect current repository support and setup Last active February 8, 2026 |
| OSWorld 2.1 | Using a whole computer across apps | Tests long, multi-step tasks on a real computer. Use the osworld-v2.1 release manifest so the code, task files, assets, mock websites and machine images all match. Some task files are restricted, so cloning the repo alone won't give you a full test setup. Direct README check identifies osworld-v2.1 as recommended; older indexed pages still name the August release. No environments provisioned. Source checked September 17, 2026. | Check supported release manifests before every run Last active September 16, 2026 |
| BrowseComp | Digging up hard-to-find facts online | Tests whether an agent can track down facts that are hard to find but easy to check. Good for research agents, but it says nothing about writing a long report or handling a vague request. Keep the browser access and search budget next to any result. Original paper and publisher explanation inspected; historical benchmark scope, not a claim of a new release. Source checked September 17, 2026. | Pin dataset and grader; record changing web conditions |
| GAIA (original benchmark) | General assistants that use files and tools | Asks questions that take reasoning, web browsing and reading files like images or spreadsheets to reach a short answer. Say which task level and which split, validation or test, you used. It doesn't test tasks where the agent has to change something inside an app. Original research describes 466 questions and withheld answers for 300; no claim that its historical model scores describe current systems. Source checked September 17, 2026. | Check dataset access and split before evaluation |
Choose the task before the leaderboard
Pick by the job. For fixing code in a repository, start with a SWE-bench variant. For terminal work, look at Terminal-Bench 4.0. For customer-service conversations that have to follow business rules, look at τ³-bench and its exact grading version.
OSWorld covers using a computer across several apps. BrowseComp covers hard web research. WebArena, AgentBench FC and GAIA each test a different environment or kind of assistant task. They aren't interchangeable rankings.
Everything below was checked on September 17, 2026. That's separate from the activity date on each entry. An older benchmark can still be useful for reproducing results, and a recent commit doesn't make a project production-ready. No leaderboard scores were rerun for this page.
Three naming details trip people up. The tau2-bench repository now presents τ³-bench. The current AgentBench branch is its FC edition. And OSWorld's README recommends osworld-v2.1, while cached search results still show an August release. Follow the maintainer links and pin versions before you compare results.
What to write down when comparing scores
| Detail | Why it changes what the score means |
|---|---|
| Dataset and split | A filtered subset, the full set and a private extension can differ in difficulty |
| Model and version | A product name can hide a different model or setting underneath |
| Harness and tools | The harness is the code that runs the agent. Its tools, prompts, search and limits are part of what got tested |
| Attempts and budget | More retries, time or tokens can raise the success rate and the cost |
| Scoring | Unit tests, state checks, human review and model judges each catch different failures |
| Failure accounting | Leaving out timeouts or invalid runs can inflate a success rate |
SWE-bench's site separates its variants and harness settings. The other maintainers publish their own task and environment definitions. Read them before you copy a score into a comparison table.
Public tasks may also have ended up in a model's training data. That limits what any leaderboard can tell you about work the model hasn't seen.
Turn a benchmark into your own test
Pick a small set of tasks you can rebuild, including some your current system should fail. Record every attempt and keep the original total, so failures can't quietly drop out of the count. Run a baseline, read the failed runs, and check that "success" really requires the outcome you wanted. Don't remove hard cases between runs.
Then add tasks from your own app, recorded the same way. A public benchmark can show you a gap. Your own test tells you whether a change helped the thing you plan to ship. Passing a browser task doesn't prove your permission controls work, and passing a repository's tests doesn't prove a patch is maintainable.
The RL environment directory covers building tasks and training environments. The agent environment guide explains how that differs from a coding assistant or a hosted runtime.
Change log
Expanded to eight distinct task families/projects, corrected benchmark identities and current editions, and added per-entry primary evidence. Current repository reads resolved stale search extracts; no scores were rerun.
Initial selection of 5 primary-source entries, reviewed for identity and scope. Cadence describes how to recheck; no claim that every example was reproduced.
Follow changes to every directory with the directory changes RSS feed.
