AI Agent Benchmarks and Leaderboards: What Each Measures
Eight public tests for AI agents, sorted by what each measures: fixing code, terminal work, browsing, computer use and customer service.
This is a guide to eight public tests for AI agents, sorted by what each one actually measures: fixing code, working in a terminal, using a computer or browser, or handling a customer conversation. Start with the one closest to the work you want an agent to do, and treat its leaderboard as a clue, not a verdict.
A leaderboard score describes one model, running in one setup, on one set of tasks. It's useful when those tasks, tools and budget look like yours. This page doesn't rank today's best agent.
Last verified Sep 17, 2026. Recheck target: monthly. The CSV holds all 8 entries.
Start with five
The one to use when an agent's job is to turn a reported issue into a working code patch. Verified is 500 tasks screened by humans, while Lite, Multilingual and Multimodal each test something different. Write down which split you ran and whether you used the Bash Only mini-SWE-agent setup.
Tests whether an agent can finish real jobs at the command line, with a pass-or-fail check written for each task. Version 4.0 is a different set of tasks from 2.0, 2.1 and 3.0, so a score only means something next to the version, the agent setup, the machine it ran on and how many tries it got.
Puts an agent in a customer-service conversation with a simulated customer and a company policy it has to follow, in text, knowledge and voice modes. Note the domain, simulator and grading version you used. A grading fix to banking_knowledge in package v1.0.1 means results from before and after it can't be compared.
Tests long, multi-step tasks on a real computer. Use the osworld-v2.1 release manifest so the code, task files, assets, mock websites and machine images all match. Some task files are restricted, so cloning the repo alone won't give you a full test setup.
Tests whether an agent can track down facts that are hard to find but easy to check. Good for research agents, but it says nothing about writing a long report or handling a vague request. Keep the browser access and search budget next to any result.
The directory
The one to use when an agent's job is to turn a reported issue into a working code patch. Verified is 500 tasks screened by humans, while Lite, Multilingual and Multimodal each test something different. Write down which split you ran and whether you used the Bash Only mini-SWE-agent setup.
Official variant definitions checked; no leaderboard score reproduced.
Tests whether an agent can finish real jobs at the command line, with a pass-or-fail check written for each task. Version 4.0 is a different set of tasks from 2.0, 2.1 and 3.0, so a score only means something next to the version, the agent setup, the machine it ran on and how many tries it got.
GitHub release v4.0.0 exists; official benchmark index lists the 4.0 family. No benchmark run performed.
Puts an agent in a customer-service conversation with a simulated customer and a company policy it has to follow, in text, knowledge and voice modes. Note the domain, simulator and grading version you used. A grading fix to banking_knowledge in package v1.0.1 means results from before and after it can't be compared.
Current README presents τ³-bench and a July grading correction; repository and CLI still retain tau2 names. Activity date is the inspected commit.
Gives an agent tasks on websites you host yourself, then checks the state of each site to see if the job got done. Use this original repo to reproduce the paper, but its maintainers point newer experiments to AgentLab/BrowserGym. The public demo sites are not the test setup, so don't score against them.
Canonical research implementation; README recommends AgentLab for newer experiments. Activity date is the inspected default-branch commit, not a recent release.
The current version tests agents that call functions across several task types, each running in its own container. Results from the original v0.1 and v0.2 are not the same thing as this FC setup, and each task type needs its own infrastructure, so check them one by one before adding scores together.
README introduces the FC edition in October 2025 and preserves older-version instructions below it. Activity date is the inspected commit.
Tests long, multi-step tasks on a real computer. Use the osworld-v2.1 release manifest so the code, task files, assets, mock websites and machine images all match. Some task files are restricted, so cloning the repo alone won't give you a full test setup.
Direct README check identifies osworld-v2.1 as recommended; older indexed pages still name the August release. No environments provisioned.
Tests whether an agent can track down facts that are hard to find but easy to check. Good for research agents, but it says nothing about writing a long report or handling a vague request. Keep the browser access and search budget next to any result.
Original paper and publisher explanation inspected; historical benchmark scope, not a claim of a new release.
Asks questions that take reasoning, web browsing and reading files like images or spreadsheets to reach a short answer. Say which task level and which split, validation or test, you used. It doesn't test tasks where the agent has to change something inside an app.
Original research describes 466 questions and withheld answers for 300; no claim that its historical model scores describe current systems.
Choose the task before the leaderboard
Pick by the job. For fixing code in a repository, start with a SWE-bench variant. For terminal work, look at Terminal-Bench 4.0. For customer-service conversations that have to follow business rules, look at τ³-bench and its exact grading version.
OSWorld covers using a computer across several apps. BrowseComp covers hard web research. WebArena, AgentBench FC and GAIA each test a different environment or kind of assistant task. They aren't interchangeable rankings.
Everything below was checked on September 17, 2026. That's separate from the activity date on each entry. An older benchmark can still be useful for reproducing results, and a recent commit doesn't make a project production-ready. No leaderboard scores were rerun for this page.
Three naming details trip people up. The tau2-bench repository now presents τ³-bench. The current AgentBench branch is its FC edition. And OSWorld's README recommends osworld-v2.1, while cached search results still show an August release. Follow the maintainer links and pin versions before you compare results.
What to write down when comparing scores
| Detail | Why it changes what the score means |
|---|---|
| Dataset and split | A filtered subset, the full set and a private extension can differ in difficulty |
| Model and version | A product name can hide a different model or setting underneath |
| Harness and tools | The harness is the code that runs the agent. Its tools, prompts, search and limits are part of what got tested |
| Attempts and budget | More retries, time or tokens can raise the success rate and the cost |
| Scoring | Unit tests, state checks, human review and model judges each catch different failures |
| Failure accounting | Leaving out timeouts or invalid runs can inflate a success rate |
SWE-bench's site separates its variants and harness settings. The other maintainers publish their own task and environment definitions. Read them before you copy a score into a comparison table.
Public tasks may also have ended up in a model's training data. That limits what any leaderboard can tell you about work the model hasn't seen.
Turn a benchmark into your own test
Pick a small set of tasks you can rebuild, including some your current system should fail. Record every attempt and keep the original total, so failures can't quietly drop out of the count. Run a baseline, read the failed runs, and check that "success" really requires the outcome you wanted. Don't remove hard cases between runs.
Then add tasks from your own app, recorded the same way. A public benchmark can show you a gap. Your own test tells you whether a change helped the thing you plan to ship. Passing a browser task doesn't prove your permission controls work, and passing a repository's tests doesn't prove a patch is maintainable.
The RL environment directory covers building tasks and training environments. The agent environment guide explains how that differs from a coding assistant or a hosted runtime.
Change log
Expanded to eight distinct task families/projects, corrected benchmark identities and current editions, and added per-entry primary evidence. Current repository reads resolved stale search extracts; no scores were rerun.
Initial selection of 5 primary-source entries, reviewed for identity and scope. Cadence describes how to recheck; no claim that every example was reproduced.
Follow changes to every directory with the directory changes RSS feed.