# AI Agent Benchmarks and Leaderboards: What Each Measures

> Compare coding, terminal, browser and tool-interaction benchmarks by task, harness and scoring assumptions, with primary sources and CSV export.

- Source: https://www.zarifautomates.com/blog/ai-agent-benchmarks-and-leaderboards
- Published: 2026-09-17
- Updated: 2026-09-17
- Pillar: Sources & Directories
- Tags: ai-engineering, directory, builders
- Author: Zarif

---

A leaderboard score describes a model-and-system combination on a particular task set. It is useful evidence when the task, environment and budget resemble your problem. This directory organizes eight evaluation projects by what they measure; it does not rank today's best agent.

Last verified 2026-09-17. Recheck target: monthly. Download: https://www.zarifautomates.com/downloads/directories/ai-agent-benchmarks-and-leaderboards.csv.

## Start with five

1. [SWE-bench variants](https://www.swebench.com/) — Check variant and leaderboard submission date before comparison. Choose repository issue resolution when the output is a code patch. Verified is a 500-instance human-filtered subset; Lite, Multilingual and Multimodal answer different questions. Record the exact split and whether the Bash Only mini-SWE-agent setting was used.
2. [Terminal-Bench 4.0](https://www.tbench.ai/benchmarks) — Dataset and harness releases; verify the version used. Tests work in terminal environments with task-specific success checks. Version 4.0 is a different task set from 2.0, 2.1 and 3.0; retain the dataset release, agent harness, resources and attempt budget with any score.
3. [τ³-bench (tau2-bench repository)](https://github.com/sierra-research/tau2-bench) — Check current benchmark version and evaluation configuration. Tests interaction with a user simulator and domain policy, with text, knowledge and voice modes. Pin domain, simulator and grading version. The banking_knowledge grading correction in package v1.0.1 makes its earlier and later results non-comparable.
4. [OSWorld 2.1](https://github.com/xlang-ai/OSWorld-V2) — Check supported release manifests before every run. Covers long-horizon tasks in real computer environments. Use the osworld-v2.1 release manifest so code, task files, assets, mocked websites and provider images agree. Some task assets are gated; a repository clone alone is not a complete evaluation setup.
5. [BrowseComp](https://openai.com/index/browsecomp/) — Pin dataset and grader; record changing web conditions. Tests finding hard-to-locate facts with short verifiable answers. Useful for research agents; it does not evaluate the quality of a long report or resolution of ambiguous user requests. Keep browser access and search budget with the result.

## The directory

| Name | Role | Why it is here | Cadence |
| --- | --- | --- | --- |
| [SWE-bench variants](https://www.swebench.com/) | Repository issue resolution | Choose repository issue resolution when the output is a code patch. Verified is a 500-instance human-filtered subset; Lite, Multilingual and Multimodal answer different questions. Record the exact split and whether the Bash Only mini-SWE-agent setting was used. Official variant definitions checked; no leaderboard score reproduced. [Source checked 2026-09-17](https://www.swebench.com/). | Check variant and leaderboard submission date before comparison |
| [Terminal-Bench 4.0](https://www.tbench.ai/benchmarks) | Terminal task completion | Tests work in terminal environments with task-specific success checks. Version 4.0 is a different task set from 2.0, 2.1 and 3.0; retain the dataset release, agent harness, resources and attempt budget with any score. GitHub release v4.0.0 exists; official benchmark index lists the 4.0 family. No benchmark run performed. [Source checked 2026-09-17](https://github.com/harbor-framework/terminal-bench/releases/tag/v4.0.0); recorded activity 2026-08-26. | Dataset and harness releases; verify the version used |
| [τ³-bench (tau2-bench repository)](https://github.com/sierra-research/tau2-bench) | Customer-service tools, knowledge and voice | Tests interaction with a user simulator and domain policy, with text, knowledge and voice modes. Pin domain, simulator and grading version. The banking_knowledge grading correction in package v1.0.1 makes its earlier and later results non-comparable. Current README presents τ³-bench and a July grading correction; repository and CLI still retain tau2 names. Activity date is the inspected commit. [Source checked 2026-09-17](https://github.com/sierra-research/tau2-bench/blob/2174a603f6d014ef94473ffa95957f6ce27100db/README.md); recorded activity 2026-09-10. | Check current benchmark version and evaluation configuration |
| [WebArena (canonical benchmark)](https://github.com/web-arena-x/webarena) | Browser-based tasks | Self-hosted browser tasks with environment-state evaluation. Use the canonical repository for paper reproduction; its maintainers direct newer experiment infrastructure toward AgentLab/BrowserGym. Demo websites are not the reproducible evaluation environment. Canonical research implementation; README recommends AgentLab for newer experiments. Activity date is the inspected default-branch commit, not a recent release. [Source checked 2026-09-17](https://github.com/web-arena-x/webarena/blob/dce04686a56253aefba7b18a4fa0937cf1dc987b/README.md); recorded activity 2025-11-26. | Research benchmark; pin the task and environment revision |
| [AgentBench FC](https://github.com/THUDM/AgentBench) | Multiple agent environments | The current branch evaluates function-calling agents across containerized task families. Original AgentBench v0.1/v0.2 results and the newer FC setup are distinct. Inspect each family and its infrastructure requirements before aggregating outcomes. README introduces the FC edition in October 2025 and preserves older-version instructions below it. Activity date is the inspected commit. [Source checked 2026-09-17](https://github.com/THUDM/AgentBench/blob/d1e4a10db08c87075c78972e48ecc182be03e2d5/README.md); recorded activity 2026-02-08. | Research benchmark; inspect current repository support and setup |
| [OSWorld 2.1](https://github.com/xlang-ai/OSWorld-V2) | Computer use across applications | Covers long-horizon tasks in real computer environments. Use the osworld-v2.1 release manifest so code, task files, assets, mocked websites and provider images agree. Some task assets are gated; a repository clone alone is not a complete evaluation setup. Direct README check identifies osworld-v2.1 as recommended; older indexed pages still name the August release. No environments provisioned. [Source checked 2026-09-17](https://github.com/xlang-ai/OSWorld-V2/blob/main/README.md); recorded activity 2026-09-16. | Check supported release manifests before every run |
| [BrowseComp](https://openai.com/index/browsecomp/) | Persistent web research | Tests finding hard-to-locate facts with short verifiable answers. Useful for research agents; it does not evaluate the quality of a long report or resolution of ambiguous user requests. Keep browser access and search budget with the result. Original paper and publisher explanation inspected; historical benchmark scope, not a claim of a new release. [Source checked 2026-09-17](https://arxiv.org/html/2504.12516). | Pin dataset and grader; record changing web conditions |
| [GAIA (original benchmark)](https://huggingface.co/gaia-benchmark) | General assistants using files and tools | Combines reasoning, browsing and files such as images or spreadsheets to reach short answers. Keep task level and validation/test split explicit. It is distinct from interactive tasks that change an application state. Original research describes 466 questions and withheld answers for 300; no claim that its historical model scores describe current systems. [Source checked 2026-09-17](https://ai.meta.com/research/publications/gaia-a-benchmark-for-general-ai-assistants/). | Check dataset access and split before evaluation |

## Choose the task before the leaderboard

For a repository patch, start with a SWE-bench variant. For terminal operations, inspect Terminal-Bench 4.0. For customer-service interaction and domain rules, inspect τ³-bench and its exact grading version. OSWorld adds application-spanning computer use; BrowseComp adds difficult web research. WebArena, AgentBench FC and GAIA offer different environment and assistant-task designs. They are not interchangeable rankings.

The evidence below was checked on September 17, 2026. A source check is separate from the recorded activity date. An older canonical benchmark can still be useful for reproduction; an unarchived repository or recent commit does not establish production suitability. No leaderboard scores were reproduced for this directory.

Three naming details matter: the `tau2-bench` repository now presents τ³-bench; the current AgentBench branch is its FC edition; and OSWorld’s direct README recommends `osworld-v2.1`, while cached search extracts still show an August release. Follow the linked maintainer evidence and pin versions before comparing results.

## A comparison record worth keeping

| Field | Why it changes the interpretation |
| --- | --- |
| Dataset and split | A filtered subset, full set and private extension can have different difficulty |
| Model and version | A product name alone may hide a different underlying model or configuration |
| Harness and tools | Tool access, prompts, retrieval and execution constraints are part of the system |
| Attempts and budget | More retries, time or tokens can improve success while raising cost |
| Scoring procedure | Unit tests, state checks, human review and model judges catch different failures |
| Failure accounting | Excluding timeouts or invalid runs can inflate a reported success rate |

SWE-bench's site distinguishes variants and harness settings. The other maintainers publish their own task and environment definitions. Read those definitions before copying a score into a comparison table. Public task exposure and overlap with training material also limit what a leaderboard can establish about unseen work.

## Turn a benchmark into a local evaluation

Choose a small task set you can reconstruct, including tasks that should fail under your current system. Record every attempt and retain the original denominator. Run a baseline, inspect failed traces and check that the success rule requires the intended outcome. Do not silently remove difficult cases between runs.

Then add tasks from your own application with the same documentation discipline. The public benchmark can reveal a capability gap; the application evaluation tells you whether a change helped the workflow you intend to ship. A passing browser task does not establish production authorization controls, and passing repository tests does not prove a patch is maintainable.

The [RL environment directory](/blog/rl-environments-for-coding-agents) covers task construction and training environments. The [agent environment guide](/blog/best-ai-agent-development-environments) explains how that differs from a coding assistant or hosted runtime.

## Change log

- 2026-09-17: Expanded to eight distinct task families/projects, corrected benchmark identities and current editions, and added per-entry primary evidence. Current repository reads resolved stale search extracts; no scores were rerun.
- 2026-09-17: Initial selection of 5 primary-source entries, reviewed for identity and scope. Cadence describes how to recheck; no claim that every example was reproduced.

## Related Guides

- [AI Conferences and Meetups: Organizer Calendars for Builders](/blog/ai-conferences-and-meetups-for-builders)
- [AI YouTube Channels for Builders: Code, Systems and Technical Talks](/blog/ai-youtube-channels-for-builders)
- [MCP Servers for AI Builders: Maintainer and Reference Projects](/blog/mcp-servers-for-ai-builders)
- [Amazon AI Updates: Bedrock and Alexa Changes](/blog/amazon-ai-updates-bedrock-alexa)
- [The Best AI Podcasts for Staying Informed](/blog/best-ai-podcasts-for-staying-informed)
- [The Best AI YouTube Channels for Education](/blog/best-ai-youtube-channels-for-education)