AI Agent Benchmarks and Leaderboards: What Each Measures
ZarifA leaderboard score describes a model-and-system combination on a particular task set. It is useful evidence when the task, environment and budget resemble your problem. This directory organizes eight evaluation projects by what they measure; it does not rank today's best agent.
Last verified September 17, 2026. Recheck target: monthly. Download CSV (8 entries).
Start with five
- SWE-bench variants · Check variant and leaderboard submission date before comparison
Choose repository issue resolution when the output is a code patch. Verified is a 500-instance human-filtered subset; Lite, Multilingual and Multimodal answer different questions. Record the exact split and whether the Bash Only mini-SWE-agent setting was used.
- Terminal-Bench 4.0 · Dataset and harness releases; verify the version used
Tests work in terminal environments with task-specific success checks. Version 4.0 is a different task set from 2.0, 2.1 and 3.0; retain the dataset release, agent harness, resources and attempt budget with any score.
- τ³-bench (tau2-bench repository) · Check current benchmark version and evaluation configuration
Tests interaction with a user simulator and domain policy, with text, knowledge and voice modes. Pin domain, simulator and grading version. The banking_knowledge grading correction in package v1.0.1 makes its earlier and later results non-comparable.
- OSWorld 2.1 · Check supported release manifests before every run
Covers long-horizon tasks in real computer environments. Use the osworld-v2.1 release manifest so code, task files, assets, mocked websites and provider images agree. Some task assets are gated; a repository clone alone is not a complete evaluation setup.
- BrowseComp · Pin dataset and grader; record changing web conditions
Tests finding hard-to-locate facts with short verifiable answers. Useful for research agents; it does not evaluate the quality of a long report or resolution of ambiguous user requests. Keep browser access and search budget with the result.
The directory
Scroll horizontally to compare the columns.
| Name | Role | Why it is here | Cadence |
|---|---|---|---|
| SWE-bench variants | Repository issue resolution | Choose repository issue resolution when the output is a code patch. Verified is a 500-instance human-filtered subset; Lite, Multilingual and Multimodal answer different questions. Record the exact split and whether the Bash Only mini-SWE-agent setting was used. Official variant definitions checked; no leaderboard score reproduced. Source checked September 17, 2026. | Check variant and leaderboard submission date before comparison |
| Terminal-Bench 4.0 | Terminal task completion | Tests work in terminal environments with task-specific success checks. Version 4.0 is a different task set from 2.0, 2.1 and 3.0; retain the dataset release, agent harness, resources and attempt budget with any score. GitHub release v4.0.0 exists; official benchmark index lists the 4.0 family. No benchmark run performed. Source checked September 17, 2026; recorded activity August 26, 2026. | Dataset and harness releases; verify the version used |
| τ³-bench (tau2-bench repository) | Customer-service tools, knowledge and voice | Tests interaction with a user simulator and domain policy, with text, knowledge and voice modes. Pin domain, simulator and grading version. The banking_knowledge grading correction in package v1.0.1 makes its earlier and later results non-comparable. Current README presents τ³-bench and a July grading correction; repository and CLI still retain tau2 names. Activity date is the inspected commit. Source checked September 17, 2026; recorded activity September 10, 2026. | Check current benchmark version and evaluation configuration |
| WebArena (canonical benchmark) | Browser-based tasks | Self-hosted browser tasks with environment-state evaluation. Use the canonical repository for paper reproduction; its maintainers direct newer experiment infrastructure toward AgentLab/BrowserGym. Demo websites are not the reproducible evaluation environment. Canonical research implementation; README recommends AgentLab for newer experiments. Activity date is the inspected default-branch commit, not a recent release. Source checked September 17, 2026; recorded activity November 26, 2025. | Research benchmark; pin the task and environment revision |
| AgentBench FC | Multiple agent environments | The current branch evaluates function-calling agents across containerized task families. Original AgentBench v0.1/v0.2 results and the newer FC setup are distinct. Inspect each family and its infrastructure requirements before aggregating outcomes. README introduces the FC edition in October 2025 and preserves older-version instructions below it. Activity date is the inspected commit. Source checked September 17, 2026; recorded activity February 8, 2026. | Research benchmark; inspect current repository support and setup |
| OSWorld 2.1 | Computer use across applications | Covers long-horizon tasks in real computer environments. Use the osworld-v2.1 release manifest so code, task files, assets, mocked websites and provider images agree. Some task assets are gated; a repository clone alone is not a complete evaluation setup. Direct README check identifies osworld-v2.1 as recommended; older indexed pages still name the August release. No environments provisioned. Source checked September 17, 2026; recorded activity September 16, 2026. | Check supported release manifests before every run |
| BrowseComp | Persistent web research | Tests finding hard-to-locate facts with short verifiable answers. Useful for research agents; it does not evaluate the quality of a long report or resolution of ambiguous user requests. Keep browser access and search budget with the result. Original paper and publisher explanation inspected; historical benchmark scope, not a claim of a new release. Source checked September 17, 2026. | Pin dataset and grader; record changing web conditions |
| GAIA (original benchmark) | General assistants using files and tools | Combines reasoning, browsing and files such as images or spreadsheets to reach short answers. Keep task level and validation/test split explicit. It is distinct from interactive tasks that change an application state. Original research describes 466 questions and withheld answers for 300; no claim that its historical model scores describe current systems. Source checked September 17, 2026. | Check dataset access and split before evaluation |
Choose the task before the leaderboard
For a repository patch, start with a SWE-bench variant. For terminal operations, inspect Terminal-Bench 4.0. For customer-service interaction and domain rules, inspect τ³-bench and its exact grading version. OSWorld adds application-spanning computer use; BrowseComp adds difficult web research. WebArena, AgentBench FC and GAIA offer different environment and assistant-task designs. They are not interchangeable rankings.
The evidence below was checked on September 17, 2026. A source check is separate from the recorded activity date. An older canonical benchmark can still be useful for reproduction; an unarchived repository or recent commit does not establish production suitability. No leaderboard scores were reproduced for this directory.
Three naming details matter: the tau2-bench repository now presents τ³-bench; the current AgentBench branch is its FC edition; and OSWorld’s direct README recommends osworld-v2.1, while cached search extracts still show an August release. Follow the linked maintainer evidence and pin versions before comparing results.
A comparison record worth keeping
| Field | Why it changes the interpretation |
|---|---|
| Dataset and split | A filtered subset, full set and private extension can have different difficulty |
| Model and version | A product name alone may hide a different underlying model or configuration |
| Harness and tools | Tool access, prompts, retrieval and execution constraints are part of the system |
| Attempts and budget | More retries, time or tokens can improve success while raising cost |
| Scoring procedure | Unit tests, state checks, human review and model judges catch different failures |
| Failure accounting | Excluding timeouts or invalid runs can inflate a reported success rate |
SWE-bench's site distinguishes variants and harness settings. The other maintainers publish their own task and environment definitions. Read those definitions before copying a score into a comparison table. Public task exposure and overlap with training material also limit what a leaderboard can establish about unseen work.
Turn a benchmark into a local evaluation
Choose a small task set you can reconstruct, including tasks that should fail under your current system. Record every attempt and retain the original denominator. Run a baseline, inspect failed traces and check that the success rule requires the intended outcome. Do not silently remove difficult cases between runs.
Then add tasks from your own application with the same documentation discipline. The public benchmark can reveal a capability gap; the application evaluation tells you whether a change helped the workflow you intend to ship. A passing browser task does not establish production authorization controls, and passing repository tests does not prove a patch is maintainable.
The RL environment directory covers task construction and training environments. The agent environment guide explains how that differs from a coding assistant or hosted runtime.
Change log
Expanded to eight distinct task families/projects, corrected benchmark identities and current editions, and added per-entry primary evidence. Current repository reads resolved stale search extracts; no scores were rerun.
Initial selection of 5 primary-source entries, reviewed for identity and scope. Cadence describes how to recheck; no claim that every example was reproduced.
Related Guides
- AI Conferences and Meetups: Organizer Calendars for Builders
- AI YouTube Channels for Builders: Code, Systems and Technical Talks
- MCP Servers for AI Builders: Maintainer and Reference Projects
- Amazon AI Updates: Bedrock and Alexa Changes
- The Best AI Podcasts for Staying Informed
- The Best AI YouTube Channels for Education