Agent Evaluation Tools: Compare Five Options on One Ticket Task
ZarifThe first question for an evaluation tool is whether it preserves the case that failed. A dashboard showing one average is less useful if you cannot recover the input, output, reference, error and scoring rule behind it.
This comparison uses the same ten-ticket task from the evaluation lesson. The plain Python harness and Pydantic Evals adapter were executed locally. Promptfoo, LangSmith and Braintrust are assessed from their official documentation, checked September 17, 2026. They were not run for this comparison. There is no product performance ranking here.
Fix the task before comparing the tools
The input is a synthetic support message. The candidate must return billing, technical or abstain, with a literal evidence quote for an action label and null for abstention. Scoring checks the output shape, evidence and reference label separately. One of the ten cases raises an injected timeout.
Every candidate tool must preserve the same ten case IDs and four scores, keep the timeout in the denominator, and retain the model or implementation configuration. The classifier receives only the input. Reference labels stay with the evaluator. Those requirements give the comparison a concrete acceptance test.
Download the shared evaluation artifacts. They include cases.json, the classifier and scorer in evaluate.py, the actual Pydantic adapter, pinned requirements and both result files.
Compare what you would implement
| Option | How this ticket task maps to it | Evidence in this lesson |
|---|---|---|
| Plain Python | Loop over cases.json, call predict(), then score(); write a JSON report and explicit exit status | Executed: 3/10 and 8/10 for two deterministic baselines |
| Pydantic Evals | One Case per ticket, a Dataset, the task function and a custom Evaluator using the same scorer | Executed with 2.44.0: conservative baseline 8/10 |
| Promptfoo | Test variables hold ticket inputs; a custom Python provider calls the candidate; Python assertions apply the scoring rules | Official provider and assertion contracts reviewed; no local run |
| LangSmith | Dataset examples supply inputs and references; an experiment records task runs and custom evaluator results | Official evaluation and result-processing docs reviewed; no local or hosted run |
| Braintrust | Dataset supplies cases; a task produces outputs; code scorers return the task's distinct metrics in an experiment | Official evaluation workflow reviewed; no local or hosted run |
The first two rows establish that the same candidate and scoring logic can reproduce a result through two harnesses. They do not establish that Pydantic improves the candidate, that another tool would score identically without configuration work, or that any hosted service was tested.
Run the Pydantic adapter
The standard-library script works without installation. For the optional framework adapter, use Python 3.10 or newer and an isolated environment. The recorded run used Python 3.12.14 and pydantic-evals==2.44.0.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-pydantic.txt
python pydantic_adapter.py > pydantic-observed.json
On Windows, activate the environment with .venv\Scripts\activate instead. Package installation needs internet access. The task itself makes no model or network calls and does not configure Logfire. The Pydantic Evals documentation describes its code-defined datasets, custom evaluators and optional tracing integration.
The adapter builds a case from each input and reference, then runs:
report = dataset.evaluate_sync(
task,
name="conservative-v2",
max_concurrency=1,
progress=False,
)
TicketScore calls the same score() used by the plain Python harness. The output contains ten rows and eight passing results. Inspect c3: its resolved refund message still receives billing, so its reference match fails. Inspect c6: its output envelope records error: "injected_timeout", one attempt and no classifier output, with all four scores false.
The adapter catches that injected task exception and returns an explicit error envelope. Pydantic therefore reports zero framework failures, while the task still fails that case. The adapter's result labels this distinction. It does not test how Pydantic retries an uncaught exception, and it makes no retry-performance claim.
The downloadable JSON export is code written for this lesson. It keeps the per-case outputs and assertion values needed for this comparison. It is not a claim about a standard export format shared by the five tools.
When the extra tool earns its place
Stay with the local script when a small, versioned dataset and an explicit CI exit status meet the team's needs. Its behavior is easy to inspect, but dataset review, experiment browsing and aggregation are your responsibility. It is also useful as a reference scorer when adopting another tool.
Consider Pydantic Evals when the application and evaluators already live in Python and you want reusable cases and structured reports. The executed adapter shows the integration cost for this narrow task. Hosted dataset storage and result transmission are separate choices. Pydantic's code-evaluation guide explains how evaluations remain local without configuring Logfire.
Consider Promptfoo when the next experiment is a matrix of prompts or providers. Its Python provider contract receives the prompt, provider options and context, and returns output or error. Its Python assertions can call custom scoring functions. For this task, keep expected labels out of the candidate prompt even though evaluation context can contain them. Verify the resulting provider-error rows and exports before replacing the existing release gate.
Consider LangSmith when failed task scores need to connect to the application's traces and a team's evaluation workflow. Its evaluation types cover dataset experiments and custom code evaluators. The result-processing guide explicitly distinguishes reading results locally from running without uploading them. “Offline evaluation” means evaluation on a dataset. It does not automatically mean that no data leaves the machine.
Consider Braintrust when dataset curation, experiment comparison and turning reviewed production examples into evaluations are the missing pieces. Its evaluation guide separates mutable playground iteration from experiment records and explains the dataset/task/scorer structure. This lesson does not verify its workspace permissions, hosting configuration, retention or integrations.
Use the failures to test an integration
Before adding a live model, run the same two deliberate scorer defects in the candidate tool: a correct label with a fabricated quote, and a wrong label with an exact quote. A single JSON-validity check will accept both shapes. Your evidence and task checks should distinguish their failures.
Then run the timeout case. Confirm it remains one of ten requested cases and that the export includes its error and attempt count. Compare the case IDs and individual scores before comparing a headline percentage. If one tool retries or omits errors by default, align those settings or explain why the experiment differs.
For the provided Pydantic adapter, compare pydantic-results.json with the conservative-v2 run in expected-results.json. The executed exports match on all four booleans for each of the ten IDs. Repeating that comparison after an adapter change checks whether it preserves the scorer’s behavior. Local duration and framework overhead are outside that equality check.
There is no need for a model judge on this task: labels, fields and quoted substrings are directly observable. If you later score writing quality or open-ended reasoning, calibrate any human or model judgment against reviewed examples and keep disagreements inspectable. Provider calls and hosted features can incur charges. The executed examples here use neither.
Continue with agent design patterns to connect these case results to control flow. The course index provides the surrounding sequence.