- Step 11 of 17 in the agent-building course
Agent Evaluation Tools: Compare Five Options on One Ticket Task
Zarif
Say your agent passes eight of ten test cases. The first thing you'll want to know is which cases it missed, and why.
So the first thing to check in any evaluation tool is whether it keeps the failed case. A dashboard with one average isn't much help if you can't get back the input, the output, the right answer, the error, and the rule that scored it.
This lesson compares five options on the same ten-ticket task from the evaluation lesson. Two of them were actually run on a local machine: a plain Python script and a Pydantic Evals adapter.
The other three, Promptfoo, LangSmith and Braintrust, were judged from their official docs, checked on September 17, 2026. They weren't run for this comparison. There's no ranking of which tool performs best.
Fix the task before comparing the tools
Every tool gets the exact same job, so the comparison is fair.
The input is a fake support message. The program being tested has to return one of three labels: billing, technical or abstain. If it picks billing or technical, it has to quote the exact words from the message that justify the label. If it abstains, the quote is null.
The scorer checks three things separately: the shape of the output, the quote, and whether the label matches the right answer. One of the ten cases is rigged to time out.
To pass this comparison, a tool has to:
- keep the same ten case IDs and all four scores for each,
- count the timeout as one of the ten, not drop it, and
- record which model or code version produced the results.
The program being tested only sees the input. The right answers stay with the scorer.
Download the shared evaluation artifacts. Inside you'll find cases.json, the classifier and scorer in evaluate.py, the Pydantic adapter, pinned requirements, and both result files.
Compare what you would implement
| Option | How this ticket task maps to it | Evidence in this lesson |
|---|---|---|
| Plain Python | Loop over cases.json, call predict(), then score(); write a JSON report and explicit exit status | Executed: 3/10 and 8/10 for two deterministic baselines |
| Pydantic Evals | One Case per ticket, a Dataset, the task function and a custom Evaluator using the same scorer | Executed with 2.44.0: conservative baseline 8/10 |
| Promptfoo | Test variables hold ticket inputs; a custom Python provider calls the candidate; Python assertions apply the scoring rules | Official provider and assertion contracts reviewed; no local run |
| LangSmith | Dataset examples supply inputs and references; an experiment records task runs and custom evaluator results | Official evaluation and result-processing docs reviewed; no local or hosted run |
| Braintrust | Dataset supplies cases; a task produces outputs; code scorers return the task's distinct metrics in an experiment | Official evaluation workflow reviewed; no local or hosted run |
The first two rows show one thing: the same program and the same scoring rules give the same result through two different setups. They don't show that Pydantic makes the program better. They don't show that another tool would score it the same without setup work. And no hosted service was tested.
Run the Pydantic adapter
The plain Python script runs with nothing to install. For the Pydantic adapter, you need Python 3.10 or newer and a separate environment so the packages don't mix with anything else. The recorded run used Python 3.12.14 and pydantic-evals==2.44.0.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-pydantic.txt
python pydantic_adapter.py > pydantic-observed.json
On Windows, activate the environment with .venv\Scripts\activate instead. Installing packages needs internet access. The task itself calls no model and no network, and doesn't set up Logfire, Pydantic's tracing service. The Pydantic Evals documentation explains its datasets defined in code, custom scorers, and optional tracing.
The adapter turns each input and right answer into a case, then runs:
report = dataset.evaluate_sync(
task,
name="conservative-v2",
max_concurrency=1,
progress=False,
)
TicketScore calls the same score() function the plain Python script uses. You should get ten rows, with eight passing. Look at two of the failures.
c3 is a refund message that's already been resolved. The program still labels it billing, so it doesn't match the right answer.
c6 is the rigged timeout. Its output records error: "injected_timeout", one attempt, and no answer from the classifier. All four of its scores are false.
There's a subtle point in how that timeout gets reported. The adapter catches the error itself and hands back a clear error record. So Pydantic reports zero failures of its own, while the task still fails that case. The adapter's output labels the two separately.
This doesn't test how Pydantic retries an error nobody caught, and it says nothing about retry speed.
The JSON export you can download was written for this lesson. It keeps each case's output and score values so the comparison works. It's not a standard format the five tools share.
When the extra tool earns its place
Stay with the local script when a small dataset in version control and a clear pass-or-fail signal in CI are all your team needs. It's easy to see what it does. The tradeoff is that reviewing datasets, browsing past runs, and rolling up results are all on you. It's also handy as a reference scorer while you adopt another tool.
Consider Pydantic Evals when your app and your scorers are already in Python, and you want reusable cases and structured reports. The adapter here shows how much work it took to hook up for this one narrow task. Storing datasets online and sending results anywhere are separate choices. Pydantic's code-evaluation guide explains how evaluations stay on your machine if you don't set up Logfire.
Consider Promptfoo when your next experiment is a grid: several prompts, several providers, every combination. Its Python provider gets the prompt, provider options and context, and returns an output or an error. Its Python assertions can call your own scoring functions.
One caution for this task. Promptfoo's evaluation context can hold the right answers, so make sure they never reach the prompt the program sees. And check how provider errors and exports look before you swap it in for your existing release check.
Consider LangSmith when you need a failed score to link back to the app's traces, and a team works through evaluations together. Its evaluation types cover dataset experiments and custom code scorers.
Watch one term. The result-processing guide separates reading results on your machine from running without uploading them. "Offline evaluation" means testing against a dataset. It doesn't mean no data leaves your machine.
Consider Braintrust when what you're missing is curating datasets, comparing experiments side by side, and turning reviewed real-world examples into new test cases. Its evaluation guide keeps quick playground tries separate from saved experiment records, and explains how datasets, tasks and scorers fit together. This lesson didn't check its workspace permissions, hosting setup, data retention or integrations.
Use the failures to test an integration
Before you hook up a live model, feed the new tool two answers that are wrong on purpose:
- the right label with a made-up quote, and
- the wrong label with a real quote.
A simple "is this valid JSON" check would pass both. Your quote check and your label check should each catch a different one.
Then run the timeout case. Make sure it still counts as one of the ten, and that the export shows its error and attempt count.
Compare case IDs and individual scores before you compare the headline percentage. If one tool retries or hides errors by default, change the settings to match, or note why the runs differ.
For the Pydantic adapter, compare pydantic-results.json with the conservative-v2 run in expected-results.json. All four scores match for all ten case IDs. Rerun that comparison any time you change the adapter. It tells you whether the scoring still behaves the same. It doesn't cover run time or how much overhead the framework adds.
This task doesn't need a model to grade answers. Labels, fields and quotes can all be checked directly.
If you later score something fuzzier, like writing quality or open-ended reasoning, check the grader against examples a person has already reviewed, and keep the disagreements where you can see them. Provider calls and hosted features can cost money. The examples here use neither.
Next, read agent design patterns to connect these case results to how an agent decides its next step. The course index has the full sequence.
