# Agent Evaluation Tools: Compare Five Options on One Ticket Task

> Reproduce a local Pydantic Evals run, then compare Promptfoo, LangSmith, Braintrust and plain Python against the same dataset and scoring rules.

- Source: https://www.zarifautomates.com/blog/best-ai-agent-testing-and-evaluation-tools
- Published: 2026-09-17
- Updated: 2026-09-19
- Pillar: Agents & AI Engineering
- Tags: agent-course, ai-engineering, practical-guide
- Author: Zarif

---

The first question for an evaluation tool is whether it preserves the case that failed. A dashboard showing one average is less useful if you cannot recover the input, output, reference, error and scoring rule behind it.

This comparison uses the same ten-ticket task from the [evaluation lesson](/blog/evaluate-ai-agent-performance-reliability). The plain Python harness and Pydantic Evals adapter were executed locally. Promptfoo, LangSmith and Braintrust are assessed from their official documentation, checked September 17, 2026. They were not run for this comparison. There is no product performance ranking here.

## Fix the task before comparing the tools

The input is a synthetic support message. The candidate must return `billing`, `technical` or `abstain`, with a literal evidence quote for an action label and null for abstention. Scoring checks the output shape, evidence and reference label separately. One of the ten cases raises an injected timeout.

Every candidate tool must preserve the same ten case IDs and four scores, keep the timeout in the denominator, and retain the model or implementation configuration. The classifier receives only the input. Reference labels stay with the evaluator. Those requirements give the comparison a concrete acceptance test.

Download the [shared evaluation artifacts](/downloads/agent-course/ticket-evaluation-v1.zip). They include `cases.json`, the classifier and scorer in `evaluate.py`, the actual Pydantic adapter, pinned requirements and both result files.

## Compare what you would implement

| Option | How this ticket task maps to it | Evidence in this lesson |
| --- | --- | --- |
| Plain Python | Loop over `cases.json`, call `predict()`, then `score()`; write a JSON report and explicit exit status | Executed: 3/10 and 8/10 for two deterministic baselines |
| Pydantic Evals | One `Case` per ticket, a `Dataset`, the task function and a custom `Evaluator` using the same scorer | Executed with 2.44.0: conservative baseline 8/10 |
| Promptfoo | Test variables hold ticket inputs; a custom Python provider calls the candidate; Python assertions apply the scoring rules | Official provider and assertion contracts reviewed; no local run |
| LangSmith | Dataset examples supply inputs and references; an experiment records task runs and custom evaluator results | Official evaluation and result-processing docs reviewed; no local or hosted run |
| Braintrust | Dataset supplies cases; a task produces outputs; code scorers return the task's distinct metrics in an experiment | Official evaluation workflow reviewed; no local or hosted run |

The first two rows establish that the same candidate and scoring logic can reproduce a result through two harnesses. They do not establish that Pydantic improves the candidate, that another tool would score identically without configuration work, or that any hosted service was tested.

## Run the Pydantic adapter

The standard-library script works without installation. For the optional framework adapter, use Python 3.10 or newer and an isolated environment. The recorded run used Python 3.12.14 and `pydantic-evals==2.44.0`.

```bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-pydantic.txt
python pydantic_adapter.py > pydantic-observed.json
```

On Windows, activate the environment with `.venv\Scripts\activate` instead. Package installation needs internet access. The task itself makes no model or network calls and does not configure Logfire. The [Pydantic Evals documentation](https://pydantic.dev/docs/ai/evals/evals/) describes its code-defined datasets, custom evaluators and optional tracing integration.

The adapter builds a case from each input and reference, then runs:

```python
report = dataset.evaluate_sync(
    task,
    name="conservative-v2",
    max_concurrency=1,
    progress=False,
)
```

`TicketScore` calls the same `score()` used by the plain Python harness. The output contains ten rows and eight passing results. Inspect `c3`: its resolved refund message still receives `billing`, so its reference match fails. Inspect `c6`: its output envelope records `error: "injected_timeout"`, one attempt and no classifier output, with all four scores false.

The adapter catches that injected task exception and returns an explicit error envelope. Pydantic therefore reports zero *framework* failures, while the task still fails that case. The adapter's result labels this distinction. It does not test how Pydantic retries an uncaught exception, and it makes no retry-performance claim.

The downloadable JSON export is code written for this lesson. It keeps the per-case outputs and assertion values needed for this comparison. It is not a claim about a standard export format shared by the five tools.

## When the extra tool earns its place

**Stay with the local script** when a small, versioned dataset and an explicit CI exit status meet the team's needs. Its behavior is easy to inspect, but dataset review, experiment browsing and aggregation are your responsibility. It is also useful as a reference scorer when adopting another tool.

**Consider Pydantic Evals** when the application and evaluators already live in Python and you want reusable cases and structured reports. The executed adapter shows the integration cost for this narrow task. Hosted dataset storage and result transmission are separate choices. Pydantic's [code-evaluation guide](https://pydantic.dev/docs/logfire/evaluate/evals-in-code/) explains how evaluations remain local without configuring Logfire.

**Consider Promptfoo** when the next experiment is a matrix of prompts or providers. Its [Python provider](https://www.promptfoo.dev/docs/providers/python/) contract receives the prompt, provider options and context, and returns output or error. Its [Python assertions](https://www.promptfoo.dev/docs/configuration/expected-outputs/python/) can call custom scoring functions. For this task, keep expected labels out of the candidate prompt even though evaluation context can contain them. Verify the resulting provider-error rows and exports before replacing the existing release gate.

**Consider LangSmith** when failed task scores need to connect to the application's traces and a team's evaluation workflow. Its [evaluation types](https://docs.langchain.com/langsmith/evaluation-types) cover dataset experiments and custom code evaluators. The [result-processing guide](https://docs.langchain.com/langsmith/read-local-experiment-results) explicitly distinguishes reading results locally from running without uploading them. “Offline evaluation” means evaluation on a dataset. It does not automatically mean that no data leaves the machine.

**Consider Braintrust** when dataset curation, experiment comparison and turning reviewed production examples into evaluations are the missing pieces. Its [evaluation guide](https://www.braintrust.dev/docs/evaluate) separates mutable playground iteration from experiment records and explains the dataset/task/scorer structure. This lesson does not verify its workspace permissions, hosting configuration, retention or integrations.

## Use the failures to test an integration

Before adding a live model, run the same two deliberate scorer defects in the candidate tool: a correct label with a fabricated quote, and a wrong label with an exact quote. A single JSON-validity check will accept both shapes. Your evidence and task checks should distinguish their failures.

Then run the timeout case. Confirm it remains one of ten requested cases and that the export includes its error and attempt count. Compare the case IDs and individual scores before comparing a headline percentage. If one tool retries or omits errors by default, align those settings or explain why the experiment differs.

For the provided Pydantic adapter, compare `pydantic-results.json` with the `conservative-v2` run in `expected-results.json`. The executed exports match on all four booleans for each of the ten IDs. Repeating that comparison after an adapter change checks whether it preserves the scorer’s behavior. Local duration and framework overhead are outside that equality check.

There is no need for a model judge on this task: labels, fields and quoted substrings are directly observable. If you later score writing quality or open-ended reasoning, calibrate any human or model judgment against reviewed examples and keep disagreements inspectable. Provider calls and hosted features can incur charges. The executed examples here use neither.

Continue with [agent design patterns](/blog/ai-agent-design-patterns-production-systems) to connect these case results to control flow. The [course index](/blog/pillar/agents-and-ai-engineering#agent-course-heading) provides the surrounding sequence.

## Related Guides

- [Choose an Agent Starter: Inspect Three Repositories and Test One](/blog/best-ai-agent-template-libraries-and-starters)
- [Plan and Execute: Build a Bounded Ticket Workflow](/blog/how-to-build-ai-agent-that-plans-and-executes-tasks)
- [Build an MCP Server and Client for a Ticket-Label Tool](/blog/how-to-build-an-ai-agent-using-mcp-model-context-protocol)
- [Best AI Tools Translation: 2026 Localization Guide](/blog/best-ai-tools-for-translation-and-localization)
- [AI APIs for Beginners: Make a Request and Handle the Response](/blog/complete-guide-ai-apis-beginners)

## Continue the course

Lesson 11 of 17.

Previous lesson: [Evaluate a Ticket Agent: Dataset, Scoring Code and Failure Report](https://www.zarifautomates.com/blog/evaluate-ai-agent-performance-reliability.md).

Next lesson: [Agent Design Patterns: Run Four Control Flows and Their Failures](https://www.zarifautomates.com/blog/ai-agent-design-patterns-production-systems.md).

[Browse available lessons](https://www.zarifautomates.com/blog/pillar/agents-and-ai-engineering#agent-course-heading).