# Evaluate a Ticket Agent: Dataset, Scoring Code and Failure Report

> Run ten ticket cases through two deterministic baselines, inspect wrong labels and timeouts, and retain failures in a reproducible evaluation report.

- Source: https://www.zarifautomates.com/blog/evaluate-ai-agent-performance-reliability
- Published: 2026-09-17
- Updated: 2026-09-19
- Pillar: Agents & AI Engineering
- Tags: agent-course, ai-engineering, practical-guide
- Author: Zarif

---

A ticket classifier can return valid JSON, quote the ticket correctly and still choose the wrong action. An evaluation needs to separate those observations. Otherwise a format check can look like evidence that the task works.

This lesson gives you a ten-case dataset, two small baseline classifiers and a scoring script. You will reproduce their results, inspect the remaining failures and run a release gate that deliberately fails. Neither baseline calls an LLM. The exercise tests the evaluation machinery before you spend time or money comparing models.

## Run the dataset locally

You need Python 3.10 or newer and a terminal. Download and extract the [ticket evaluation example](/downloads/agent-course/ticket-evaluation-v1.zip). Keep `evaluate.py` and `cases.json` together and run:

```bash
python3 evaluate.py > observed.json
```

Use `python` if that is the command installed on your system. No dependencies or credentials are required. Do not add `-O`. The script uses assertions to verify the scorer.

The script evaluates all ten cases once with `keyword-v1`, then once with `conservative-v2`. It prints each output, error, score and local duration. It also records the dataset revision, dataset hash, implementation hash, Python version and explicit `model: null`. The download includes an observed run in `expected-results.json`. Timings will differ on another run.

This is proposal evaluation. It does not apply labels, measure approval correctness or establish that an agent can safely operate a ticket system. The [approval lesson](/blog/how-to-build-ai-agent-human-in-loop-approval) tests that separate execution boundary.

## State the labeling policy before scoring

The example has two action labels, `billing` and `technical`, plus `abstain`. Its policy permits a label only when exactly one request category is present. Ambiguous, unrelated, instruction-only and already-resolved messages should abstain.

That is an authored task policy, not a universal customer-support standard. A real team might route mixed requests to a different queue or permit multiple labels. Change the reference answers and scorer together if the actual task differs.

Each case contains an ID, group, input, expected label and rationale. For example:

```json
{
  "id": "c3",
  "group": "challenge",
  "input": {
    "text": "My refund was processed. Thank you, no action needed."
  },
  "expected": "abstain",
  "rationale": "Resolved issue explicitly needs no action."
}
```

Four development cases cover a charge, a crash, an unrelated request and a mixed request. Six challenge cases add refund vocabulary, sign-in trouble, a resolved issue, an instruction-only message, another mixed request and an injected timeout. Five references require abstention: `d3`, `d4`, `c3`, `c4` and `c5`.

Both groups are public and were inspected while building this lesson. The challenge group is not a held-out test set. A new evaluation of your own model should reserve separately reviewed cases that were not used to edit its prompt or rules.

## Keep four checks separate

`score()` returns four booleans:

| Check | What it establishes |
| --- | --- |
| `shape` | Exactly `label` and `evidence`, an allowed label, and a string or null evidence value |
| `evidence` | A nonempty exact substring of the ticket for an action label; null for abstention |
| `label_match` | The output label equals the reviewed reference |
| `passed` | The shape, evidence rule and expected label all pass |

A quote appearing in the ticket does not prove it justifies the label. That is why the reference comparison remains separate. The rule also accepts the full ticket as evidence. It does not score whether a quote is concise or useful to a reviewer.

Four scorer checks run before the dataset. They prove that a wrong-but-well-formed label fails, an invented quote fails, an extra output field fails the shape check, and missing output fails. The first two cases are especially useful: a format-only evaluator would miss both defects.

The task functions receive only `case["input"]`. They do not receive the expected label or rationale. Those remain on the evaluator side. When adapting the harness to a model, preserve that separation so reference answers cannot leak into its prompt.

## Compare the two executed baselines

`keyword-v1` returns `billing` when it sees “charged” and otherwise returns `technical`. `conservative-v2` checks several billing and technical terms, abstaining when both categories or neither category appear. Both use deterministic string matching, and both return a literal quote for action labels.

The executed results were:

| Metric | keyword-v1 | conservative-v2 |
| --- | --- | --- |
| Completed attempts / all cases | 9/10 | 9/10 |
| Correct result / all cases | 3/10 | 8/10 |
| Valid output shape / completed attempts | 9/9 | 9/9 |
| Evidence rule / completed attempts | 9/9 | 9/9 |
| Correct abstention / five abstention cases | 0/5 | 4/5 |
| Development cases passed | 2/4 | 4/4 |
| Challenge cases passed | 1/6 | 4/6 |

The same injected timeout, `c6`, accounts for the uncompleted attempt in both runs.

These ratios describe these ten authored cases. They are not estimates of production accuracy, model capability or a statistically reliable improvement. The useful finding is concrete: both baselines satisfy the format and evidence rules on every completed attempt, but their task results differ substantially.

Open the `conservative-v2` rows and inspect `c3` and `c6`. The resolved refund still triggers `billing`, because the baseline has no resolved-state rule. The other failure is an injected `TimeoutError`. It raises immediately. The example does not wait for or measure a real network timeout. The failed case keeps its ID, reference label, one attempt and error string in the report.

## Do not improve the headline by dropping failures

Removing the timeout would change the conservative baseline from 8/10 to 8/9. That answers a narrower question: correctness among completed attempts. The main result keeps all ten requested cases in its denominator.

The report includes each local execution duration, but these microsecond-scale classifier timings are not model latency. There is one attempt per case, no retry and no tool call. API cost is zero because no API is called. The field excludes machine, hosting and development costs. With a live provider, collect actual usage and all attempts rather than copying these fixture values.

The script normally exits successfully when it produces a valid report, even if the candidate has failures. To enforce the example's deliberately strict release rule, run:

```bash
python3 evaluate.py --require-perfect > gate-results.json
```

It writes the full report and exits with status `1`. The known failures are the reason. Do not weaken the threshold merely to obtain a green run. A real release policy can have different thresholds for task quality, forbidden actions and infrastructure availability, but define them before inspecting the result.

## Change one reference and inspect the consequences

In a disposable copy of `cases.json`, change the expected label of `d1` from `billing` to `technical`. Run again. Both classifiers still return `billing`, but their total passing counts drop by one: 2/10 and 7/10. The dataset hash also changes.

This mutation confirms that the report depends on the reference answers. It does not make the changed answer correct. Restore the original case afterward.

Next, open `evaluate.py` and inspect `scorer_checks()`. Its invented-quote output has the correct label but text absent from the ticket. Run that candidate through `score()` yourself: `shape` and `label_match` should be true, while `evidence` and `passed` are false. Keep failures like this when you add a model judge. Agreement from a judge should not override an observable broken constraint.

[Pydantic Evals](https://pydantic.dev/docs/ai/evals/evals/) provides cases, task execution and custom evaluators if you outgrow this small script. The [evaluation-tools lesson](/blog/best-ai-agent-testing-and-evaluation-tools) compares that path with other tools using this same ticket task. The [course index](/blog/pillar/agents-and-ai-engineering#agent-course-heading) links the full sequence.

## Related Guides

- [Plan and Execute: Build a Bounded Ticket Workflow](/blog/how-to-build-ai-agent-that-plans-and-executes-tasks)
- [Build an MCP Server and Client for a Ticket-Label Tool](/blog/how-to-build-an-ai-agent-using-mcp-model-context-protocol)
- [Use Cursor to Build a Ticket Approval Interface You Can Test](/blog/how-to-use-cursor-ai-to-build-web-applications)

## Continue the course

Lesson 10 of 17.

Previous lesson: [Human Approval for Agents: Store and Check the Exact Decision](https://www.zarifautomates.com/blog/how-to-build-ai-agent-human-in-loop-approval.md).

Next lesson: [Agent Evaluation Tools: Compare Five Options on One Ticket Task](https://www.zarifautomates.com/blog/best-ai-agent-testing-and-evaluation-tools.md).

[Browse available lessons](https://www.zarifautomates.com/blog/pillar/agents-and-ai-engineering#agent-course-heading).