Evaluate a Ticket Agent: Dataset, Scoring Code and Failure Report

A ticket classifier can return valid JSON, quote the ticket correctly and still choose the wrong action. An evaluation needs to separate those observations. Otherwise a format check can look like evidence that the task works.

This lesson gives you a ten-case dataset, two small baseline classifiers and a scoring script. You will reproduce their results, inspect the remaining failures and run a release gate that deliberately fails. Neither baseline calls an LLM. The exercise tests the evaluation machinery before you spend time or money comparing models.

Run the dataset locally

You need Python 3.10 or newer and a terminal. Download and extract the ticket evaluation example. Keep evaluate.py and cases.json together and run:

python3 evaluate.py > observed.json

Use python if that is the command installed on your system. No dependencies or credentials are required. Do not add -O. The script uses assertions to verify the scorer.

The script evaluates all ten cases once with keyword-v1, then once with conservative-v2. It prints each output, error, score and local duration. It also records the dataset revision, dataset hash, implementation hash, Python version and explicit model: null. The download includes an observed run in expected-results.json. Timings will differ on another run.

This is proposal evaluation. It does not apply labels, measure approval correctness or establish that an agent can safely operate a ticket system. The approval lesson tests that separate execution boundary.

State the labeling policy before scoring

The example has two action labels, billing and technical, plus abstain. Its policy permits a label only when exactly one request category is present. Ambiguous, unrelated, instruction-only and already-resolved messages should abstain.

That is an authored task policy, not a universal customer-support standard. A real team might route mixed requests to a different queue or permit multiple labels. Change the reference answers and scorer together if the actual task differs.

Each case contains an ID, group, input, expected label and rationale. For example:

{
  "id": "c3",
  "group": "challenge",
  "input": {
    "text": "My refund was processed. Thank you, no action needed."
  },
  "expected": "abstain",
  "rationale": "Resolved issue explicitly needs no action."
}

Four development cases cover a charge, a crash, an unrelated request and a mixed request. Six challenge cases add refund vocabulary, sign-in trouble, a resolved issue, an instruction-only message, another mixed request and an injected timeout. Five references require abstention: d3, d4, c3, c4 and c5.

Both groups are public and were inspected while building this lesson. The challenge group is not a held-out test set. A new evaluation of your own model should reserve separately reviewed cases that were not used to edit its prompt or rules.

Keep four checks separate

score() returns four booleans:

A quote appearing in the ticket does not prove it justifies the label. That is why the reference comparison remains separate. The rule also accepts the full ticket as evidence. It does not score whether a quote is concise or useful to a reviewer.

Four scorer checks run before the dataset. They prove that a wrong-but-well-formed label fails, an invented quote fails, an extra output field fails the shape check, and missing output fails. The first two cases are especially useful: a format-only evaluator would miss both defects.

The task functions receive only case["input"]. They do not receive the expected label or rationale. Those remain on the evaluator side. When adapting the harness to a model, preserve that separation so reference answers cannot leak into its prompt.

Compare the two executed baselines

keyword-v1 returns billing when it sees “charged” and otherwise returns technical. conservative-v2 checks several billing and technical terms, abstaining when both categories or neither category appear. Both use deterministic string matching, and both return a literal quote for action labels.

The executed results were:

The same injected timeout, c6, accounts for the uncompleted attempt in both runs.

These ratios describe these ten authored cases. They are not estimates of production accuracy, model capability or a statistically reliable improvement. The useful finding is concrete: both baselines satisfy the format and evidence rules on every completed attempt, but their task results differ substantially.

Open the conservative-v2 rows and inspect c3 and c6. The resolved refund still triggers billing, because the baseline has no resolved-state rule. The other failure is an injected TimeoutError. It raises immediately. The example does not wait for or measure a real network timeout. The failed case keeps its ID, reference label, one attempt and error string in the report.

Do not improve the headline by dropping failures

Removing the timeout would change the conservative baseline from 8/10 to 8/9. That answers a narrower question: correctness among completed attempts. The main result keeps all ten requested cases in its denominator.

The report includes each local execution duration, but these microsecond-scale classifier timings are not model latency. There is one attempt per case, no retry and no tool call. API cost is zero because no API is called. The field excludes machine, hosting and development costs. With a live provider, collect actual usage and all attempts rather than copying these fixture values.

The script normally exits successfully when it produces a valid report, even if the candidate has failures. To enforce the example's deliberately strict release rule, run:

python3 evaluate.py --require-perfect > gate-results.json

It writes the full report and exits with status 1. The known failures are the reason. Do not weaken the threshold merely to obtain a green run. A real release policy can have different thresholds for task quality, forbidden actions and infrastructure availability, but define them before inspecting the result.

Change one reference and inspect the consequences

In a disposable copy of cases.json, change the expected label of d1 from billing to technical. Run again. Both classifiers still return billing, but their total passing counts drop by one: 2/10 and 7/10. The dataset hash also changes.

This mutation confirms that the report depends on the reference answers. It does not make the changed answer correct. Restore the original case afterward.

Next, open evaluate.py and inspect scorer_checks(). Its invented-quote output has the correct label but text absent from the ticket. Run that candidate through score() yourself: shape and label_match should be true, while evidence and passed are false. Keep failures like this when you add a model judge. Agreement from a judge should not override an observable broken constraint.

Pydantic Evals provides cases, task execution and custom evaluators if you outgrow this small script. The evaluation-tools lesson compares that path with other tools using this same ticket task. The course index links the full sequence.

Related Guides

• Plan and Execute: Build a Bounded Ticket Workflow
• Build an MCP Server and Client for a Ticket-Label Tool
• Use Cursor to Build a Ticket Approval Interface You Can Test