Evaluate a Ticket Agent: Dataset, Scoring Code and Failure Report

A support-ticket classifier can return perfectly formatted JSON, quote the ticket word for word, and still pick the wrong action. If your test only checks the format, that wrong answer passes. This lesson builds a test that catches it.

You'll get a ten-case dataset, two small classifiers to test, and a scoring script. You'll run them, look at the cases they get wrong, and run a release check that is built to fail. Neither classifier calls an LLM. The point is to make sure your testing setup works before you spend time or money comparing models.

Run the dataset locally

You need Python 3.10 or newer and a terminal. Download and extract the ticket evaluation example. Keep evaluate.py and cases.json in the same folder and run:

python3 evaluate.py > observed.json

If your system uses python instead of python3, use that. You don't need to install anything or add any keys. Don't add the -O flag. The script uses assertions to check its own scorer, and -O switches assertions off.

The script runs all ten cases through keyword-v1, then through conservative-v2. For each case it prints the output, any error, the score and how long it took on your machine.

It also records exactly what produced the result: the dataset revision, a dataset hash, an implementation hash, the Python version, and model: null to show no model was used. A hash is a short fingerprint of a file. Change the file and the fingerprint changes.

The download includes one observed run in expected-results.json. Your timings will differ.

This lesson only tests the label a classifier proposes. It doesn't apply labels to real tickets, check a human approval step, or show that an agent can safely run a ticket system. The approval lesson covers that part.

Write down the labeling rule before scoring

The example has two action labels, billing and technical, plus abstain, which means "I won't label this one." The rule is to label a ticket only when it asks about exactly one category. Messages that are unclear, unrelated, only give instructions, or are already resolved should get abstain.

This rule was written for this lesson. It isn't an industry standard. A real support team might send mixed requests to their own queue or allow two labels. If your task works differently, change the reference answers and the scorer together.

Each case has an ID, a group, the input, the expected label and the reason for that label. For example:

{
  "id": "c3",
  "group": "challenge",
  "input": {
    "text": "My refund was processed. Thank you, no action needed."
  },
  "expected": "abstain",
  "rationale": "Resolved issue explicitly needs no action."
}

Four development cases cover a charge, a crash, an unrelated request and a mixed request. Six challenge cases add refund wording, sign-in trouble, a resolved issue, a message that only gives instructions, another mixed request, and a timeout injected on purpose. Five cases expect abstain: d3, d4, c3, c4 and c5.

Both groups are public, and both were looked at while this lesson was built. So the challenge group isn't a true held-out test set, the kind nobody sees while tuning. When you evaluate your own model, set aside a separate batch of reviewed cases that you never used to edit its prompt or rules.

Keep four checks separate

score() returns four true-or-false values:

A quote that appears in the ticket doesn't prove the label is right. That's why comparing against the reference answer is its own check. The evidence rule also accepts the whole ticket as a quote. It doesn't judge whether a quote is short or useful to a reviewer.

Before it touches the dataset, the script runs four checks on the scorer itself. They confirm that a wrong label in the right format fails, an invented quote fails, an extra output field fails the shape check, and a missing output fails. The first two matter most. A test that only checks format would miss both.

The classifier functions only get case["input"]. They never see the expected label or the reason. Those stay on the scoring side. When you swap in a real model, keep it that way, so the answers can't leak into its prompt.

Compare the two classifiers

keyword-v1 returns billing when it sees "charged" and technical for everything else. conservative-v2 looks for several billing words and several technical words. If it finds both kinds or neither, it abstains.

Both only match strings, so they give the same answer every time you run them. Both return an exact quote when they pick an action label.

The results were:

Each run has one unfinished attempt, and it's the same case: c6, the injected timeout.

These numbers describe these ten hand-written cases and nothing more. They don't predict production accuracy or say anything about model ability, and the gap between them isn't statistically meaningful. The useful finding is simpler. Both classifiers pass the format and evidence rules on every completed attempt, yet their actual results are far apart.

Open the conservative-v2 rows and look at c3 and c6. The resolved refund still gets billing, because the classifier has no rule for "already resolved."

The other failure is the injected TimeoutError. It's raised right away. The example doesn't wait for or measure a real network timeout. The report still keeps the failed case's ID, its reference label, the one attempt and the error message.

Do not improve the headline by dropping failures

Say you drop the timeout case. The conservative classifier goes from 8/10 to 8/9. That looks better, but it answers a smaller question: how often was it right when it finished? The main result keeps all ten cases in the count.

The report includes how long each case took. Those are microsecond timings for string matching on your machine, not how long a model takes to answer. There's one attempt per case, no retry and no tool call.

API cost is zero because nothing calls an API. That cost field also leaves out machine, hosting and development costs. With a real provider, record actual usage and every attempt instead of copying these numbers.

The script normally exits successfully whenever it produces a valid report, even if the classifier failed cases. To apply this example's strict release rule, where every case must pass, run:

python3 evaluate.py --require-perfect > gate-results.json

It writes the full report and exits with status 1. The known failures are why. Don't lower the bar just to get a passing run. A real release rule can set different bars for answer quality, forbidden actions and uptime, but decide them before you look at the results.

Change one reference and inspect the consequences

In a throwaway copy of cases.json, change the expected label of d1 from billing to technical. Run it again. Both classifiers still return billing, but each passing count drops by one: 2/10 and 7/10. The dataset hash changes too.

This shows the report depends on your reference answers. It doesn't make the changed answer right. Put the original back when you're done.

Next, open evaluate.py and look at scorer_checks(). Its invented-quote output has the right label but a quote that isn't in the ticket. Run it through score() yourself. shape and label_match should be true, and evidence and passed should be false.

Keep checks like this when you add a model judge, a second model that grades the first one's answers. If a check you can see is broken, the judge agreeing shouldn't override it.

If you outgrow this small script, Pydantic Evals gives you cases, task runs and custom scoring. The evaluation-tools lesson compares that option with other tools on this same ticket task. The course index links the full sequence.

Related Guides

• Plan and Execute: Build a Bounded Ticket Workflow
• Build an MCP Server and Client for a Ticket-Label Tool
• Use Cursor to Build a Ticket Approval Interface You Can Test