How to Set Up AI-Powered Customer Support Triage

AI triage targets a narrow support problem: deciding what a ticket is about, how urgent it is, and who should handle it. It does not need to answer the customer to create value. Here is how to build a triage layer, measure its routing quality, and estimate cost from current model prices and your own ticket lengths.

Why triage is the smart first AI win in support

Most support teams jump straight to "let AI answer the tickets." That is a mistake for two reasons. Answer quality is hard to evaluate at scale, and a wrong answer to a customer is costly. Triage has the opposite profile. Mistakes are cheap (a ticket goes to the wrong queue), accuracy is easy to measure (did the human reroute it?), and the time savings are immediate.

If triage works well, the support team spends less time sorting the inbox and more time handling customer problems. Measure that change against the review and maintenance burden before claiming ROI.

What "good triage" actually means

A complete triage system makes four decisions on every ticket:

1. Category — billing, technical, account, sales, abuse, spam
2. Urgency — P1 (down), P2 (degraded), P3 (question), P4 (feature request)
3. Product area — which product line or component
4. Sentiment — neutral, frustrated, angry, churn-risk

You also want a confidence score on each decision and a fallback to "needs human review" if any score is below threshold.

The architecture

Five stages:

1. Trigger — webhook from Zendesk, Intercom, HubSpot, Help Scout, or Freshdesk
2. Context fetch — pull the customer's plan, tenure, and ticket history
3. Classifier — single LLM call returning structured JSON
4. Router — applies business rules to the classification
5. Action — assign queue, set priority, add tags, optionally Slack-ping the on-call

The whole thing should complete in under 5 seconds from ticket arrival to routed.

Step 1: Define your label taxonomy before you touch the API

This is the step everyone skips and regrets. If your labels are vague, your accuracy will be vague. Write them down explicitly:

• Each category gets a 1-sentence definition
• Each category gets 3 example tickets
• Mutually exclusive — a ticket fits one category, not two
• Include a "needs human" category as the fallback

I keep a labels.yaml file in the repo. The system prompt references it directly. When the taxonomy changes, the prompt changes in one place.

Step 2: Pull customer context, not just ticket text

A ticket with no context is a coin flip. The same words "this is broken" mean P1 from an enterprise customer and P3 from a free trial. Pull:

• Account plan and MRR
• Tenure (days since signup)
• Open ticket count
• Last 3 ticket categories
• NPS score if you have one

Pass that as a structured block in the prompt, then test whether it improves routing enough to justify the added data access and privacy surface.

Step 3: Build the classifier prompt

Use OpenAI Responses API with response_format set to a JSON schema, or Anthropic's tool-use API with a structured tool. Either works.

My production prompt outline:

You are a customer support triage agent. Classify the ticket below.

Return JSON with these fields:
- category: one of [billing, technical, account, sales, abuse, spam, unknown]
- urgency: one of [P1, P2, P3, P4]
- product_area: one of [api, dashboard, billing, mobile, integrations, other]
- sentiment: one of [neutral, frustrated, angry, churn_risk]
- confidence: a number from 0 to 1
- reasoning: one sentence explaining your decision

Rules:
- If the customer mentions cancellation, the sentiment is churn_risk.
- If a paying customer mentions production is down, urgency is P1.
- If you are not 80 percent sure, return category "unknown".

The "unknown" escape valve is critical. Forcing a model to choose a category when it cannot is how you get garbage routing.

Step 4: Add business rules on top of the classification

The LLM gives you the raw classification. Business rules turn that into routing decisions. Example rules:

• If urgency == P1, page the on-call engineer in PagerDuty
• If sentiment == churn_risk and mrr greater than 1000, assign to the customer success manager directly
• If category == billing and tenure less than 30, assign to the onboarding queue
• If confidence less than 0.8, route to "needs review" queue

Keep the rules in a YAML file or a Postgres table, not in code. Support managers should be able to edit them without a deploy.

Step 5: Wire to your helpdesk

Zendesk, Intercom, HubSpot, and Help Scout all have webhooks for new ticket events and APIs to update tags, priority, and assignee. The integration:

1. Helpdesk fires webhook on ticket.created
2. Your service receives it, runs the classifier
3. PATCH the ticket with new tags, priority, and assignee_id

For Zendesk, the endpoint is PUT /api/v2/tickets/{id}.json with a body containing {"ticket": {"priority": "high", "assignee_id": 123, "tags": ["..."]}}.

For Intercom, it is PUT /conversations/{id} with similar fields.

Step 6: Pick your model and budget

The model choice for triage is straightforward: start with a low-cost model, then promote only the categories that miss your eval target. Current published rates are:

• GPT-5.6 Luna is $0.20 per 1M short-context input tokens and $1.20 per 1M output tokens; long-context requests cost more.
• Claude Haiku 4.5 is $1 per 1M input tokens and $5 per 1M output tokens, with cache reads priced separately.
• Gemini 2.5 Flash is $0.30 per 1M text/image/video input tokens and $2.50 per 1M output tokens.
• Gemini 2.5 Flash-Lite is $0.10 per 1M text/image/video input tokens and $0.40 per 1M output tokens.

For a team handling 1,000 tickets a day, estimate monthly API cost from sampled input and output tokens, cache behavior, retries, and the share sent to a larger fallback model. Hosting and helpdesk charges are separate.

If you would rather buy a managed AI agent than build, compare the product's billing unit with your use case. Intercom prices Fin chat and email outcomes from $0.99. Zendesk now includes AI agents in Suite and Support plans and bills successful outcomes through resolution tiers and allowances, so the older flat Advanced AI add-on math is no longer a reliable quote.

Step 7: Build the human-review feedback loop

Every misclassification is data. Every reroute by a human agent should feed back into your eval set. Implement:

1. When a human changes the assignee or priority, log the original prediction
2. Weekly, dump the last 7 days of corrections to a CSV
3. Review the top 10 misclassifications and decide if they reflect a prompt fix, a taxonomy gap, or just an edge case
4. Update the prompt or labels accordingly

Without this loop, your accuracy degrades silently as new ticket types emerge. With it, your system gets sharper every week.

Step 8: Ship in shadow mode first

Do not let the AI take routing actions on day one. Run in shadow mode for at least a week:

1. Classifier runs on every new ticket
2. Result is logged to a database, not applied to the ticket
3. Compare classifier output to the human's actual routing decision
4. Measure agreement rate per category

When agreement crosses 90 percent, flip the auto-route switch. Keep shadow logging on permanently for monitoring.

What this costs in production

For a team handling 1,000 tickets per day:

• LLM API: calculate from sampled tokens using the current model rates above; 30,000 tickets can vary materially with thread length, context fetches, retries, and fallback routing
• Orchestration: self-hosted infrastructure varies; n8n Cloud Starter is currently €20/month billed annually for 2,500 executions, which is below a 30,000-ticket monthly workload before extra workflow executions are considered
• Helpdesk API calls: free within plan limits
• Engineering time: estimate from the actual helpdesk integration, security review, eval harness, monitoring, and support requirements

Do not compare a triage-only build directly with a managed agent's resolution charge: they perform different jobs. Build the business case from measured triage time, reroute rate, review workload, infrastructure, and maintenance, then compare that total with the vendor quote for the same scope.

Common failure modes

The model misclassifies abuse as sales when the customer is polite while threatening legal action. Mitigation: add explicit examples of polite-but-hostile to the prompt.

Tickets with attachments get sent without OCR'd context. Mitigation: pre-process attachments through GPT-5 vision input, Claude Sonnet 4.6 vision, or AWS Textract before classifying.

Tickets in non-English languages drop accuracy. Mitigation: detect language first with a simple library and route non-English tickets straight to bilingual reviewers.

Long ticket threads exceed context. Mitigation: only classify on the first message, or summarize before classifying.

FAQ

Related Guides

• AI SOP Template: Customer Support Handling
• How to Build an AI Agent That Handles Customer Support
• What Is AI Model Temperature and How to Set It

The team that wins at AI in support is not the one that automates the answer. It is the one that automates the routing so humans only see tickets that need them. Build the triage layer first.