Auto-triage & route inbound support tickets
The AI sorts, prioritizes, routes, and auto-closes the obvious-junk tickets on its own — while you skim a daily audit log, spot-check a sample, and can reopen or re-route anything in seconds.
Where this task lands on the loop spectrum
The two axes — verifiability × reachability
We judge with two independent questions: Verifiability (can you cheaply check the output? → x-axis) and Reachability (can AI even generate it at all? → y-axis). Top-right is the automation zone. Below, this task's four stages are plotted — they sit high (sorting text is squarely within reach) but at mid verifiability (routing is only checkable after reading), so they land just left of "full automation".
Plot each stage by its two-axis position; classify the whole task by its highest-stakes stage. Here every stage is reachable, but read-to-check verifiability keeps it one notch back from full automation — HUMAN-ON-THE-LOOP. chollet-measure
What this means, in plain language
Let the AI sort the mail — but keep reading the daily log.
"On the loop" means the AI does the work and acts on its own (it tags, routes, and closes junk tickets), while you watch over it like a pilot watching autopilot. Each morning you skim an audit log, you can reopen or re-route anything with one click, and you stay the person in command. It's a step more cautious than full automation because a wrongly-closed ticket could hide a real problem — and a step more relaxed than approving every single routing by hand.
The task, as I understood it
Tickets arrive all day from several channels (email, web form, in-app). Today a person reads each one, decides the topic and urgency, tags it, drops it into the right queue, and deletes the spam and duplicates. You want AI to do that sorting automatically and at volume — read every ticket, classify topic and priority, apply tags, route to the right team, and auto-close the obvious junk — so your team only works real, pre-sorted tickets. A misroute is annoying but cheap to fix. Importantly, the AI does not reply to customers; your people still write the answers.
If any of this is wrong, tell me and I'll re-run the assessment — the level can shift.
Why this level — factor by factor
Each row is one lever. The "pushed toward" chip shows whether your answer argued for more automation, more human involvement, or set a hard floor. Every lever links to the source it comes from.
| Factor | Your answer | Pushed toward | Source |
|---|---|---|---|
| Verifiability Can you check it's right cheaply, no expert needed? |
"Yes, but I have to read it." Routing is checkable, yet categories aren't a single machine-scored right/wrong (V2). | Needs audit, not hands-off | karpathy-verifiability, mindstudio-map |
| Consumption Does a human read the output before it matters? |
Mostly auto-consumed — tickets flow into queues without a human reading each — but you read a daily audit log. | Auto-flow, with a human audit | mindstudio-map |
| Reversibility How hard to undo a mistake? |
Easy — a misroute or wrong close is reopened/re-routed in seconds. The AI acts, then you can veto. | Allows act-then-notify autonomy | sheridan-verplank, parasuraman-2000 |
| Stakes / risk How bad if it slips through? |
Low-to-moderate: usually a slow first reply, not financial or legal harm — provided urgent items escalate. | Allows more autonomy | anthropic-autonomy, openai-agents-guide |
| Volume / frequency High-volume and repetitive, or one-off? |
Many tickets a day, continuously — clearly worth automating. | Justifies building automation | zwingmann-aug |
| Specifiability Could you write the steps as a flowchart? |
Mostly: read → classify topic & priority → tag → route → close if spam/dupe. A repeatable recipe. | Build as a fixed workflow | anthropic-agents |
| Reachability — can AI even generate this? Does it need a brand-new idea AI has not seen? |
No — it classifies text that already exists (R1). Nothing out-of-distribution is required. | Reachable — allows automation | chollet-measure |
| People-impact (regulatory floor) Does it decide about a specific person? |
No (as scoped) — it sorts content, it doesn't judge a person. Would change if it denied refunds or flagged users for bans. | No legal floor triggered (yet) | gdpr-22, eu-ai-act-14 |
| Supervisability Can a human monitor and step in within time? |
Yes — tickets are async and reviewable in bulk, so you have time to audit and intervene. | On-the-loop is realistic (not the Level-3 trap) | sae-j3016 |
| Motivation / ownership Do you want to own this part? |
No — triage is a chore you'd happily delegate. | Safe to delegate | lubars-tan |
Net: high volume, reversible, supervisable, specifiable, no people-impact and reachable (R1) would allow full automation — but because routing is only checkable after reading (V2) and a wrong auto-close can bury a real issue, we hold one notch back at HUMAN-ON-THE-LOOP: let it run, but keep eyes on the log. mindstudio-map, anthropic-autonomy
Stage breakdown — where the task can split
Most tasks are really four smaller steps, and they don't all need the same level of oversight. We classify the whole task by its highest-stakes stage, but you can automate the easy stages more aggressively. parasuraman-2000
Ingesting tickets from each channel is mechanical and machine-checkable.
Topic, priority, tags — AI labels on its own; you audit a sample.
Which queue? Spam/dupe? Low-confidence cases divert to a human queue.
Move to the queue, auto-close junk — acts then notifies, all reversible.
Stage-splitting tip: to start more cautiously, keep auto-close at HUMAN-IN-THE-LOOP — the AI flags "spam/duplicate", you confirm the close — while routing already runs on-the-loop (it's so easily reversed). And legal / safety / security / VIP tickets should never auto-close: always escalate to a human.
Recommended setup — what to build
Tooling level: a predefined workflow (a fixed classify-and-route pipeline), not a free-roaming agent. Your steps are stable and writable as a flowchart, so a simple, repeatable workflow is the most reliable and the easiest to trust. Save the full autonomous-agent approach for tasks whose steps change each time. anthropic-agents
Build this first (the minimum viable version):
- One intake that collects all incoming tickets into a single stream.
- One classifier prompt returning topic, priority, tags, suggested queue, a spam/duplicate flag, and a confidence score.
- A routing rule: high-confidence → auto-route; low-confidence → a "needs-human" queue.
- A daily audit digest dropped in front of you each morning — so for the first weeks you still confirm closes.
Guardrails & checkpoints
- Confidence threshold: below a set confidence, the ticket diverts to a human queue instead of being guessed.
- Hard-escalation rules: legal, safety, security, payment-dispute, or VIP tickets are never auto-closed and route to a human immediately.
- Daily audit log: every auto-action is logged with one-click reopen / re-route; you skim it each morning.
- Sample audit: review a small random percentage weekly to measure accuracy even when nothing looks wrong.
- No auto-reply to customers: the AI sorts; it does not answer. A customer-facing reply would raise this to HUMAN-IN-THE-LOOP.
- Stop switch: one obvious way to pause the whole pipeline — you stay in command. eu-ai-act-14
Start-small graduation plan
Autonomy is earned, not assumed. Start more cautious, then relax oversight as the AI proves itself reliable. hitl-hotl
Weeks 1–2 · Flag-only HUMAN-IN-THE-LOOP
AI suggests topic/priority/queue and a spam flag; a human confirms every routing and close. You're calibrating trust and tuning the prompt.
Weeks 3–8 · Auto-route, audit daily HUMAN-ON-THE-LOOP
Once labels are reliably right, let it route automatically with a daily audit and reopen button; keep closes manual. This is your recommended steady state.
Month 3+ · Auto-close the obvious junk HUMAN-ON-THE-LOOP
Add auto-close for high-confidence spam/duplicates; widen the audit interval as accuracy holds — but keep the escalation rules, the human queue, and the stop switch forever.
Because a wrong close can bury a real ticket, we'd not push this all the way to FULL AUTOMATION. On-the-loop, with escalation rules, is the right ceiling here. anthropic-autonomy
Example starter prompts
You are a support-ticket triage assistant. Read the ticket below and return JSON only:
{
"topic": one of [billing, bug, how-to, account, feature-request, abuse, other],
"priority": one of [urgent, normal, low],
"tags": [short keywords],
"suggested_queue": which team should handle it,
"spam_or_duplicate": true/false,
"confidence": 0.0-1.0,
"escalate": true/false // true if legal, safety, security, payment dispute, or VIP
}
Rules: if confidence < 0.8, set suggested_queue to "needs-human".
Never mark spam_or_duplicate when escalate is true. Use only what the ticket says.
TICKET:
[paste the ticket here]
Below is everything the triage workflow did automatically in the last 24 hours. Produce a short audit digest: 1. Counts by topic, priority, and queue. 2. Every ticket it AUTO-CLOSED, one line each, so I can scan for mistakes. 3. Low-confidence (<0.8) tickets sent to "needs-human". 4. Anything that looks like a possible misroute or a buried urgent issue. Keep it skimmable. ACTIONS LOG: [paste the day's triage actions here]
Here are 20 randomly sampled tickets and the queue/priority the workflow assigned each. For each, say AGREE or DISAGREE with the routing and priority; if you disagree, give the correct one in a few words. End with an accuracy percentage. Be strict. SAMPLE: [paste sampled tickets + their assigned routing here]
Risks to watch — and when to pull back
A buried urgent ticket
The costliest miss is an urgent/safety issue auto-closed or mis-prioritized. The escalation rules and the daily auto-close review are your net — test them monthly with a planted example. openai-agents-guide
Over-aggressive auto-close
Real tickets closed as "duplicate." Tighten the confidence threshold and keep closes manual longer; reopening is cheap only if you catch it in the audit. karpathy-verifiability
Quiet drift
As your product or customer mix changes, old categories underperform. Re-run a spot-check (Prompt 3) monthly and re-tune. parasuraman-2000
Scope creep into people-decisions
If it starts auto-replying, approving refunds, or flagging users for bans, the people-impact floor kicks in — move that part to HUMAN-IN-THE-LOOP. gdpr-22
The evidence behind this advice
Open any card to see what the source says and follow the link to read it yourself.