AI Automation Level Advisor · Report

Auto-triage & route inbound support tickets

Date: 2026-06-06
Verdict: HUMAN-ON-THE-LOOP (Supervise and intervene)

The AI sorts, prioritizes, routes, and auto-closes the obvious-junk tickets on its own — while you skim a daily audit log, spot-check a sample, and can reopen or re-route anything in seconds.

1

Where this task lands on the loop spectrum

2

The two axes — verifiability × reachability

We judge with two independent questions: Verifiability (can you cheaply check the output? → x-axis) and Reachability (can AI even generate it at all? → y-axis). Top-right is the automation zone. Below, this task's four stages are plotted — they sit high (sorting text is squarely within reach) but at mid verifiability (routing is only checkable after reading), so they land just left of "full automation".

Automate AI acts, human audits Human originates Deeply human-led Verifiability — easier to check → Reachability — AI can generate it ↑ overall — on-the-loop cluster 1 2 3 4
1Gather (ingest) 2Classify 3Decide (route/close) 4Act (move/close)

Plot each stage by its two-axis position; classify the whole task by its highest-stakes stage. Here every stage is reachable, but read-to-check verifiability keeps it one notch back from full automation — HUMAN-ON-THE-LOOP. chollet-measure

3

What this means, in plain language

Let the AI sort the mail — but keep reading the daily log.

"On the loop" means the AI does the work and acts on its own (it tags, routes, and closes junk tickets), while you watch over it like a pilot watching autopilot. Each morning you skim an audit log, you can reopen or re-route anything with one click, and you stay the person in command. It's a step more cautious than full automation because a wrongly-closed ticket could hide a real problem — and a step more relaxed than approving every single routing by hand.

4

The task, as I understood it

Tickets arrive all day from several channels (email, web form, in-app). Today a person reads each one, decides the topic and urgency, tags it, drops it into the right queue, and deletes the spam and duplicates. You want AI to do that sorting automatically and at volume — read every ticket, classify topic and priority, apply tags, route to the right team, and auto-close the obvious junk — so your team only works real, pre-sorted tickets. A misroute is annoying but cheap to fix. Importantly, the AI does not reply to customers; your people still write the answers.

If any of this is wrong, tell me and I'll re-run the assessment — the level can shift.

5

Why this level — factor by factor

Each row is one lever. The "pushed toward" chip shows whether your answer argued for more automation, more human involvement, or set a hard floor. Every lever links to the source it comes from.

FactorYour answerPushed towardSource
Verifiability
Can you check it's right cheaply, no expert needed?
"Yes, but I have to read it." Routing is checkable, yet categories aren't a single machine-scored right/wrong (V2). Needs audit, not hands-off karpathy-verifiability, mindstudio-map
Consumption
Does a human read the output before it matters?
Mostly auto-consumed — tickets flow into queues without a human reading each — but you read a daily audit log. Auto-flow, with a human audit mindstudio-map
Reversibility
How hard to undo a mistake?
Easy — a misroute or wrong close is reopened/re-routed in seconds. The AI acts, then you can veto. Allows act-then-notify autonomy sheridan-verplank, parasuraman-2000
Stakes / risk
How bad if it slips through?
Low-to-moderate: usually a slow first reply, not financial or legal harm — provided urgent items escalate. Allows more autonomy anthropic-autonomy, openai-agents-guide
Volume / frequency
High-volume and repetitive, or one-off?
Many tickets a day, continuously — clearly worth automating. Justifies building automation zwingmann-aug
Specifiability
Could you write the steps as a flowchart?
Mostly: read → classify topic & priority → tag → route → close if spam/dupe. A repeatable recipe. Build as a fixed workflow anthropic-agents
Reachability — can AI even generate this?
Does it need a brand-new idea AI has not seen?
No — it classifies text that already exists (R1). Nothing out-of-distribution is required. Reachable — allows automation chollet-measure
People-impact (regulatory floor)
Does it decide about a specific person?
No (as scoped) — it sorts content, it doesn't judge a person. Would change if it denied refunds or flagged users for bans. No legal floor triggered (yet) gdpr-22, eu-ai-act-14
Supervisability
Can a human monitor and step in within time?
Yes — tickets are async and reviewable in bulk, so you have time to audit and intervene. On-the-loop is realistic (not the Level-3 trap) sae-j3016
Motivation / ownership
Do you want to own this part?
No — triage is a chore you'd happily delegate. Safe to delegate lubars-tan

Net: high volume, reversible, supervisable, specifiable, no people-impact and reachable (R1) would allow full automation — but because routing is only checkable after reading (V2) and a wrong auto-close can bury a real issue, we hold one notch back at HUMAN-ON-THE-LOOP: let it run, but keep eyes on the log. mindstudio-map, anthropic-autonomy

6

Stage breakdown — where the task can split

Most tasks are really four smaller steps, and they don't all need the same level of oversight. We classify the whole task by its highest-stakes stage, but you can automate the easy stages more aggressively. parasuraman-2000

Stage A
Gather info
FULL AUTOMATION

Ingesting tickets from each channel is mechanical and machine-checkable.

Stage B
Analyze / classify
HUMAN-ON-THE-LOOP

Topic, priority, tags — AI labels on its own; you audit a sample.

Stage C
Decide
HUMAN-ON-THE-LOOP

Which queue? Spam/dupe? Low-confidence cases divert to a human queue.

Stage D
Act / execute
HUMAN-ON-THE-LOOP

Move to the queue, auto-close junk — acts then notifies, all reversible.

Stage-splitting tip: to start more cautiously, keep auto-close at HUMAN-IN-THE-LOOP — the AI flags "spam/duplicate", you confirm the close — while routing already runs on-the-loop (it's so easily reversed). And legal / safety / security / VIP tickets should never auto-close: always escalate to a human.

7

Recommended setup — what to build

Tooling level: a predefined workflow (a fixed classify-and-route pipeline), not a free-roaming agent. Your steps are stable and writable as a flowchart, so a simple, repeatable workflow is the most reliable and the easiest to trust. Save the full autonomous-agent approach for tasks whose steps change each time. anthropic-agents

Build this first (the minimum viable version):

  • One intake that collects all incoming tickets into a single stream.
  • One classifier prompt returning topic, priority, tags, suggested queue, a spam/duplicate flag, and a confidence score.
  • A routing rule: high-confidence → auto-route; low-confidence → a "needs-human" queue.
  • A daily audit digest dropped in front of you each morning — so for the first weeks you still confirm closes.
8

Guardrails & checkpoints

  • Confidence threshold: below a set confidence, the ticket diverts to a human queue instead of being guessed.
  • Hard-escalation rules: legal, safety, security, payment-dispute, or VIP tickets are never auto-closed and route to a human immediately.
  • Daily audit log: every auto-action is logged with one-click reopen / re-route; you skim it each morning.
  • Sample audit: review a small random percentage weekly to measure accuracy even when nothing looks wrong.
  • No auto-reply to customers: the AI sorts; it does not answer. A customer-facing reply would raise this to HUMAN-IN-THE-LOOP.
  • Stop switch: one obvious way to pause the whole pipeline — you stay in command. eu-ai-act-14
9

Start-small graduation plan

Autonomy is earned, not assumed. Start more cautious, then relax oversight as the AI proves itself reliable. hitl-hotl

1

Weeks 1–2 · Flag-only HUMAN-IN-THE-LOOP

AI suggests topic/priority/queue and a spam flag; a human confirms every routing and close. You're calibrating trust and tuning the prompt.

2

Weeks 3–8 · Auto-route, audit daily HUMAN-ON-THE-LOOP

Once labels are reliably right, let it route automatically with a daily audit and reopen button; keep closes manual. This is your recommended steady state.

3

Month 3+ · Auto-close the obvious junk HUMAN-ON-THE-LOOP

Add auto-close for high-confidence spam/duplicates; widen the audit interval as accuracy holds — but keep the escalation rules, the human queue, and the stop switch forever.

Because a wrong close can bury a real ticket, we'd not push this all the way to FULL AUTOMATION. On-the-loop, with escalation rules, is the right ceiling here. anthropic-autonomy

10

Example starter prompts

1 · Classify & route a ticket (the workhorse)
You are a support-ticket triage assistant. Read the ticket below and return JSON only:
{
  "topic": one of [billing, bug, how-to, account, feature-request, abuse, other],
  "priority": one of [urgent, normal, low],
  "tags": [short keywords],
  "suggested_queue": which team should handle it,
  "spam_or_duplicate": true/false,
  "confidence": 0.0-1.0,
  "escalate": true/false   // true if legal, safety, security, payment dispute, or VIP
}
Rules: if confidence < 0.8, set suggested_queue to "needs-human".
Never mark spam_or_duplicate when escalate is true. Use only what the ticket says.

TICKET:
[paste the ticket here]
2 · Daily audit digest (your supervision view)
Below is everything the triage workflow did automatically in the last 24 hours.
Produce a short audit digest:
1. Counts by topic, priority, and queue.
2. Every ticket it AUTO-CLOSED, one line each, so I can scan for mistakes.
3. Low-confidence (<0.8) tickets sent to "needs-human".
4. Anything that looks like a possible misroute or a buried urgent issue.
Keep it skimmable.

ACTIONS LOG:
[paste the day's triage actions here]
3 · Spot-check helper (measure accuracy)
Here are 20 randomly sampled tickets and the queue/priority the workflow assigned each.
For each, say AGREE or DISAGREE with the routing and priority; if you disagree, give the
correct one in a few words. End with an accuracy percentage. Be strict.

SAMPLE:
[paste sampled tickets + their assigned routing here]
11

Risks to watch — and when to pull back

A buried urgent ticket

The costliest miss is an urgent/safety issue auto-closed or mis-prioritized. The escalation rules and the daily auto-close review are your net — test them monthly with a planted example. openai-agents-guide

Over-aggressive auto-close

Real tickets closed as "duplicate." Tighten the confidence threshold and keep closes manual longer; reopening is cheap only if you catch it in the audit. karpathy-verifiability

Quiet drift

As your product or customer mix changes, old categories underperform. Re-run a spot-check (Prompt 3) monthly and re-tune. parasuraman-2000

Scope creep into people-decisions

If it starts auto-replying, approving refunds, or flagging users for bans, the people-impact floor kicks in — move that part to HUMAN-IN-THE-LOOP. gdpr-22

12

The evidence behind this advice

Open any card to see what the source says and follow the link to read it yourself.

karpathy-verifiability Verifiability is the top lever for what to automate
Automate work whose output you can check faster and cheaper than redoing it. Routing is checkable but must be read (V2), which is why we stop at supervise-don't-leave. The Verifiability Thesis — Andrej Karpathy
mindstudio-map The V1/V2/V3 + consumption map
Turns verifiability into buckets (V1 instant/objective, V2 read-to-check, V3 taste-only) and adds the "who consumes the output" axis — both used in the scorecard. Karpathy Verifiability Framework — MindStudio
sheridan-verplank Act-then-notify vs approve-before-act
The classic 10 levels of automation distinguish acting first with a human veto window (the shape of on-the-loop) from requiring approval before each action — exactly the line between this bucket and the next. Levels of Automation (Sheridan & Verplank, 1978) — PetroWiki
parasuraman-2000 Automate by stage; rate by the riskiest one
The four-stage model (gather / analyze / decide / act) and the rule to classify a task by its highest-stakes stage — the backbone of the stage breakdown. Types and Levels of Human Interaction with Automation — Parasuraman, Sheridan & Wickens (2000)
sae-j3016 Avoid the unstable supervision middle
The driving-automation levels warn about "Level 3," where a human must take over but can't in time. Async tickets make supervision genuinely feasible here. SAE Levels of Driving Automation (J3016) — ANSI
anthropic-agents Start simple: prompt → workflow → agent
Recommends the simplest thing that works and reserving full agents for genuinely open-ended tasks. Your stable steps fit a predefined workflow. Building Effective Agents — Anthropic
anthropic-autonomy Autonomy is earned per deployment
Frames autonomy as an emergent, risk-weighted property of each deployment — supporting our "graduate carefully, keep a ceiling" stance. Measuring AI Agent Autonomy in Practice — Anthropic
openai-agents-guide Risk-based gating & human triggers
Lays out when to insert human checkpoints based on the cost of an action — informs the escalation rules and the keep-closes-manual stance. A Practical Guide to Building Agents — OpenAI
zwingmann-aug Augmentation vs automation; volume matters
Distinguishes work to augment from work to automate and notes high volume justifies automating — the case for building this pipeline. AI Augmentation vs AI Automation — Tobias Zwingmann
chollet-measure Reachability: AI interpolates, it struggles with true novelty
Defines intelligence as skill-acquisition efficiency over novel tasks; AI excels inside its experienced distribution but struggles with genuine novelty. Classifying existing tickets is R1 — well within reach. The unreachable region (a new method, a paradigm shift, an original cross-field leap) would need a human to originate it. On the Measure of Intelligence — François Chollet (2019)
lubars-tan What makes a task safe to delegate
Delegability factors — motivation, difficulty, risk, trust — support reading "you'd happily hand this off" as a point toward automation. Towards a Framework of Task Delegability — Lubars & Tan (NeurIPS 2019)
eu-ai-act-14 Keep oversight and a stop button
Requires human oversight proportionate to risk, including the ability to stop or override — the basis for the mandatory stop switch and audit log. EU AI Act Article 14: Human Oversight
gdpr-22 People-decisions need a human
Gives individuals a right to human intervention in consequential automated decisions. Not triggered by routing, but it's the floor if scope creeps toward decisions about people (bans, refund denials). GDPR Article 22: Automated individual decision-making
hitl-hotl Graduate from in-the-loop to on-the-loop
Describes the maturity path of starting with tight human approval and relaxing as trust grows — the exact shape of our flag-only → auto-route → auto-close ladder. From Human-in-the-Loop to Human-on-the-Loop — ByteBridge