When
On ticket create & update
This agent watches Zendesk and, for every ticket, drafts a full triage — what it's about, what to do, and the evidence behind it — without ever changing the ticket. It runs in observe mode: everything it produces is a draft on this dashboard for a human to use. Here's exactly what happens, when, and how — for both new and updated tickets — plus how the system teaches itself to get better over time.
Resolve — handled / closable
Suggest — a draft for a human to send
Escalate — route to a team
Needs review — low confidence, human required
The pathFrom a Zendesk event to a triage on your screen
Every ticket takes the same eight-stage journey. Stages 1–4 decide whether and when to run; stages 5–8 are the actual triage.
1
A Zendesk trigger fires
Zendesk's “GHE Agentic Workflow” trigger fires when a ticket is created or updated and POSTs a small signed webhook to the container. It carries just the ticket id — the container always re-fetches the live ticket, so it never works from stale data.
How
Bearer-authed
POST /webhook/zendesk2
Debounce & coalesce
A brand-new ticket is triaged immediately so it appears fast. Updates are debounced: the first edit schedules one re-triage ~10 minutes out, and any further edits in that window fold into that single run — so a flurry of edits becomes one re-triage, not one per edit.
New ticket
Runs now
Updated ticket
≤ 1 re-triage per 10-min window
3
Queue
Runs are processed one ticket at a time (a serial queue), so nothing collides. Within a single ticket, the sub-agents in stage 6 still run in parallel.
Concurrency
1 ticket at a time
Within a ticket
Sub-agents run in parallel
4
Loop guard
A safety check skips the run if the ticket's latest activity came from the bot itself — defense against triage loops. (In observe mode the agent barely touches Zendesk, so this rarely trips.)
Purpose
Never react to our own writes
Otherwise
Proceed to triage
5
Read the screenshots when there are any
A third of tickets carry an image, and on the ones that matter the text collapses to almost nothing. Ticket 181537 is a P1 with multiple locations down and a six-word description: “image attached of the error message.” Triage used to classify that blind. Now every screenshot is transcribed and described before the classifier runs, so the error text inside the picture becomes searchable text for every step below — the classifier, the knowledge search, the dedup match, all of it. No sub-agent had to change: they read the transcription out of the same ticket context they already open.
Where the images are
Mostly pasted inline, not uploaded — Zendesk leaves inline images out of a comment's
attachments[], so we read the message HTML too (33% of tickets, vs. 10% looking only at attachments)Screenshot vs. email signature
Filename can't tell them apart — Outlook calls the first embedded image
image001.png every time. A content hash can: a signature is byte-identical across hundreds of tickets, a screenshot is uniqueModel & cost
Haiku 4.5 — transcription is perception, not reasoning. A typical Excede screenshot bills ~448 visual tokens, and cached images are never re-read
If it fails
Degrades to text-only triage — a bad image never fails a ticket
6
Enrich → Classify → Route → Sub-agents
The heart of the run: pull the full ticket + comments + metadata — including the owner (assignee), the group that owns the ticket, requester, support swim lane and billing status — classify it across seven dimensions, decide which specialist sub-agents are needed, and run them. Detailed below ↓
Classifier
Sonnet 5 on every ticket
Sub-agents
0–4, chosen by the classification
7
Synthesize the proposal
All the sub-agent output is merged into one coherent proposal: an internal note, an optional customer-reply draft written in the ticket owner's (assignee's) voice, proposed tags & Jira actions, and a single recommendation — resolve, suggest, escalate, or needs review.
8
Save → dashboard
Token usage & cost are tallied per step, the record is saved, and it shows on the triage dashboard within seconds. Re-triages append to the ticket's run log, so you can see the true cost per ticket over time.
Stored
Result + per-run cost log
Zendesk write
An
ai-reviewed tag (once) + a link back to this analysisIf the run fails, the ticket still lands here. A crash writes a record marked needs review rather than leaving the ticket invisible — with the stage that failed recorded on it, so a crashed run is never mistaken for one the agent judged uncertain. Those rows are deliberately left out of the accuracy figures and the emergent-signal panel: a run that died says nothing about what the ticket was, and counting it would report an outage as a product trend. They still show on the grid, because a ticket nobody triaged is exactly the one a person needs to see.
TimingNew tickets vs. updated tickets
The only difference is when the run happens. The triage itself (stages 5–8) is identical.
Incoming immediate
A ticket we've never seen triages right away.
- t = 0s — Ticket created in Zendesk → webhook fires
- t = 0s — First sighting → runs immediately
- ~30–90s — Classify + sub-agents + synthesize
- done — Appears on the dashboard;
ai-reviewedtag + triage link written once
Updated debounced
Edits tend to arrive in bursts, so we wait for the dust to settle.
- t = 0m — Ticket edited → schedules a re-triage for +10m
- t = 2m — More edits → fold into the same pending run
- t = 10m — One re-triage fires, re-fetching the latest ticket
- done — Dashboard refreshes; run log gains one entry (no re-tag)
Why debounce? A support agent editing a ticket can trigger several updates in a minute. Without coalescing, each would spawn its own triage — noise and cost. One re-triage per window reflects the final state and keeps spend bounded. The window is configurable.
Inside a runClassify, then fan out to specialists
Every ticket is first read by the classifier. Its answers decide which specialist sub-agents run — so a simple how-to and a production bug take very different paths.
🖼️ Image understanding Haiku 4.5 · only when the ticket has screenshots
Runs before the classifier and writes what it sees into the ticket context, so the error text in a screenshot is ordinary searchable text by the time anything below reads it. On a ticket whose whole description is “image attached of the error message,” this is the difference between triaging the problem and triaging six words.
— then, on every ticket —
🧠 Classifier Sonnet 5 · runs on every ticket
Reads the ticket, comments, Zendesk metadata and any image transcriptions from above, then decides seven things — each with a confidence score:
ModuleProductIssue type
Functional areaRoot causeUrgency
Action required
— routes to 0–5 specialists —
Knowledge resolve draft answer
Searches the KB + product docs and drafts a first-pass answer.
Model: Sonnet 5
Fires for: how-to, configuration, printing, access, report issues — or as the fallback so every ticket gets a drafted proposal.
Fires for: how-to, configuration, printing, access, report issues — or as the fallback so every ticket gets a drafted proposal.
Dedup & link find related
Searches Jira + dev history for duplicate or related issues and proposes links.
Model: Haiku 4.5
Fires for: bugs, enhancements, data / performance / sync issues, or anything headed to dev.
Fires for: bugs, enhancements, data / performance / sync issues, or anything headed to dev.
Escalation package for dev
Assembles repro steps, affected module, and evidence for confirmed bugs.
Model: Sonnet 5
Fires for: escalate to dev / integration / infrastructure.
Fires for: escalate to dev / integration / infrastructure.
Product signal state the need
States the need behind an enhancement request and drafts the PD Jira item for it — read-only, nothing filed.
Model: Sonnet 5
Fires for: tickets the classifier calls an enhancement request, and only while the feature is switched on — see below.
Fires for: tickets the classifier calls an enhancement request, and only while the feature is switched on — see below.
Config assist setup steps
Looks up configuration / how-to steps from the knowledge base.
Model: Haiku 4.5
Fires for: configuration issues or “apply configuration” actions.
Fires for: configuration issues or “apply configuration” actions.
Shortcut: if the classifier decides the ticket needs no action (e.g. the customer already self-resolved), it skips every sub-agent and just records a “close — no action” proposal.
What ends up on the ticket
Evidence it attaches
Similar past tickets — each linked straight into Zendesk — plus the docs, KB articles and related dev work items it actually used, so you can check its reasoning rather than take it on trust.
Same-customer history
How many tickets this dealer has already filed about the same product or module recently, with clickable examples — and a ↻ badge when it judges the issue genuinely recurring for them.
Sliceable on the grid
Group, module, product, issue type, dealer, owner, resolution, urgency, billing status and support swim lane are all filters on the triage dashboard. Group comes first because it is a scope — pick your team and the page becomes that team's queue. It is a different field from the support swim lane, and the one that works for every team: swim lanes are support's own product-area tagger, and most other groups do not use them.
A second questionIs this ticket also product feedback?
Support's question is “how do we resolve this ticket.” Product's is “what does this ticket tell us to build.” The same triage run can answer both, because it has already read the ticket — so a sixth specialist states the need behind an enhancement request and drafts the PD Jira item a Product Owner would act on. Nothing is created in Jira. The draft stops on the dashboard, waiting for a human, and the step that would file it does not exist yet.
Why it is a separate agent and not another classifier question
Only ~9% of tickets need it
A deep extraction that most tickets do not need does not belong in the prompt every ticket pays for. The classifier already decides whether a ticket is an enhancement request; this agent only runs on the ones where it said yes, so the other 91% cost nothing.
The classifier is eval-gated
Its issue-type answer is one of only two dimensions graded against what humans actually did. Adding a field to that prompt can shift its other answers, so it would need a full eval run to land. A separate agent asks the same question at zero blast radius.
It cannot break a triage
It runs inside the same parallel block as the other specialists and under the same rule: if it fails, that one drawer section is missing and everything else about the ticket is unaffected. A ticket that is not an enhancement request is triaged byte-for-byte as before.
The need, not the request
This is the whole reason the agent exists rather than a keyword filter. A customer asks for the thing they can picture; the need is the outcome they are trying to reach. Getting it wrong forecloses the design before anyone has looked at it — so what they asked for is recorded separately, as one option among several, and never as the problem statement.
| What the ticket says | What the draft says |
|---|---|
| “Can you add a markup column to the PO screen?” | Parts managers cannot see transfer markup when they create a purchase order, so margin errors surface only at month-end close. |
| “We need a report.” | The dealership needs to reconcile vendor credits against received POs to catch margin leakage before the period closes. |
What the code checks, so the model is not taken on trust
The quote is real
A customer quote must appear verbatim in a message a customer actually wrote — checked against the ticket's own comments, with the author's role resolved from Zendesk. A paraphrase, or a line lifted from a Procede engineer's reply, is discarded and the draft flagged.
The item type is derived, not chosen
Whether the draft is an Epic or a Defect is decided in code from the evidence the ticket carried — documented behaviour or a regression makes it a Defect. The sections the body is then checked against cannot be picked by the same output being checked.
Some drafts cannot be filed at all
PD refuses a create without a Feature, and a few of our modules — add-ons especially — map to no honest Feature value. Those drafts are marked unfilable with the reason, rather than being offered to a reviewer as ready.
Where it stops. Reads are free; writes are not — and this agent does neither: it is given only the ability to read the ticket it was handed and write its answer to a file. No Jira tool, no Zendesk tool, no search. (Jira is searched for this ticket, by the separate dedup specialist above, and that search is read-only too.) The draft waits on the dashboard, and the Product signal page shows what the agent is seeing across every ticket — volume, what it concluded, why drafts are not filable, and whether the same pains actually recur.
Who does the thinkingModels & cost
Reasoning-heavy steps use Sonnet; mechanical search/lookup steps use the cheaper Haiku. The learning loop (below) uses Opus, because a prompt edit shapes all future triage. List pricing, per million tokens:
| Model | Used for | Input | Output |
|---|---|---|---|
| Opus 5 | Learning loop — authoring prompt edits | $5 | $25 |
| Sonnet 5 | Classify · Knowledge · Escalation · nightly analysis | $3 | $15 |
| Haiku 4.5 | Image understanding · Dedup & link · Config assist | $1 | $5 |
Every run records its exact tokens and cost per step — see the Analytics page for spend by model, by module, and per day.
GuardrailsObserve mode — what we do & don't touch
✅ What it writes
Two harmless breadcrumbs, and nothing else: an ai-reviewed tag added once on a ticket's first triage, and a Procede AI Triage Link field holding a one-click link from the Zendesk ticket back to this analysis.
🚫 What it never does
No comments or replies to customers. No status, priority, assignee, group or swim-lane changes — it never re-routes a ticket to another team. Every reply, note, tag and Jira action it produces is a draft shown here, applied only if a human chooses to.
♻️ Why that can't loop
Writing the link field counts as updating the ticket, which would normally trigger another triage. It writes the link only when the field doesn't already hold it — so the run it triggers finds nothing to change and stops. The loop ends on the data, not on a flag that could fail open.
🙂 Your feedback
On any ticket you can 👍 or 👎 the proposed action and leave a note either way — a correction on a 👎, or praise-with-a-caveat on a 👍. That verdict and note are saved to the ticket and become the strongest signal in the learning loop.
Across every ticketCatching a pattern before someone reports it twice
Everything above concerns one ticket at a time. The Trending page reads them together and looks for shape — what's spiking, which dealers are struggling, and what is breaking out right now.
📈 Leaderboards & trend lines
Volume by product, dealer, module, issue type and owner, plus a time-series you can pivot by any of them. Click anything to land on that slice of the triage grid.
🔥 Hot / emergent
What is arriving faster than normal for that particular thing — ranked on how unusual the volume is, how severe the tickets are, and how many dealers are affected.
🚨 Burst alerts
Five or more tickets of the same kind within three hours from three or more different dealers raises a callout here and on the main dashboard.
↻ Dealers heating up
Dealers filing above their own normal rate — scored the same way, but ranked against other dealers so a small customer having a bad week isn't buried by a big one. They drop off once back to normal.
Why “hot” isn't simply “3× more than usual”. A plain multiple misleads at low volume: three tickets where one was expected looks like a 3× spike, while a real 17-ticket surge across 13 dealers reads as only 2.8× and sinks below it. So the ranking asks how statistically surprising the volume is instead — which grows with the actual numbers — then weights that by how severe the tickets are and how many dealers are hit. A trickle of low-urgency tickets from one dealer no longer crowds out a genuine outage.
And if nothing is hot, it says so. The list is not padded to a fixed length; an item has to clear a bar to appear at all. When everything is flagged, nothing is — so an empty panel is a real answer, not a broken one: arrival rates are within their normal range.
Getting smarterThe self-improvement loop
Once a week the system grades its own recent work against what humans actually did, proposes one small fix to its own prompts, proves the fix does not regress anything, and opens it for review. A human is always the gate — nothing ships unattended. It is a loop with exits, not a circle: most weeks it stops early, on purpose.
What runs, when, and what it costs
Nothing here is triggered by a person remembering to. Every row is a schedule, and every one of them fails loudly rather than going green empty.
| What | When | Cost per run |
|---|---|---|
| Triage a ticket — the pipeline above | on every Zendesk create or update, four at a time | ~3.6 min |
| Capture outcomes — re-read the ticket, record what humans did | daily, 05:00 UTC | no AI spend |
| Reconcile — ask Zendesk what we never stored | daily, 06:00 UTC | no AI spend |
| Grade the week and dispatch one proposal | Saturday, 03:00 UTC | the 14 gate fixtures, twice — ~5 min, ~$5 a pass |
| Verify whether last week’s merged change actually moved production | Saturday, 04:00 UTC | no AI spend |
| Watchdog — did the loop run, and did any of it matter | Monday, 10:00 UTC | no AI spend |
| Golden-set audit — is the eval still measuring the right thing | monthly, 1st at 07:00 UTC | no AI spend |
| Field-tier eval — does the set still look like production | monthly, 2nd at 08:00 UTC | 19 fixtures — ~7 min, ~$7 |
Grading is cheap; being graded is not. Only two of those rows spend anything on models — everything else is reading tickets we already have. The expensive part is the eval, at roughly 0.37 minutes and $0.37 per fixture, per pass (measured: the full 33-fixture set ran in 12.3 minutes for $12.10, four fixtures at a time). Running the baseline and the candidate serially used to take about two hours; the baseline half is now cached on a content hash of the prompts, labels and fixtures, so a week’s proposals compare against one firm baseline instead of re-sampling a noisy one each time.
What the golden set is — and what it isn’t
The eval scores the classifier against tickets whose right answer a human has already written down. That set does two different jobs, and confusing them is how it quietly stopped being useful.
🔒 gate — 14 fixtures
The regression guard. Every proposal runs it, before and after. These are the unambiguous ones: a fixture that flaps here would block good changes at random, so nothing joins this tier until it has held still.
🎯 field — 19 fixtures
Not a guard — a mirror. It asks whether the set still resembles the tickets actually arriving. That is a monthly question, so it runs monthly rather than charging every proposal for it.
💸 Why it can’t just keep growing
Every fixture costs ~$0.37 and ~0.37 minutes per pass, forever. Two passes inside a 180-minute job puts a hard ceiling at 241 fixtures — and the money bites long before the clock does. “Add more fixtures” is usually aimed at the wrong constraint.
🚧 What it still can’t tell us
Which fixtures have never once changed a verdict — i.e. which are pure overhead. Every eval run is now recorded so that becomes answerable, but it needs a few months of history before “it never moved” says anything about the fixture rather than the sample size.
A near-perfect eval score and a third of production disagreeing are not a contradiction — they are the symptom. For months the eval sat at 100% on every dimension while humans were re-routing roughly a third of real tickets. Both numbers were true: the fixtures were the cases we already understood, so the eval was measuring how well the classifier handles what it is already good at. Six of the fourteen taxonomy modules had no fixture at all, including the one that produced the largest disagreement cluster of that week. An eval can only ever tell you about the tickets in it, which is why the monthly audit ranks coverage gaps against real production frequency and the set now grows from real tickets rather than when someone remembers.
And a fixture is never labelled by the thing being graded. New fixtures are curated from real tickets by a workflow that redacts them and attaches the human’s own resolution — but deliberately asserts no module label, because there is no honest mechanical source for one: the swim-lane tag is departmental and is fed to the classifier, and the routing group names a team, not a module. Inventing ground truth for the exact dimension the eval exists to measure is the failure that has cost this project weeks, twice. So the machine does the tedium and a person does the judgement.
Loop memory — so it stops re-sending a fix you already declined
| State | What it means | Blocks a repeat for |
|---|---|---|
proposed | A PR was opened. Nobody has decided yet. | 14 days |
rejected | A human looked at it and said no. | 90 days |
merged | It shipped, and its effect has not been read yet. | its dimension, for one cycle |
The two horizons are different on purpose.
rejected is a durable human verdict, so it binds for a long time. proposed is merely the absence of one — and it expires fast, because an entry that nothing ever flips reads forever as “a human is still reviewing this”. That is not hypothetical: one week the loop proposed nothing at all because two rows sat at proposed long after their PRs had closed, silently suppressing the two largest routing patterns with no way to recover. Expiry makes the suppression self-terminating by data instead of dependent on one event firing exactly once.A no is not always the end of the conversation. When the evidence behind a rejected fix is still the strongest signal of the week, the loop may offer a deliberately narrower attempt at it, marked as superseding the earlier one — so it arrives as attempt two, not as the same request again. But two noes settle it: once a narrowed retry has also been rejected, the pattern is closed rather than narrowed a third time. And one change per dimension per cycle — a merged change whose effect nobody has measured yet blocks another edit to the same dimension, because two overlapping changes make both unmeasurable.
Most of the machinery above exists because something was being measured wrongly. Each of these cost real weeks:
One signal doesn't need anyone to disagree. Everything above grades the agent against what a human did next — so it can only find mistakes a person visibly corrected. That leaves a blind spot: when the taxonomy has no word for something, nobody disagrees, because every available answer is equally wrong and the human tag set is too coarse to tell them apart. The loop ran for months over 55 tickets sharing one auto-generated subject, split down the middle between two issue types on near-identical text, and never mentioned it — nothing it measured had moved. So the weekly run now also looks for contradictions in its own output: values the classifier invented that aren't in the taxonomy, and near-identical tickets it filed differently. Identical tickets getting different answers is provably wrong without knowing which answer is right. Findings arrive as a drafted proposal in the weekly issue — a new category always needs a person, because it has to move the taxonomy, the labeled test set and its examples together.
Not every disagreement was a mistake — and that took six weekly reports to pin down. Accuracy was graded against the support swim lane, which turns out to answer a different question than we were asking it. On six tickets in one week the bot was called wrong and was actually right: the swim lane records the department that filed the ticket, not what the ticket is about. An accounting ticket raised by the parts desk is tagged parts, and the bot saying "accounting" scored as an error. So the weekly run now also reads the ticket’s group, and specifically whether a human moved it to a different one after triage. That is a much cleaner test: the agent never touches the group, so a re-route is a person actively disagreeing, not a tag that happened to be sitting there. A disagreement humans acted on is treated as a real miss; one where nobody touched the ticket is treated as what it usually is — a tag about who reported the problem.
Two accuracy numbers moved in August, and neither means triage got worse. The agent is shown the ticket’s existing tags before it classifies, which means "the agent agreed with the tag" is sometimes just the agent repeating what it was handed. That was already discounted for module routing; it was not discounted for issue type, so that figure had been quietly flattered for months. It is discounted now, which makes the number drop — because agreements are what leave the calculation, not errors. The same caveat applies to the module figure in the other direction: the discount can only remove agreements, never disagreements, so the "independent signal" rate reads worse than real agreement and always has. Both are measurement corrections. If you are comparing this month’s report to one from July, the denominators are different things.
When the loop finds a problem it cannot fix, it now says so somewhere permanent. The weekly run is only allowed to change one thing: the wording of the classifier prompts. Anything else it notices — a number measuring the wrong thing, a gap in what we capture, a bug — used to be written into that week’s report and then scrolled past, because every Saturday opens a brand-new report. It happened six weeks running with the same finding. Those now open their own tracking issue that survives the week, and a recurrence lands on the existing one rather than starting a fresh pile. There is also a Monday check for the quieter failure: a loop that runs perfectly, passes every health check, and proposes nothing week after week. Everything else was asking did it run; nothing was asking did any of it matter.
What you write in the wrap-up form is now part of how the agent learns. When you record what a ticket turned out to be, a second pass reads it and works out what it implies — which module it really belonged to, whether it was a defect or a config matter, and why it happened. It is never shown what the agent guessed, so it cannot just nod along; the two answers are compared afterwards. This matters most for root cause, which nothing else could ever check: no tag on a ticket records why something broke, so until now that part of the agent’s reasoning was never graded against anything real. It counts for less than a 👎 — a thumb is you judging the recommendation directly, this is a reading of your notes — but it is the next best thing, and unlike a tag it is your actual words. Roughly one ticket in five carries a resolution today; every wrap-up you fill in adds one. It also has to quote your words back for each thing it claims, and that quote is checked against what you actually wrote — if the phrase is not really there, the claim is thrown away rather than counted. So a short, specific wrap-up is worth more than a long vague one.
Why 👎 matters most. Tag changes and comments are indirect signals — a tag tells us where a ticket ended up, never whether the proposed action was right. A thumbs-down on the proposed action, with your note on why, is a direct human verdict on the exact thing the loop optimizes, so it's weighted above everything else. Your feedback literally steers next week's fix.
Right now almost nobody is voting, and that costs the loop its only direct signal. Votes ran at 23 in the week of 18 July and fell to zero by mid-August. With none, every accuracy figure in the weekly report is inferred from routing tags over a small subset of tickets — the loop is grading itself against its own inputs. Rating five tickets takes about two minutes and is the highest-value thing anyone can do for triage accuracy: open the tickets nobody has rated, pick one, and hit 👍 or 👎 next to “Was this right?”. A 👎 with one line of explanation becomes a pattern in next week's report; a tag disagreement can only ever be a guess about what you meant.
This dashboard records who uses it. Every signed-in request is logged with your name, what you opened (which page, which ticket), when, and how long the server took — failed and unauthenticated requests are not. It is kept for 13 months, then deleted automatically. It exists to answer two questions we could not answer before: is this tool worth maintaining, and when votes dry up, did people stop voting or stop opening the dashboard? It is not a measure of anyone's output — tickets-viewed is a poor proxy for support work, and the Adoption page is ordered by who was here most recently rather than ranked by volume. You can see exactly what is recorded about you there.
Runtime: a TypeScript service on Azure Container Apps. Triage behavior lives in shared prompts; model choices in
service/src/pipeline/models.ts; the debounce window in TRIAGE_UPDATE_COOLDOWN_MINUTES; the burst-alert and hot/emergent thresholds in TRENDING_ALERT_* and TRENDING_HOT_*. This page is documentation — see the dashboard for live triage.