Cost & Usage Analytics

Module (team routing) — did the ticket reach the right team? The classifier never picks a team directly; it picks a primary_module (a fine-grained, partly technical axis like accounting or oem_integration). We derive from that which operational team the classification implies, and compare it against the swimlane_* tag a human ended up putting on the ticket. So this is a team vs. team comparison, not a module-string match — a ticket can land on a slightly different module and still be counted correct, provided it would have gone to the same team.

Issue type — did we call the kind of ticket right? The bot's issue_type (bug / enhancement / how-to / configuration …) against the class the human's tags imply. A simpler, more direct comparison, which is part of why it usually scores higher.

Agent approval — the share of 👍 among tickets a support agent actually rated. This is the only series where a human judged our recommendation rather than us inferring agreement from tags they happened to apply, which makes it the one number here that the circularity problem cannot touch. Two caveats, both real: rating is voluntary, so it is biased toward the notably good and the notably bad rather than being a fair sample; and very few tickets get rated, so its band is by far the widest on the chart. Treat a movement in it as a prompt to go read the 👎 notes below, not as a measurement. If you want this series to become trustworthy, the fix is more ratings, not more maths.

What counts toward a window. Only tickets carrying usable ground truth. A ticket is skipped when a human never applied a tag we can read, so we genuinely cannot say who was right. Dividing by all tickets instead would mean a stretch where outcome capture ran poorly looked like a stretch where triage performed poorly — and improving capture would show up as a regression.

Why module and issue type are each plotted twice. If a swimlane tag was already on the ticket when triage ran, the classifier was fed that tag as input. It agreeing with the tag afterwards proves nothing — it is an echo, not a verdict. Module — clean (solid) throws those out; Module — raw (dashed) keeps them. The catch is that the exclusion can only ever fire on an agreement — a disagreement is never an echo — so it strips wins and keeps every loss. Clean is therefore the number to trust after a merge, but read alone it looks like the bot's hit rate when it is a rate over a deliberately adversarial subsample; raw is much closer to how often the ticket actually reached the right team. The vertical gap between the two lines is the share of the signal that is circular, which is worth watching in its own right: as it widens, less and less of your traffic is independently measurable. Measured on 2026-09-08 that gap was 71.0% raw against 17.7% clean — 240 agreements excluded against 1 disagreement, so 91% of the raw agreements were echoes. Neither end of that bracket is "the accuracy"; the honest reading is the range, and which end you quote depends on the question.

Issue type is now plotted twice for the same reason. It has carried the identical exclusion since 2026-08-24 and had no raw twin until this was added, so when the guard began landing the only line on the chart slid downward with nothing beside it to say why. The pair makes that legible the way it already was for module.

Group — clean (dotted, same blue) is a third read on the same question as module: the team implied by the bot's module against the team implied by the Zendesk group the ticket sat in. Worth having because its echo path is much weaker — metadata.module maps one-to-one onto the answer space and the prompt makes the model declare whether it overrode it, whereas a group is an org-unit name a model would have to infer a module from. It is not echo-free, though: the group is handed to the classifier too, so it carries the same exclusion. Expect it to be thin and gappy — the support queue spans every team, so it yields no verdict at all rather than a fabricated one — and expect a wide band for a while. It only accrues history if it is drawn.

So it will not match the weekly ai-eval issue. That report's headline module disagreement rate is computed over every ticket with any readable tag, circular ones included, so it reads much better. Both are arithmetically correct; they answer different questions. This chart answers "when the human's decision was independent of ours, how often were we right?" — the harder and more useful question.

Reading the chart. Every point aggregates the days behind it, so the newest point is always current — there is no "still filling" week to discount. Two windows are drawn: 28 days bold, because that is the shortest window where module's interval narrows to roughly ±8pp and becomes worth acting on, and 7 days faint, which responds fast but is noise-dominated at this volume. Both are whole numbers of weeks on purpose — weekend ticket volume is a fraction of a weekday's, so a 30-day window would contain 4.3 weeks and quietly over-weight whichever weekdays fell in it five times. A rate is left as a gap only when nothing was measurable; thin-but-real data shows up as a wide band instead, so "we measured a little and know little" no longer looks like "we measured nothing". Hover any day for the counts, the interval, and the merges that landed on it.

Where the series starts. It begins on the first day anything was measurable, not the first day a ticket was triaged. Outcome capture began months after triage did — roughly 500 tickets from May carry no outcome at all — so an unfiltered view would otherwise open on two months of blank axis and read as a broken chart. Only that leading stretch is trimmed: a gap in the middle is left drawn, because that one means capture or triage genuinely stopped and is worth seeing. The coverage line above the chart always states the true total either way.

Adjacent points are not independent. Two neighbouring 28-day points share 27 of their 28 days. The line is a smoothed level, not a sequence of fresh measurements, so a gentle slope is not evidence of a trend on its own — that is what the band is for. If the band after a merge still overlaps the band before it, the honest reading is "we cannot tell yet".

The module lines stop before 2026-07-20. That day (PR #75) the module signal was redefined: a raw module-string match became a team-vs-team comparison, and the circularity exclusion was introduced at the same time. Outcomes captured before then are in the old frame and are not comparable, so any window reaching back past that date omits both module lines rather than averaging two definitions into a fake step change. They reappear on their own as the older rows age out of the window. Green spans the full range because the issue-type comparison has not changed since 2026-06-16 — but that is not the same as the plotted line never changing meaning: see the next note.

The clean issue-type line steps down on 2026-08-24, and that is the ruler moving. Its circularity exclusion shipped that day; before it, every issue-type agreement counted, circular or not. Because the exclusion can only fire on an agreement, the clean line falls as it lands while the raw line does not move at all — over the 28 days to 2026-09-08 the clean rate went 69.3% → 61.7% while its denominator fell 189 → 120, losing agreements at nearly five times the rate of disagreements. It is marked ◆ rather than gated: the pre-2026-08-24 rows are still valid readings of a less guarded signal, so blanking a month of history to avoid a visible step would cost more than labelling the step. ◆ means the ruler changed; ▲ means the classifier changed. Do not read one as the other.

Same measure as the loop's own verdicts. The per-ticket judgement here is shared code with the post-merge verification that labels an improvement improved / regressed on the agentic-changelog page (PRODUCTION_SIGNALS), so the chart and those verdicts cannot drift apart.