My agent pipeline will happily spend an hour on a ticket that no pull request can close. Some tickets in a real backlog are already fixed. Some are a dashboard setting, a question, or a production data problem dressed up as a bug. Others need a product decision before anyone should touch the code.
So before the pipeline picks up work, something has to answer one question for each ticket: can a pull request alone resolve this? This post is about the triage step I built to answer it, and about the part that usually gets skipped: checking its answers against what actually happened. The code, the evaluation and the full write-up are in the ticket-triage repository.
Start with a rule
The first fix wasn’t a model at all. The pipeline got a rule: only queue tickets whose resolution is a commit in this repository, and say nothing at all on the others. No comment, no status change. In the skill’s words: “That silence is the point of the rule, so a run that comments on one has broken it even if it queued nothing.”
That handled the obvious cases. The harder question came with a backlog of 104 open tickets: which of them could the pipeline actually finish?
Narrow questions, decisions in code
I didn’t ask a model “can this be done with a PR?” and take its answer. I used TypeSafe’s Jev, a model that answers typed questions with probabilities rather than prose, and split the judgment into ten narrow questions asked in one request per ticket: a multiple choice for what resolving the ticket requires (a PR only, a PR plus a manual step, ops work only, a decision first, a question, or already resolved), eight yes/no questions (must the repo change? must someone do something outside it? is a product question open? does something need checking in production first?) and a size score.
The model gets the ticket, its comments, and a short description of what a pull request to this repository can and can’t change. Everything that decides the outcome lives in code. A bucket function reads the stored answers and applies thresholds, and its docstring says what they are: “Thresholds are starting points, not truths.”
That split paid for itself within two minutes of the first run. The first version checked the flags before it listened to the multiple-choice answer, so 28 tickets landed in “PR plus a manual step” and only 8 in “not a PR”. I rewrote the policy so the path answer leads and the flags only break ties, and re-applied it to the stored answers. Fifteen tickets moved out of “PR plus a manual step”, and it took no new model calls at all. Every later policy change was tested the same way, against answers already on disk.
The policy itself is just an ordered list of rules, and the first one that matches wins. A ticket is “already done” if the comments say so, or if a merged PR names it. It’s “pure PR” only if the path answer is clearly a pull request, every flag agrees, it’s small, and the thing it asks for doesn’t already exist. Then come “probably PR”, worth a quick look first; “not a PR” for ops work and questions; “needs a decision”; and “PR plus a manual step”.
People call this kind of thing an AI decision. It’s an if-chain, and the model’s job is to fill in the conditions.
Reviewing my own classifier
Once it worked, I asked for an adversarial review of it, the same way the pipeline reviews its own code. It found seven problems. The two that mattered most: there were no labels, so every threshold was a guess; and the model never saw the codebase, so a fix merged under a different ticket still read as open work. It also caught that the pipeline’s own comments on tickets were leaking into the input, so the classifier was partly grading the pipeline’s output.
The fix for “never sees the code” wasn’t more prose. It was facts that code can compute. For each ticket the script now finds the file paths and identifiers the ticket mentions and checks whether they exist on the base branch, and looks for pull requests and commits that name the ticket. Three new questions read that evidence, and the policy gained gates on size and on evidence. The comment above that rule puts it well: “Code evidence outranks the ticket text: a merged PR naming the ticket is a fact, not a reading.”
The best example was a ticket asking for an alert that already existed, built under another ticket. No PR named this one, so “does the evidence show it’s done?” said no. The new question, “does the thing it asks for already exist?”, said 0.73, and moved it out of the top bucket.
Checking it against what happened
A classifier with no evaluation is an opinion. I built labels from the pipeline’s own history: 28 tickets that shipped a pull request, and 26 that closed without code. Then I scored the classifier on them.
The important part is what it was allowed to see. Each ticket was rebuilt as it was when work began: comments added after pickup were dropped, and links from the ticket to its eventual pull request were withheld. The code comment explains why: those links “would leak the label.”
Counting “pure PR” and “probably PR” together as a yes, the classifier got about 64% precision and 50% recall on those 54 tickets. Fourteen of its 22 yeses were right, and it found 14 of the 28 tickets that really shipped. With so few tickets the uncertainty is wide: the 95% interval on precision runs from roughly 43% to 80%.
Two things make those numbers better than they look. Ten of its fourteen misses went to “needs a decision”, and that bucket caught 7 of the 11 shipped tickets where the pipeline itself later stopped to ask a person first. When it was wrong, it mostly erred toward asking. And four of its five strict false positives were tickets that were decided or done elsewhere after pickup. With today’s comments and evidence they come back “already done”, but the as-of-pickup evaluation, correctly, can’t see that.
The evidence from git barely moved the evaluation: precision went from 61% to 64%, recall stayed the same. That’s expected, because the evaluation hides exactly the links the evidence depends on. Where it helps (duplicates and already-merged work) shows up in live runs, not in this test. The gain is real but anecdotal, and I’d rather say so than claim it.
My conclusion at the time was to treat it “as a shortlist to review, not a list to hand straight to the pipeline.”
Using it, and what that does and doesn’t prove
The final run put 24 tickets in “pure PR” and 19 in “probably PR”. I reviewed those 43 and fed them to the pipeline. Thirty-three shipped as pull requests, 4 were parked with a decision brief, and 6 were closed without a PR, three of them because the work turned out to be done already.
Of the 33 that shipped, 27 passed review on the first attempt: 82%, against 67% for every pipeline pull request before them. It’s tempting to credit the classifier, and I can’t. All but one of the pull requests in that run came from the shortlist, and no shortlisted ticket shipped outside it. So I can’t separate the classifier from the run: five tickets at a time, a month of pipeline fixes behind it, and whatever made those tickets easy. Without the shortlist in the baseline, the rest of the pipeline’s history passed first time 60% of the time, and the intervals overlap.
The honest claim is smaller: the shortlist was a decent queue, and it didn’t send the pipeline into many dead ends.
The labeler, and the labels
The same approach answered a second question: what type is this ticket (bug, feature, enhancement and so on), and how workable, ambiguous, large, risky and easy to locate is it? The labeler reads the ticket with its human label withheld (“The human label must not reach the model”), scores each dimension separately, and a rule in code routes each ticket to the next human step.
To check the type labels, I compared them against what actually shipped, judged by the same model reading each merged pull request’s description, not the ticket. The ticket can’t be the referee: “The ticket text is what the humans mislabelled from, so it cannot arbitrate its own label.”
The result was more about the labels than the model. On 130 shipped tickets, the human type label agreed with what the pull request did only about half the time (54%), while the model reading just the ticket agreed 71% of the time. About a quarter of shipped tickets (32 of 130, or 35% of those that had a type label at all) carried a type that disagreed with the model’s reading of the merged change. That’s a model’s judgment, not a human ruling, and a third of those calls were low-confidence.
The cause wasn’t careless labelling. Two of the labels had no stated boundary between them, and one had become the place small polish work went. Folding it into its neighbour lifted agreement from 54% to 65%, more than any change to the prompt. Two of the labeler’s worst early scores turned out to be measuring a loose label and a quirk of the process, not the model.
What I’d tell someone building this
- Ask narrow questions and decide in code. Store the answers and you can change your mind for free.
- Give the model facts your code can compute. Does this file exist? Does a merged PR name this ticket? Don’t ask it to guess what a lookup can answer.
- Build labels from what actually happened, and hide the answer when you score.
- Check the ground truth before blaming the model. Two of my worst numbers measured a sloppy label and a process artifact.
- Prefer a classifier that fails toward asking. A shortlist that sends its misses to “needs a decision” is more useful than one that’s right a bit more often but fails silently.
- Report the confound next to the good number.
The triage step was most useful as a way to ask better questions of my own process. The repository has the code, the evaluation with every number’s method, and the design history. It sends ticket text to the model provider, so check your own data policy before pointing it at a real backlog.