Routing support tickets with a confidence threshold that means something
Consider a support team that handles roughly 400 tickets a day across three product lines. Two of those lines account for 85% of volume, and the third is a newer enterprise tier where each ticket carries more weight — a missed routing there costs a customer, not just a rating.
The team wants a model to triage incoming tickets into the right queue. The straightforward part is the classification. The harder part, and the part that determines whether the feature survives contact with production, is deciding when the model should admit it doesn't know.
Say you fine-tune a DistilBERT classifier on 18 months of labeled tickets and it hits 94% accuracy on a held-out test set. That sounds fine until you look at the distribution of errors. In our experience the model is confident on the high-volume product lines and noticeably worse on the thin tier, where training data is sparse. A 94% aggregate accuracy can easily hide the fact that 1 in 8 enterprise tickets is misrouted.
This is where the confidence threshold comes in. Most classifier APIs return a probability distribution over labels. The top probability is the model's confidence in its prediction. Below some threshold, you don't auto-route — you send the ticket to a general queue for a human to triage.
Setting that threshold is not a tuning exercise. It's a decision about how much manual review you can afford and how much misrouting you can tolerate. We work backwards from the team's capacity.
Say the support team can review about 60 tickets per day in the general queue without falling behind. That's 15% of total volume. So the threshold needs to be low enough that at most 60 tickets per day fall below it. It also needs to be high enough that the enterprise tier's misrouting rate drops to something acceptable — say, under 3%.
Run the model against a week of unlabeled production tickets and plot the confidence distribution. In a scenario like this you would typically see a clear inflection — somewhere around 0.72 is common — below which accuracy drops sharply across all categories. Then walk the threshold up until the expensive tier's misrouting rate is under target while the general queue stays under capacity. For the numbers above, that might land around 0.78. That is the threshold, and it is specific to this team's capacity and this model's calibration, not a constant to copy.
The implementation is a thin layer around the model call:
import httpx
MODEL_URL = "https://classify.internal/api/v1/predict"
CONFIDENCE_THRESHOLD = 0.78
async def route_ticket(ticket_text: str, ticket_id: str) -> dict:
resp = await httpx.AsyncClient().post(
MODEL_URL,
json={"text": ticket_text},
timeout=5.0,
)
result = resp.json()
top_label = result["labels"][0]
top_prob = result["probabilities"][0]
if top_prob >= CONFIDENCE_THRESHOLD:
return {"queue": top_label, "auto": True}
return {"queue": "general", "auto": False, "model_suggestion": top_label}The `model_suggestion` field matters. When a human picks up a ticket from the general queue, seeing what the model guessed — and how far off the confidence was — helps them triage faster and gives us labeled data to retrain on.
We log every routing decision: the ticket ID, the model's full probability distribution, the threshold at the time, and whether a human overrode the suggestion. Every couple of weeks, re-run the threshold analysis against the latest data. Expect the threshold to move — up when a thin category grows and misrouting ticks up, back down after a retraining cycle improves the model's confidence calibration on that category.
One thing we deliberately do not do is build a UI for the support team to adjust the threshold themselves. The threshold is a lever that trades automation against manual labor cost, and the people feeling the pain of misrouting are not the same people who feel the pain of an overflowing general queue. That's a management decision, not a user preference.
How you know the threshold is working: the share of tickets landing in the general queue stays near the capacity you planned for, and humans agree with the model's suggestion on a meaningful fraction of them. If they agree on nearly all of them, the threshold is too conservative and is sending work to people that the model could have handled. If they agree on almost none, the model is not calibrated and the threshold is doing the wrong job. The number that matters most is the misrouting rate on the tier where errors are expensive, tracked before and after — that is the figure the whole feature is accountable to.