Routing support tickets with a confidence threshold that means something
A client's support team handles roughly 400 tickets a day across three product lines. Two of those lines account for 85% of volume, and the third is a newer enterprise tier where each ticket carries more weight — a missed routing there costs a customer, not just a rating.
They wanted a model to triage incoming tickets into the right queue. The straightforward part is the classification. The harder part, and the part that determines whether the feature survives contact with production, is deciding when the model should admit it doesn't know.
We fine-tuned a DistilBERT classifier on 18 months of labeled tickets. On a held-out test set it hit 94% accuracy, which sounded fine until we looked at the distribution of errors. The model was confident on the high-volume product lines and noticeably worse on the enterprise tier, where training data was thinner. A 94% aggregate accuracy hid the fact that 1 in 8 enterprise tickets was misrouted.
This is where the confidence threshold comes in. Most classifier APIs return a probability distribution over labels. The top probability is the model's confidence in its prediction. Below some threshold, you don't auto-route — you send the ticket to a general queue for a human to triage.
Setting that threshold is not a tuning exercise. It's a decision about how much manual review you can afford and how much misrouting you can tolerate. We worked backwards from the team's capacity.
The support team could review about 60 tickets per day in the general queue without falling behind. That's 15% of total volume. So we needed a threshold high enough that at most 60 tickets per day fell below it. We also needed the threshold high enough that the enterprise tier's misrouting rate dropped to something acceptable — under 3%.
We ran the model against a week of unlabeled production tickets and plotted the confidence distribution. The curve had a clear inflection around 0.72. Below that, accuracy dropped sharply across all categories. At 0.78, the enterprise misrouting rate fell to 2.1%, and the general queue received about 52 tickets per day on average. That was the threshold.
The implementation is a thin layer around the model call:
import httpx
MODEL_URL = "https://classify.internal/api/v1/predict"
CONFIDENCE_THRESHOLD = 0.78
async def route_ticket(ticket_text: str, ticket_id: str) -> dict:
resp = await httpx.AsyncClient().post(
MODEL_URL,
json={"text": ticket_text},
timeout=5.0,
)
result = resp.json()
top_label = result["labels"][0]
top_prob = result["probabilities"][0]
if top_prob >= CONFIDENCE_THRESHOLD:
return {"queue": top_label, "auto": True}
return {"queue": "general", "auto": False, "model_suggestion": top_label}The `model_suggestion` field matters. When a human picks up a ticket from the general queue, seeing what the model guessed — and how far off the confidence was — helps them triage faster and gives us labeled data to retrain on.
We log every routing decision: the ticket ID, the model's full probability distribution, the threshold at the time, and whether a human overrode the suggestion. Every two weeks we re-run the threshold analysis against the latest data. The threshold has moved twice in six months — up to 0.80 when the enterprise tier grew and misrouting ticked up, then back to 0.78 after a retraining cycle improved the model's confidence calibration on that category.
One thing we deliberately did not do is build a UI for the support team to adjust the threshold themselves. The threshold is a lever that trades automation against manual labor cost, and the people feeling the pain of misrouting are not the same people who feel the pain of an overflowing general queue. That's a management decision, not a user preference.
The feature has been in production for four months. About 13% of tickets go to the general queue. Of those, humans agree with the model's suggestion 41% of the time, which tells us the threshold is conservative enough to catch real uncertainty without becoming a bottleneck. The enterprise tier misrouting rate sits at 2.4%, down from 12% before the threshold was in place.