Showing why the model decided, without showing the prompt
A loan officer clicks "reject" on an application that your model flagged as high-risk. She wants to know why. Your model returned a confidence score of 0.87 and a one-line explanation buried in the response payload — but the raw prompt that produced that explanation is 400 tokens of system instructions, few-shot examples, and internal taxonomy codes that would confuse anyone outside your team.
The instinct is to show the model's full output and let the user figure it out. That works when the output is a single sentence. It falls apart when the output is structured, when it references internal categories, or when the prompt contains business logic you would rather not expose.
We ran into this on a claims triage feature for an insurance client. The model classifies incoming claims into urgency tiers and returns a short justification. Support agents needed to see the justification. They did not need to see the prompt's instructions about weighting "hospital admission" higher than "prescription refill," or the five example claims we passed in to anchor the output format.
The fix was to separate the reasoning from the prompt mechanics. We asked the model to return structured output with a `reasoning` field written for a non-technical reader, alongside the classification and confidence score. The prompt itself stayed internal. The user saw the `reasoning` string, the confidence as a percentage, and the factors the model said it weighed — all in plain language the model generated, not the prompt's internal scaffolding.
The structured output looks like this:
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": TRIAGE_SYSTEM_PROMPT},
{"role": "user", "content": claim_text},
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "triage_result",
"schema": {
"type": "object",
"properties": {
"urgency": {"type": "string"},
"confidence": {"type": "number"},
"reasoning": {"type": "string"},
"factors": {
"type": "array",
"items": {"type": "string"}
}
},
"required": ["urgency", "confidence", "reasoning", "factors"]
}
}
}
)The `reasoning` field is a sentence the model writes fresh each call, addressed to the support agent. The `factors` array lists three to five specific details from the claim that drove the decision. The confidence score is a number between 0 and 1. None of these expose the system prompt, the few-shot examples, or the internal urgency taxonomy.
There is a tradeoff. The model sometimes writes a `reasoning` string that is vaguer than you want — "the claim suggests moderate urgency based on several factors." You can tighten this with a prompt instruction like "name the specific details from the claim that influenced your decision," but you are still relying on the model to comply. If the reasoning needs to be auditable to a regulator, a model-generated sentence is not sufficient on its own. You need the structured factors logged alongside it, and you need a human review path for anything above a confidence threshold.
We log the full structured response — urgency, confidence, reasoning, factors — to a table that the support team can query. The raw prompt template is versioned in our codebase. If an agent disputes a classification, we can pull the log, see exactly what the model returned, and compare it against the prompt version that produced it. The agent never sees the prompt. The auditor can.
This split — user-facing reasoning in the response, prompt mechanics in the codebase — has held up across three features since. The reasoning quality depends on how well you instruct the model in the prompt, but the separation itself is clean. The user gets an explanation written for them. The prompt stays where it belongs.