Three quarters of AI feature cost comes from calls that did not need the model

A client's support summarization feature was burning ₹2.1 lakh a month in OpenAI tokens. We spent two days profiling the request log and found that 74% of spend came from calls where the model was doing work a regex could handle. The remaining 26% were the calls that actually justified a frontier model.

The first thing we look at is not the prompt. It is the call site. Most AI features have a handler that decides whether to invoke the model at all, and that handler is usually too permissive.

Caching at three levels

Semantic cache first. If a user asks "how do I reset my password" and someone asked the same question forty minutes ago, the answer has not changed. We use a pgvector table with a cosine similarity threshold of 0.92. Anything above that threshold returns the cached response and skips the model call entirely.

import psycopg2

def get_cached(query_vec, conn):
    cur = conn.cursor()
    cur.execute("""
        SELECT answer FROM cache
        WHERE embedding <=> %s < 0.08
        ORDER BY embedding <=> %s
        LIMIT 1
    """, (query_vec, query_vec))
    row = cur.fetchone()
    return row[0] if row else None

The 0.08 threshold corresponds to roughly 0.92 cosine similarity for normalized embeddings. We arrived at that number by running a hundred known-similar and hundred known-dissimilar pairs through the model and checking where false positives dropped off. It is not universal. For your data, measure.

Second level: exact-match cache on the raw prompt string, before embedding. Hash it, check Redis, skip the embedding call too. This sounds obvious, but it catches repeat traffic from frontend retries and webhook storms. We saw 18% of requests were exact duplicates within a five-minute window.

Third level: cache the embedding itself. If the same document chunk gets embedded on every retrieval call, you pay for the embedding API twice. Embed once, store the vector alongside the text.

Shorter prompts, not shorter context

The instinct when token costs rise is to truncate context. That makes the model worse and pushes costs up downstream when users retry because the answer was wrong. What we actually do is strip the system prompt.

A 1,200-token system prompt explaining the model's role, the output format, and ten bullet points of company policy runs on every single call. If you handle 80,000 calls a month, that system prompt alone costs more than the actual user queries. We rewrote ours to 180 tokens by moving the output format into a JSON schema passed via structured outputs, and moving the policy bullets into the retrieval context where they only get included when relevant.

The tradeoff: the model needs clearer instructions when the system prompt is thin. We spent a week tuning and testing. That is real work, not a free saving.

Knowing which calls do not need the big model

After caching and prompt reduction, we looked at what was left. The call distribution looked like this: classification (41%), summarization of short text (22%), drafting responses (19%), and complex reasoning (18%).

Classification does not need GPT-4o. A fine-tuned BERT classifier or even a well-prompted Llama 3 8B handles intent classification at roughly 4% of the cost per call. We routed classification to a local Llama 3 8B deployment on a ₹12,000/month GPU instance. Payback was nine days.

Short-text summarization went to Claude Haiku. Drafting and complex reasoning stayed on the frontier model.

The routing decision itself is cheap. A rule-based classifier handles most of it: if the input is under 200 tokens and the task is tagging, route to the small model. If the input is a complex multi-turn conversation about a disputed invoice, use the big one.

We made the routing observable. Every logged request records which model handled it, the input token count, the output token count, and the cost. When a user complains about a bad answer, we can see immediately whether it came from the cheap path — and we can decide whether to upgrade that specific intent rather than re-evaluating the whole routing strategy.

The client's monthly spend after these changes: ₹58,000. Most of that is the frontier model doing the 18% of calls that genuinely needed it. The cheap instances, the caching layer, and the prompt trimming together handle the rest for under ₹15,000.