Three quarters of AI feature cost comes from calls that did not need the model

Take a support summarization feature burning ₹2 lakh a month in frontier-model tokens. Spend two days profiling the request log and the usual finding is that around three quarters of the spend comes from calls where the model is doing work a regex, a cache, or a much smaller model could handle. The remaining quarter is the set of calls that actually justify a frontier model.

The first thing we look at is not the prompt. It is the call site. Most AI features have a handler that decides whether to invoke the model at all, and that handler is usually too permissive.

Caching at three levels

Semantic cache first. If a user asks "how do I reset my password" and someone asked the same question forty minutes ago, the answer has not changed. We use a pgvector table with a cosine similarity threshold of 0.92. Anything above that threshold returns the cached response and skips the model call entirely.

import psycopg2

def get_cached(query_vec, conn):
    cur = conn.cursor()
    cur.execute("""
        SELECT answer FROM cache
        WHERE embedding <=> %s < 0.08
        ORDER BY embedding <=> %s
        LIMIT 1
    """, (query_vec, query_vec))
    row = cur.fetchone()
    return row[0] if row else None

The 0.08 threshold corresponds to roughly 0.92 cosine similarity for normalized embeddings. We arrived at that number by running a hundred known-similar and hundred known-dissimilar pairs through the model and checking where false positives dropped off. It is not universal. For your data, measure.

Second level: exact-match cache on the raw prompt string, before embedding. Hash it, check Redis, skip the embedding call too. This sounds obvious, but it catches repeat traffic from frontend retries and webhook storms. It is not unusual to find 15 to 20% of requests are exact duplicates within a five-minute window.

Third level: cache the embedding itself. If the same document chunk gets embedded on every retrieval call, you pay for the embedding API twice. Embed once, store the vector alongside the text.

Shorter prompts, not shorter context

The instinct when token costs rise is to truncate context. That makes the model worse and pushes costs up downstream when users retry because the answer was wrong. What we actually do is strip the system prompt.

A 1,200-token system prompt explaining the model's role, the output format, and ten bullet points of company policy runs on every single call. If you handle 80,000 calls a month, that system prompt alone costs more than the actual user queries. A prompt like that can usually be brought down to under 200 tokens by moving the output format into a JSON schema passed via structured outputs, and moving the policy bullets into the retrieval context where they only get included when relevant.

The tradeoff: the model needs clearer instructions when the system prompt is thin. Expect to spend a week tuning and testing. That is real work, not a free saving.

Knowing which calls do not need the big model

After caching and prompt reduction, look at what is left. For a support workload, the call distribution tends to look something like this: classification (around 40%), summarization of short text (around 20%), drafting responses (around 20%), and complex reasoning (the remainder).

Classification does not need GPT-4o. A fine-tuned BERT classifier or even a well-prompted Llama 3 8B handles intent classification at roughly 4% of the cost per call. Route classification to a local Llama 3 8B deployment on a ₹12,000/month GPU instance and, at the volumes above, it pays for itself within a couple of weeks.

Short-text summarization can go to Claude Haiku. Drafting and complex reasoning stay on the frontier model.

The routing decision itself is cheap. A rule-based classifier handles most of it: if the input is under 200 tokens and the task is tagging, route to the small model. If the input is a complex multi-turn conversation about a disputed invoice, use the big one.

We made the routing observable. Every logged request records which model handled it, the input token count, the output token count, and the cost. When a user complains about a bad answer, we can see immediately whether it came from the cheap path — and we can decide whether to upgrade that specific intent rather than re-evaluating the whole routing strategy.

Run the example above through these changes and the ₹2 lakh a month lands somewhere around ₹60,000. Most of that is the frontier model doing the minority of calls that genuinely need it. The cheap instances, the caching layer, and the prompt trimming together handle the rest for a fraction of that.