Evaluating model output with 80 labelled rows and a script

Evaluating model output with 80 labelled rows and a script

A legal-tech client in Delhi asked us to add a clause-classifier to their contract review tool. The previous vendor had shipped a prompt that "worked well in testing," but nobody could say what testing meant. When we asked how quality was measured, the answer was: the founder looked at ten outputs and they seemed right.

That is not evaluation. That is vibes.

The problem with adding model-backed features to an existing product is that the model is a moving target. OpenAI tweaks GPT-4o, your prompt starts producing different classifications, and nobody notices until a customer complains. You need a way to detect drift before it reaches production, and you need it to run without a research team.

We use a fixed labelled set and a script. The labelled set is 80 contracts, each with the correct clause classifications hand-annotated by the client's legal lead. Eighty is not a statistically significant sample for a research paper. It is enough to catch a prompt regression before a user does.

The script is straightforward. It sends each contract through the model, compares the output to the labelled ground truth, and prints a confusion matrix plus precision and recall per class. It runs in under three minutes and costs about $0.40 in API calls.

import json, openai, collections
from sklearn.metrics import classification_report

LABELS = ["indemnity", "termination", "governing_law", "payment", "confidentiality"]

with open("labelled_set.json") as f:
    rows = json.load(f)

y_true, y_pred = [], []
for row in rows:
    resp = openai.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": row["system_prompt"]},
            {"role": "user", "content": row["contract_text"]},
        ],
        response_format={"type": "json_object"},
    )
    preds = json.loads(resp.choices[0].message.content)["clauses"]
    for clause in row["expected"]:
        y_true.append(clause["type"])
        match = next((p for p in preds if p["text"] == clause["text"]), None)
        y_pred.append(match["type"] if match else "none")

print(classification_report(y_true, y_pred, labels=LABELS + ["none"]))

The labelled set stays in version control alongside the prompt. When we change the prompt, we run the script. When OpenAI ships a model update, we run the script. When the client says the classifier seems off, we run the script. The output is a single page of numbers. If recall on "indemnity" dropped from 0.91 to 0.74, we know exactly where to look.

There are tradeoffs. Eighty rows will not catch edge cases that appear once in 5,000 contracts. The labelled set reflects what the legal lead saw in 2024, and contract language drifts. We refresh about 20 rows per quarter, pulling from production traffic that the client flags as misclassified. The set grows slowly, which is fine. The point is repeatability, not coverage.

A common mistake is to make the labelled set too large too early. We have seen teams build 500-row evaluation sets before shipping the feature, then discover the prompt needs fundamental changes and half the labels are wrong for the new approach. Start with 50, ship, and grow it from real failures.

Another mistake is to evaluate against a held-out test set that nobody has read. If the labels are wrong, the metrics are wrong. Every row in the set should have been read by a human who can defend the label. For this client, that is the legal lead. For an e-commerce categorization feature we built last year, it was the catalog manager. The script does the counting; a person owns the truth.

The script also produces the raw mismatches as a JSON file. When a metric drops, we open the mismatches and read them. Sometimes the model is wrong. Sometimes the label is wrong. Sometimes the contract uses language that neither the model nor the legal lead had seen before. Distinguishing between those three cases is the actual work. The script just tells you where to look.

We keep the evaluation script in the same repository as the feature, and it runs on every CI pipeline trigger. A prompt change that drops recall below a threshold fails the build. The threshold is not 0.95. It is whatever the current production performance is, minus two points. The goal is to catch regressions, not to enforce perfection.

For a three-person workshop, this is the part of AI integration that matters. Not the model choice, not the prompt engineering tricks. Having a number you trust, produced by a script you can run, against a labelled set a person owns.