Versioning prompts so a quality regression can be traced and rolled back

A prompt is a build artifact

A client's document-classification feature went from 94% accuracy on internal tests to 71% over a weekend. No code changed. No model changed. The prompt did — someone shortened the system message to save tokens, and the model started misclassifying edge-case invoices as receipts.

We found the cause in 20 minutes because the prompt lived in the same Git repo as the application code. The diff showed exactly what was removed, the commit linked to a ticket, and `git revert` brought the old prompt back before the next batch run.

If the prompt had been sitting in a config file on S3, or edited directly in a prompt-management SaaS dashboard, we would have been guessing.

Prompts belong in source control

This sounds obvious, but in practice most teams store prompts in one of three places: environment variables, a database table, or a hosted prompt-management tool with its own versioning that nobody checks. The first two have no history. The third has history, but it lives outside the code that calls the prompt, so the connection between a deploy and a prompt change is invisible.

We keep prompts as plain text files in the repo, usually under `prompts/` with a naming convention:

prompts/
  invoice_classifier/
    v1.txt
    v2.txt
    v3.txt

The application loads the active version from a config that is also versioned:

PROMPT_VERSIONS = {
    "invoice_classifier": "v3",
}

def load_prompt(name: str) -> str:
    version = PROMPT_VERSIONS[name]
    path = Path("prompts") / name / f"{version}.txt"
    return path.read_text()

When someone wants to change the prompt, they add `v4.txt`, update the config, and open a PR. The diff is readable. Reviewers can see exactly what changed in the prompt text, not just that "a config value was updated."

Why not just use a database table

A `prompts` table with a `version` column works fine for retrieval, but it pushes versioning out of the review flow. A developer editing a row in a database does not trigger a pull request. Nobody reviews the diff. The change ships the moment the UPDATE commits.

The tradeoff is deployment. File-based prompts mean a prompt change requires a code deploy. For a three-person team, that is fine — we deploy often and the deploys are small. For a larger org where a content team wants to iterate on prompts without engineering involvement, a database or external tool makes sense. But then the versioning discipline has to be enforced somewhere else, and usually it isn't.

What to commit alongside the prompt

The prompt file is necessary but not sufficient. We also commit, next to each prompt version, a small JSON file with the evaluation results from the last run against a fixed test set:

{
  "version": "v3",
  "test_set": "invoice_classifier/eval_set.json",
  "accuracy": 0.94,
  "evaluated_at": "2025-01-08T14:22:00Z",
  "model": "gpt-4o-2024-08-06"
}

This is not a substitute for running evaluations in CI. It is a record that the evaluation happened and what the numbers were. When v4 is proposed, the PR can include its own results file, and the reviewer sees the delta.

Rollback is just a config change

When a prompt regression ships, rolling back means updating one line in the version map and deploying. The old prompt file is still there, untouched. No database migrations, no digging through audit logs in a dashboard.

The invoice-classification regression was caught on a Monday morning. The fix was a single-line config change and a deploy that took four minutes. The team then spent the rest of the afternoon understanding why the shorter prompt performed worse, which is the actual work — rollback is just what buys you time to do it.