Versioning prompts so a quality regression can be traced and rolled back

A prompt is a build artifact

Picture a document-classification feature that drops from 94% accuracy on internal tests to 71% over a weekend. No code changed. No model changed. The prompt did — someone shortened the system message to save tokens, and the model started misclassifying edge-case invoices as receipts.

If the prompt lives in the same Git repo as the application code, you find the cause in minutes. The diff shows exactly what was removed, the commit links to a ticket, and `git revert` brings the old prompt back before the next batch run.

If the prompt is sitting in a config file on S3, or was edited directly in a prompt-management SaaS dashboard, you are guessing.

Prompts belong in source control

This sounds obvious, but in practice most teams store prompts in one of three places: environment variables, a database table, or a hosted prompt-management tool with its own versioning that nobody checks. The first two have no history. The third has history, but it lives outside the code that calls the prompt, so the connection between a deploy and a prompt change is invisible.

We keep prompts as plain text files in the repo, usually under `prompts/` with a naming convention:

prompts/
  invoice_classifier/
    v1.txt
    v2.txt
    v3.txt

The application loads the active version from a config that is also versioned:

PROMPT_VERSIONS = {
    "invoice_classifier": "v3",
}

def load_prompt(name: str) -> str:
    version = PROMPT_VERSIONS[name]
    path = Path("prompts") / name / f"{version}.txt"
    return path.read_text()

When someone wants to change the prompt, they add `v4.txt`, update the config, and open a PR. The diff is readable. Reviewers can see exactly what changed in the prompt text, not just that "a config value was updated." Old versions stay in the tree rather than being overwritten, so a rollback never depends on someone remembering what the previous wording was.

Why not just use a database table

A `prompts` table with a `version` column works fine for retrieval, but it pushes versioning out of the review flow. A developer editing a row in a database does not trigger a pull request. Nobody reviews the diff. The change ships the moment the UPDATE commits.

The tradeoff is deployment. File-based prompts mean a prompt change requires a code deploy. For a small team, that is fine — we deploy often and the deploys are small. For a larger org where a content team wants to iterate on prompts without engineering involvement, a database or external tool makes sense. But then the versioning discipline has to be enforced somewhere else, and usually it isn't.

What to commit alongside the prompt

The prompt file is necessary but not sufficient. We also commit, next to each prompt version, a small JSON file with the evaluation results from the last run against a fixed test set:

{
  "version": "v3",
  "test_set": "invoice_classifier/eval_set.json",
  "accuracy": 0.94,
  "evaluated_at": "2025-01-08T14:22:00Z",
  "model": "gpt-4o-2024-08-06"
}

This is not a substitute for running evaluations in CI. It is a record that the evaluation happened and what the numbers were. When v4 is proposed, the PR can include its own results file, and the reviewer sees the delta.

Rollback is just a config change

When a prompt regression ships, rolling back means updating one line in the version map and deploying. The old prompt file is still there, untouched. No database migrations, no digging through audit logs in a dashboard.

In the scenario above, the regression gets caught on Monday morning. The fix is a single-line config change and a deploy that takes a few minutes. The team then spends the rest of the afternoon understanding why the shorter prompt performed worse, which is the actual work — rollback is just what buys you time to do it.