A signup lands at 2 AM and someone has to be awake
A new customer signs up, pays, and gets a confirmation email. Their workspace is empty. The database schema is not initialized, their admin user does not exist, and the billing subscription is in a pending state. Someone has to run a script, paste the tenant ID into three services, and verify it worked. At 2 AM, nobody does. The customer logs in at 9, sees nothing, and emails support.
We built tenant provisioning as a single state machine driven by a queue. The trigger is the payment success webhook. Everything else is automatic, idempotent, and observable.
The entry point
The webhook handler does one thing: it enqueues a job with the tenant ID and the plan details. No database work, no schema creation, no user seeding. If the handler is slow or crashes, the payment gateway retries, and the job is already queued.
@router.post("/webhooks/payments")
async def payment_webhook(request: Request):
event = verify_signature(await request.body())
if event["type"] == "payment.succeeded":
await queue.enqueue(
"provision_tenant",
{"tenant_id": event["tenant_id"],
"plan": event["plan"],
"seats": event["seats"]}
)
return {"status": "ok"}The handler returns in under 50ms. The gateway gets its 200 and stops retrying.
The provisioning job
The job runs through five steps in order. Each step writes its status to a `tenant_provisioning` table before it starts, so a retry knows where to pick up.
The steps: create the tenant schema or row-level isolation scope, seed the admin user from the signup data, configure the billing subscription in the provider, provision storage resources (S3 prefix, search index), and send the welcome email with a login link.
Each step is idempotent. If the job crashes after creating the schema but before seeding the user, the retry checks for the schema, finds it, and moves on. This matters because queue workers restart, deployments happen, and databases fail over. A provisioning job that cannot survive a mid-run interruption is a provisioning job that requires human intervention.
async def provision_tenant(job):
tenant_id = job["tenant_id"]
state = await get_state(tenant_id)
if state.schema_created is False:
await db.execute(f"CREATE SCHEMA tenant_{tenant_id}")
await mark_step(tenant_id, "schema_created")
if state.admin_seeded is False:
await db.execute(
"INSERT INTO users (tenant_id, email, role) "
"VALUES ($1, $2, 'admin') ON CONFLICT DO NOTHING",
[tenant_id, job["email"]]
)
await mark_step(tenant_id, "admin_seeded")
if state.billing_active is False:
sub = await stripe.subscriptions.create(
customer=job["customer_id"],
items=[{"price": PLAN_PRICES[job["plan"]]}]
)
await mark_step(tenant_id, "billing_active")What we chose not to automate
We do not provision infrastructure per tenant. No separate database, no separate Redis instance, no dedicated compute. We use schema-based isolation in Postgres for tenants on paid plans and row-level isolation for trial tenants. The tradeoff: a noisy tenant can affect others on the same database. We accept that because per-tenant infrastructure for a product at this stage costs more in operational complexity than it saves in isolation guarantees. If a tenant grows large enough to matter, we move them to a dedicated schema on a separate database, and that migration is scripted but triggered by a human.
We also do not send the welcome email from inside the provisioning job directly. The email is enqueued as a separate job that depends on the provisioning job completing. If the email provider is down, the tenant is still fully provisioned and can log in. The email is a courtesy, not a dependency.
Observability
The `tenant_provisioning` table is the source of truth. Every step has a timestamp, a status, and an error message if it failed. A failed step after three retries marks the tenant as `provisioning_failed` and pages us. The customer sees a holding page that says their workspace is being set up, not an error.
We query this table from an internal dashboard. When a customer writes to support saying their workspace is not ready, we can see exactly which step is stuck and why, without SSHing into anything.
The goal is not zero manual intervention. Things will fail in ways we did not predict. The goal is that manual intervention is rare, diagnostic, and fast — and that the default path from signup to live workspace has no human in it.