Queues, Retries, and Idempotency for Reliable AI Jobs

Expensive AI work must assume that requests time out, workers disappear, networks split, and users click twice. Reliability comes from giving every user intent a stable identity, recording every attempt, and making side effects safe to reconcile.

A logical job represents what the user asked to create. An attempt represents one execution on one worker or provider. Keep those identifiers separate.

job_456
  intent: render scene 7, take 2
  state: running
  idempotency_key: create-scene-7-take-2

attempt_1
  server: sage3
  provider_prompt_id: abc
  state: timed_out

attempt_2
  server: sage2
  state: running

This distinction answers important questions. Did the user create one job or two? Which server performed the work? Has cost already been reserved? Does a provider prompt ID exist that can be checked before retrying?

An idempotency key names one mutation. The client sends the same key when retrying the same request. The server either returns the original result or rejects mismatched parameters instead of creating another job.

Stripe’s API is a useful public reference: it stores the first result for an idempotency key and returns that result for matching retries. It also compares parameters to prevent accidental reuse. The same principle applies to generation creation even when payments are not involved.

Keys should not contain sensitive prompts or personal data. A random identifier or stable client operation ID is enough. Scope keys to the authenticated owner so two users cannot collide.

Backoff reduces synchronized retry storms. A common shape doubles the delay and adds jitter:

A queue handing a job to a worker does not prove the worker will finish. A lease marks temporary ownership. The worker renews it with heartbeats. If the lease expires, a recovery process can inspect provider state and decide whether to requeue.

Lease expiry is evidence that the worker stopped reporting, not evidence that external work stopped. ComfyUI or a cloud provider may still be generating. Store the provider prompt ID before waiting so recovery can query history.

A worker can complete generation and crash before acknowledging the queue. The queue may then deliver the job again. Preventing every duplicate execution across all failures is difficult. Preventing duplicate user-visible effects is tractable.

• Use a unique constraint on the idempotency scope and key.
• Use conditional state transitions, such as running to completed only once.
• Store output identity and hash before publishing completion.
• Make cost settlement compare-and-set or transactionally guarded.
• Deduplicate progress events by job, attempt, and sequence number.
• Reconcile provider history before launching another expensive attempt.

Reserve cost before execution if overspending must be prevented. Settle actual usage only once. Release or refund reservations on terminal failure according to a documented policy.

Outputs need the same discipline. A completed provider file should be copied into durable application storage and attached to the logical job. A copy retry should target that file. Regenerating creates a new stochastic result and may charge twice.

queued -> running -> completed
   |         |
   |         +-> retry_wait -> queued
   |         +-> reconciling -> running | completed | failed
   +-> cancelled
   +-> failed

Rules:
- terminal transitions are conditional
- attempts are append-only records
- completion owns one durable output
- settlement happens once

State changes should be logged with safe identifiers and reasons. The log should explain why a retry happened without leaking the user’s private prompt or media URL.

Primary queue and API reliability documentation used for the retry and idempotency model.

• Stripe idempotent requests defines key reuse, stored responses, parameter checks, and retention behavior.
• Stripe advanced error handling distinguishes content, network, and server failures and explains reconciliation risk.
• RQ jobs documents job records, statuses, results, and automatic retries.
• RQ exceptions and retries documents retry configuration and failed-job handling.