The scheduler is not the guarantee
A scheduler can decide when work should begin. It cannot guarantee that the work runs exactly once. Deploys overlap, workers restart, messages are delivered more than once, and two processes can wake up at the same time. Treating the scheduler as a lock makes those normal events become data problems.
I prefer to separate the concerns: scheduling expresses intent, the queue owns delivery, a distributed mutex limits concurrent execution, and the job itself remains safe to retry. Each layer has one job and one failure mode.
Lock narrowly and fail clearly
A Redis mutex keyed to the logical job is often enough to prevent overlapping runs. Give it an explicit lease, record why acquisition failed, and keep the protected region as small as possible. A lock without an expiry can turn one interrupted worker into a permanent outage.
The database still needs its own consistency strategy. Transactions, appropriate row locks, and retry handling protect the state even if the queue or mutex behaves unexpectedly. Infrastructure reduces collisions; domain-level idempotency makes collisions survivable.
Design the second attempt first
Before shipping a background task, ask what happens when it succeeds halfway and then runs again. Stable operation keys, upserts, checkpoints, and append-only event records are more useful than hoping the first attempt completes cleanly.
The goal is not a system that never retries. It is a system where a retry is ordinary, observable, and boring.
def run_job(job_id: str) -> None:
lock = locks.acquire(f"job:{job_id}", expires_in=300)
if not lock:
return
try:
if jobs.was_completed(job_id):
return
result = perform_work(job_id)
jobs.mark_completed(job_id, result)
finally:
lock.release()