Scheduler and background work

Purpose

Work that happens without a request behind it: a reminder SMS an hour before a booking, an outbox drained every minute, a card charged at the start of a billing cycle, a mailbox polled for tickets, a cache warmed before the dashboard asks for it. The invariant that makes it trustworthy is that a timer firing twice, or on two processes at once, or not at all for an hour, never changes what the database ends up saying: exactly one process owns the schedule, and every job is idempotent through a committed column, never through the lock that elected it.

This foundation fixes the runtime shape every backend recipe with a "job" or a "drain" assumes: an in-process APScheduler elected by a Postgres session advisory lock, a standby probe on every other worker, BackgroundTasks for the after-response tail of a request, and a thread for anything that must not block the event loop.

When to use

Include it from any recipe whose Application logic says "every N minutes", "nightly", "retry until delivered", "after the response is sent", or "poll". Trigger phrases: "the drain", "the sweep", "the reaper", "leader-only", "a job that charges", "the bot's long-poll", "warm the cache", "the retention job".

Do not use it for work that must survive the process and be observable by the caller (a run a customer polls for by id): that is agent-runtime/multi-replica-job-routing (B78), which addresses jobs to the node that accepted them. Do not use it for the outbox's data model and backoff table, which is _shared/outbox-pattern (S11); this recipe covers the timer that calls the outbox's drain_once, not the outbox. Do not use it for a one-shot host timer (a nightly backup, a collection run driven by curl from a systemd unit); those are _shared/cloudflare-worker-and-estate-deploy (S13) and belong to the host, not the app.

Stack assumptions

Pinned as measured in alchemy_labs/backend/requirements.txt at fd8558f (2026-09-11): APScheduler==3.11.0, imported lazily inside services/scheduled_jobs.py so a missing package degrades to a logged warning rather than a boot failure. The scheduler is BackgroundScheduler() with no arguments, which means the default MemoryJobStore and a thread-pool executor; there is no apscheduler_jobs table anywhere in the schema (Argus checks/apps.py lines 1096 to 1105 at e42e467 says the same and reads the container log instead). APScheduler's own job defaults at that version, read from schedulers/base.py line 908 of the installed package: misfire_grace_time=1, coalesce=True, max_instances=1; the flags this repo writes out explicitly on newer jobs are the defaults made visible, not a change.

Process model: the Labs image runs uvicorn ... --workers ${WEB_CONCURRENCY:-2} (Dockerfile line 117, default WEB_CONCURRENCY=2 at line 77), so init_scheduler() is called once per worker from the FastAPI lifespan (app_fastapi.py lines 318 to 323) and shutdown_scheduler() from the same function's exit (lines 353 to 358). Everything below exists because of that: N workers, one schedule.

Foundations assumed and declared in depends_on: _shared/fastapi-project-layout (S4) for where services/ and the lifespan live, _shared/alembic-migrations (S8) for the columns a job stamps, _shared/workspace-scoping (S6) because every job that touches tenant rows filters by workspace inside the job body. _shared/outbox-pattern (S11) is a cross-reference, not a dependency: it declares this recipe (its line 9), so listing it back would make a cycle, and this recipe covers the timer that calls a drain, not the outbox.

Environment names the scheduler reads, with fail behaviour:

Variable	Read by	Default	When missing or wrong
ENVIRONMENT	_acquire_scheduler_singleton_lock (through os.environ, line 105, not through settings)	development	in production a lock-check error starts no scheduler (fail closed); elsewhere it starts one unguarded (fail open)
MONITORING_ENABLED, MONITORING_CHECK_INTERVAL_MINUTES	job registration (line 267) and the job body	True, 5	the monitoring job is not registered
SCRIBE_WARM_ENABLED, SCRIBE_WARM_INTERVAL_MINUTES	job registration (line 404) and the job body	True, 60	the warm job is not registered
PRIMA_KEY_SYNC_MAX_ATTEMPTS, PRIMA_CREDIT_GRANT_MAX_ATTEMPTS	the outbox drains	24, 24	after that many attempts a row becomes stuck, never deleted
ACCOUNTING_TELEGRAM_BOT_TOKEN, DESK_TELEGRAM_BOT_TOKEN (SecretStr)	the two bots' start_polling	unset	the bot logs one line and does not start; nothing else changes
TRACKING_EVENT_RETENTION_DAYS	the daily retention sub-job	365	the default applies
SCHEDULER_ROLE	nothing (see Gotchas)		ignored

Data model

The scheduler stores nothing. Job identity is the id= string passed to add_job and lives in process memory; a restart re-registers every job from code (replace_existing=True on every call) and the first run of an interval job is one interval after boot unless next_run_time is set (only scribe_report_warm sets it, 30 seconds after boot, line 413).

What jobs do store, and what makes them idempotent, is one committed column per job kind:

Job kind	The column that makes a second run a no-op	Where
booking reminders	calendar_bookings.sms_reminder_24h_sent_at / sms_reminder_1h_sent_at IS NULL in the select; stamped and committed per send	run_booking_reminders, lines 1193 to 1262
standing orders	next_due_date advanced and committed as each entry is generated, so get_due_templates no longer returns it	run_accounting_standing_orders, lines 1583 to 1651
outbox drains (key sync, credit grants, Sparks grants)	rows selected FOR UPDATE SKIP LOCKED; attempts, next_attempt_at, status; the receiver's UNIQUE (provider_event_id) or sync_seq predicate. The status vocabulary per outbox, measured 2026-09-11: pending, granted, stuck for credit grants (prima_credit_grants.py lines 318, 338, 375) and Sparks grants (sparks_grants.py lines 391, 406, 433); pending, synced, stuck for key syncs (prima_key_sync.py lines 413, 433, 465). No outbox writes delivered; a console filtering on it shows nothing	prima_credit_grants.py lines 267 to 428, prima_key_sync.py
subscription charging	next_charge_at advanced and committed before the PSP call (claim before send)	subscription_service._convert_one, lines 311 to 407
auto top-up retry (w1 branch)	pending_reconcile_ref plus last_attempt_at written under the row lock before the send; a marker another runner resolved is answered resolved_elsewhere	auto_topup_service.charge_if_due, 8b5361c
Meta CAPI backfill	the receiver dedups by event_id; the job stores nothing	run_meta_capi_backfill_hourly, line 599
contact linking	contact_id IS NULL filter, batched 100 rows, at most 500 batches per run (lines 840 to 843)	run_contact_linking_job, line 875

The Postgres advisory keys this codebase owns, arbitrary and fixed:

Key	Owner	Meaning
911_000_001 (session)	scheduled_jobs.SCHEDULER_SINGLETON_LOCK_KEY (line 56)	"this process owns the schedule"
911_000_003 (transaction, with a per-tenant second key)	prima/runtime/spend.py ADMISSION_LOCK_NAMESPACE (line 158, cd2c248); the same constant at backend/services/primalayered/spend.py line 160 in Labs is the legacy in-Labs copy (decision D3); the shared number is harmless because Labs and Prima run against separate databases (alchemycloud/scripts/distribute-env.sh lines 366 and 586, b3d34da) and advisory locks are per database	serialises job admission per tenant; its docstring says why it is deliberately not 911_000_001 and leaves 911_000_002 free for a future drain loop
8110731 (session)	scripts/migrate.sh MIGRATION_LOCK_ID	serialises alembic upgrade head; listed here so nobody reuses the number

NOT stored: run history, last-success timestamps, a job table. The only observable signal that a job ran is APScheduler's executor log line Running job "<name> (trigger: ..., next run at: ...)" (scheduled at ...) (apscheduler/executors/base.py line 129 of the installed package), which is why the name= string is an interface (see Deploy & ops).

API surface

There is no HTTP surface for the scheduler itself at fd8558f; jobs are triggered by time and by the two bots' long-polls. Routes that hand work to BackgroundTasks after the response (a sample of the 19 live add_task call sites in 10 files, measured 2026-09-11):

Route	What is deferred	Why deferred
POST /api/auth/register, /login, resend-verification (auth_router.py lines 129, 329, 341, 892, 946, 1110)	email_service.send_verification_email and its siblings	a mail-transport error must not turn a login into a 500; the response is already decided
POST /api/billing/webhook (billing_router.py line 743)	stripe_service.process_webhook_event	acknowledge the PSP inside its timeout, process after
POST /api/prima/v1/accounts (router prefix at primaapi_router.py line 48, decorator at 636, add_task at 682; the legacy in-Labs copy per decision D3)	the signup confirmation mail	same
POST /api/scribe/event (router prefix at tracking_router.py line 260, handler at 538, add_task at 602 and 634; the /api/track legacy_router at line 263 is mounted at app_fastapi.py line 952 and registers zero routes at fd8558f, so there is no legacy alias for this path)	GA4 forward, Meta CAPI page view	the config is read in request scope and the detached values handed to the task; the task does no DB access
file upload (enhanced_upload_router.py lines 152, 179, 287)	thumbnail generation, upload analytics	the bytes are stored; the derived artefact can lag

Contract for every deferred task: the row the request promised is committed before add_task is called, the task carries plain values (an email address, a token, a detached config object), never an ORM instance bound to the request's session, and its failure is logged, never raised. BackgroundTasks runs in the same worker after the response is flushed; it is not durable (a SIGTERM between response and task loses it), so anything money-bearing or user-visible on failure goes through an outbox row and the scheduled drain instead (prima_credit_grants.py header, lines 1 to 40: inline attempt after commit, drain as the durability half).

Sync code called from async handlers goes through a thread: Labs uses Starlette's run_in_threadpool (tracking_router.py line 620), Prima uses asyncio.to_thread for the meter write (prima/runtime/invoker.py line 227) and for the heartbeat (prima/runtime/jobs/execution.py line 221). A sync SQLAlchemy call directly inside an async def blocks the loop for every other request on that worker.

Application logic

1. Boot: lifespan calls init_db() (which runs verify_schema_at_head(), core/database.py line 327) and then init_scheduler() inside a try that logs and continues; a scheduler failure never blocks the API.
2. _acquire_scheduler_singleton_lock() opens one dedicated connection from the engine, runs SELECT pg_try_advisory_lock(911000001), and on True keeps that connection open in module state (_scheduler_lock_connection, line 62) for the life of the process (line 89); on False closes it and returns. The lock is session-level on purpose: it is released when the connection drops, which is what happens when the process dies. It is also released when the server drops the connection while the process lives (a Postgres restart, a failover, idle_in_transaction_session_timeout), and the reference never notices (see Gotchas). A new app does three more things here: it puts the lock connection in AUTOCOMMIT (engine.connect().execution_options(isolation_level='AUTOCOMMIT')) so the holder sits idle rather than idle in transaction and no idle-transaction timeout can reap it; it runs SET application_name = 'scheduler-leader:<os pid>' on that connection before the lock statement, so pg_stat_activity.application_name names the holder's process (behind a docker proxy client_port is the proxy's, and the holder cannot otherwise be found from psql; the 2026-09-12 rebuild used it for the leader tick and for check 5); and it keeps the lock connection's pg_backend_pid() so the leader tick in step 4 can verify ownership.
3. Leader: _start_full_scheduler() builds one BackgroundScheduler, registers the job table (step 6), calls scheduler.start() (line 461), then starts the two Telegram bots' daemon threads (lines 467 and 475). An ImportError or any other exception releases the lock so another worker can take over.
4. Standby: _start_singleton_probe() builds a second, tiny BackgroundScheduler with one job, _singleton_probe_tick every 60 seconds (id='scheduler_singleton_probe', name='Scheduler singleton failover probe', replace_existing=True, coalesce=True, max_instances=1, lines 165 to 173), which retries step 2 and, on success, runs step 3 in this worker. The two schedulers are two module globals, scheduler (the leader's, set in _start_full_scheduler) and _probe_scheduler (the standby's, guarded by if _probe_scheduler is not None: return at line 158); the tick returns early while scheduler is not None (line 184), so the reference runs no ownership check on the leader. Failover is therefore at most one minute after the leader's connection drops. A new app runs the probe on the leader too, as a leader tick: every 15 seconds (not 60, see below) it runs SELECT 1 FROM pg_locks WHERE locktype = 'advisory' AND objid = 911000001 AND database = (SELECT oid FROM pg_database WHERE datname = current_database()) AND pid = pg_backend_pid() on the lock connection (a terminated connection raises here, which counts as loss); on loss it calls scheduler.shutdown(wait=False), sets scheduler = None, closes the dead lock connection and re-enters standby. pg_locks is cluster-wide and an advisory key is per database, so every query against it carries the database predicate above: a leader in another database on the same server (the Labs test database beside the harness database on :5434, or the app database beside the test database) otherwise makes the count 2 and the demotion test fail (measured 2026-09-12; the pid predicate on the tick already scopes it, since a backend belongs to one database, and the filter is kept there so the query has one shape everywhere). The order in which the loser stops and the winner starts is not free either: with the leader tick and the standby probe both at 60 s it is a race, and after pg_terminate_backend the standby logged failover: acquired before the old leader's tick logged the demotion (measured 2026-09-12). A new app therefore runs the leader tick at 15 s against the probe's 60 s and, on acquiring in the probe, waits one leader-tick interval before scheduler.start(), so the old leader has ticked and shut down before the new one fires a job; the invariant to assert is that the (database-filtered) holder count never exceeds 1 and no job fires twice within one interval, not the order of two log lines.
5. Shutdown: shutdown_scheduler() stops both schedulers with wait=False, then pg_advisory_unlock and closes the lock connection (lines 489 to 504). The bots' threads are daemon threads and die with the process; nothing calls stop_polling.
6. The job table registered by the leader at fd8558f (21 registrations, 19 unconditional and 2 behind settings; the w1 branch adds a 22nd):

id	name (the log string Argus matches on)	Trigger	Flags
booking_reminders	Send booking reminder SMS	every 15 min	defaults
message_drain	Send due scheduled messages	every 5 min	defaults
hourly_jobs	Hourly maintenance jobs	every 1 h	defaults
daily_jobs	Daily maintenance jobs	cron 06:00	defaults
deploy_window_recompute	Recompute deploy-window activity heatmap	cron 04:17	defaults
outreach_queue	Process outreach email queue	every 15 min	defaults
outreach_reply_detection	Detect outreach email replies	every 1 h	defaults
monitoring_checks	System monitoring checks	every MONITORING_CHECK_INTERVAL_MINUTES	only if MONITORING_ENABLED
accounting_gmail_sync	Accounting Gmail receipt sync	every 15 min	defaults
accounting_drive_sync	Accounting Drive document sync	every 10 min	defaults
accounting_overdue_check	Check overdue invoices	cron 07:00	defaults
accounting_standing_orders	Generate standing order entries	cron 06:00	defaults
prima_key_sync_drain	Deliver queued Prima API key syncs	every 1 min	coalesce, max_instances=1
prima_credit_grant_drain	Deliver queued Prima credit top-ups	every 1 min	same
sparks_grant_drain	Deliver queued Sparks grants	every 1 min	same
meta_capi_backfill	Forward scribe events to Meta CAPI (ad-block recovery)	every 1 h	same
contact_linking_job	Contact ID backfill linking job	every 1 h	same
scribe_report_warm	Warm Scribe dashboard report cache	every SCRIBE_WARM_INTERVAL_MINUTES, first run 30 s after boot	only if SCRIBE_WARM_ENABLED
commerce_subscription_conversion	Commerce subscription token conversion/charging	every 15 min	coalesce, max_instances=1
desk_reminders	Service Desk reminders (unaccepted / due soon / overdue / review timeout)	every 5 min	same
desk_email_intake	Service Desk inbound email intake (IMAP → tickets)	every 5 min	same; the arrow is in the string (line 454), so a monitor matches it verbatim
prima_auto_topup_retry (w1 8b5361c only)	Retry pending Prima auto top-ups	every 5 min	same; its own timer because it talks to a PSP with a 30 s read timeout

7. Every job body has the same shape (run_message_drain, lines 572 to 585, is the smallest): db = get_db_session() (a fresh session from api.dependencies.get_db, line 588), a lazy import of the service, the call, a log line only when something happened, except Exception logged with exc_info, finally: db.close(). A job never raises into APScheduler; a raised exception would be logged by the executor and the next tick still fires, but the session would leak.
8. Async service code inside a sync job gets its own loop: run_monitoring_checks creates asyncio.new_event_loop(), runs run_all_checks(db) to completion and closes it (lines 1670 to 1676). A job never touches uvicorn's loop.
9. Drains claim rows FOR UPDATE SKIP LOCKED in batches of 50 (prima_credit_grants.drain_once, line 410), attempt delivery, and on failure bump attempts and set next_attempt_at from the backoff table (5, 15, 60, 300, 900, 3600) seconds (line 82); a 4xx from the receiver other than 408 and 429 (deliver, line 332: if 400 <= resp.status_code < 500 and resp.status_code not in (408, 429)) or attempts >= MAX_ATTEMPTS (_fail, lines 364 to 367) sets status='stuck'; 408 and 429 retry like a 5xx. Nothing is deleted. The drain commits per row, so one row's unexpected exception does not return its neighbours to pending (docstring, lines 385 to 392).
10. Money-bearing periodic work claims before it sends: _convert_one locks the subscription row with_for_update(), advances next_charge_at to the next cycle, commits, and only then charges (lines 339 to 346); a crash after the claim loses one charge, a crash before it charges nothing, and neither double-bills. The w1 auto top-up job adds a reference to the claim (pending_reconcile_ref) so a time-based claim cannot lapse and re-fire (auto_topup_service.py header, item 2, lines 20 to 53).
11. Prima (cd2c248) has no APScheduler. Its periodic work is two things in create_app's lifespan (prima/app.py lines 289 to 320): a startup reap of this node's own abandoned budget rows (_reap_this_nodes_abandoned_jobs, keyed by PRIMA_NODE_ID, fails open with a warning) and an asyncio task _credit_sweep_loop that every 300 seconds refunds credit reservations older than STALE_RESERVATION_AFTER (15 minutes, credits.py line 149); it sleeps before its first pass and is cancelled on shutdown. Two daemon threads (prima-jwks-refresh, jwks.py line 318; prima-revocation-poll, wiring.py line 274) refresh the auth material. None of it takes a lock: every replica sweeps, and the sweep is idempotent because a refund of an already-settled reservation matches zero rows.

Invariants

• At most one process holds 911_000_001 at a time, and a process runs scheduler.start() and the Telegram long-polls only while its lock connection lives. In the reference the second half holds only until the server drops the lock connection (Gotchas, 2026-09-11, idle in transaction); a new app makes it hold by the leader tick in step 4, so a leader whose lock is gone demotes within 15 seconds, before a standby that acquired the lock starts a job. Tested by test_init_scheduler_single_lock_holder_under_two_workers (two real connections) and, for a new app, by the terminated-holder test under Tests.
• A lock-check error in production starts no scheduler. ENVIRONMENT=production is read from the process environment, not from settings, at line 105.
• A job's idempotency is a committed column or a receiver-side UNIQUE, never the singleton lock and never a transaction-level advisory lock: pg_try_advisory_xact_lock is released at the first commit() inside a job that commits per row, which is every job here. Pinned by test_xact_advisory_lock_is_released_by_a_commit and test_no_per_job_xact_lock_helper_exists.
• A drain is safe to run from two processes at once (SKIP LOCKED plus the receiver's UNIQUE); coalesce and max_instances on it are about log noise, not correctness (the comments on lines 320, 335 and 354 say exactly this).
• A money job claims before it sends, under a row lock, and commits the claim.
• A job never raises and always closes the session it opened.
• BackgroundTasks is for the tail of a request that may be lost; anything that must eventually happen is an outbox row plus a scheduled drain.
• One getUpdates consumer per Telegram bot token, started on the leader only (service_desk/telegram_bot.py lines 1 to 8). A second consumer on the same token makes both bots drop updates.
• A job's name= string is read by the monitor; it changes only together with argus/checks/apps.py SCHEDULER_JOBS.
• No new advisory key without adding it to the table in Data model; a reused key silently disables whichever side loses the race.

Frontend

The scheduler has no page of its own at fd8558f. The surfaces that render its effects, by inventory id:

• billing-credits/ui/billing-outbox-admin-console (F30, planned): one table per outbox (prima_key_syncs, prima_credit_grants, sparks_grants, prima_auto_topups) with status, attempts, next_attempt_at, the stuck rows and a reset action (status='pending', attempts=0, next_attempt_at=now()), plus a drain heartbeat. Today the stuck_grants and stuck_syncs queries exist in the services and no router calls them (measured 2026-09-11: zero hits for stuck_ under backend/api/routers/), so the admin reads them with SQL in the container.
• monitoring-ops/ui/admin-dashboard-and-system-logs (F85, SPA admin): the monitoring job's results and the Meta CAPI backfill trigger.
• monitoring-ops/ui/argus-telegram-surface (F86, Telegram; the inventory slug at operations/prima-modules-inventory.md line 528, d4f5b95): apps.scheduler_jobs grades the age of each job's last Running job line against its interval plus a grace of one interval, floored at ten minutes, for the seven jobs listed in SCHEDULER_JOBS; a stopped SMS sender shows there before a customer notices.
• infra-deploy/ui/deploy-and-ops-cli-surfaces (F88, CLI): docker logs <container> | grep 'Running job' is the operator's view of the schedule.

Minimum admin surface a new app ships with this foundation: a read-only table of registered jobs (scheduler.get_jobs(): id, name, trigger, next run) behind the admin dependency, and the outbox table above. Both are missing from the reference; F30 is the recipe that adds them.

Security & tenancy

The scheduler runs as the application user with the application's database credentials; there is no separate worker identity. Every job body that touches tenant rows scopes by workspace inside the query (_link_direct_workspace_table, poll_all per workspace with a connected mailbox, run_accounting_standing_orders per fiscal year of a workspace), because there is no request and therefore no X-Workspace-ID to derive from. A job that iterates "all workspaces" is the one place in the codebase where cross-tenant iteration is legitimate, and it commits per workspace so one tenant's failure does not roll back another's.

Fail direction: the singleton election fails closed in production and open elsewhere (Stack assumptions). Drains fail toward pending and then stuck, never toward delete. Money jobs fail toward "charged once or not at all", never "charged twice". The Prima meter write from a job fails open (invoker.py line 227, the run continues); admission fails closed (spend.py).

Credentials the jobs hold: the two bot tokens (SecretStr, read inside start_polling, never logged), per-workspace IMAP and OAuth material read through the workspace settings service and decrypted per poll, PSP credentials from settings. Nothing job-related is written to disk; the LinkedIn reminder backfill that kept dedup state in /tmp was disabled for exactly that reason (app_fastapi.py lines 326 to 334: /tmp is wiped per deploy, so it re-fired on every boot).

Attacker-controlled inputs reaching a job: inbound email bodies (desk intake, deduped by Message-ID, rendered through the desk's sanitiser), Telegram update payloads (routed by command, the pairing code is the only state written), PSP webhook bodies handed to BackgroundTasks after signature verification in the request. Rate limits: the Telegram loops sleep 5 seconds after any non-ok answer or transport error (telegram_bot.py lines 245 to 258); drains are capped at 50 rows per tick; contact linking at 500 batches per run.

Tests

backend/tests/v2/test_scheduler_singleton.py (496 lines) is the file a new app copies; it needs the real Postgres on localhost:5434 (_shared/test-layout, S9) because advisory locks are a Postgres feature:

• Gate A: test_singleton_lock_granted_when_uncontended, test_guarded_job_runs_body_when_uncontended (the lock does not amputate the job).
• Gate B, residual: test_xact_advisory_lock_is_released_by_a_commit (the root-cause fact), test_no_per_job_xact_lock_helper_exists (nobody re-adds the false guard), test_init_scheduler_single_lock_holder_under_two_workers, test_singleton_lock_released_allows_new_holder.
• Gate C: test_standing_orders_durable_idempotency_via_next_due_date drives the real job twice through the real database and asserts the second run posts nothing; test_two_init_scheduler_calls_start_scheduler_once fakes the apscheduler modules and counts instantiations.

Measured 2026-09-11 at fd8558f against the running postgres-test container on 5434 (python -m pytest tests/v2/test_scheduler_singleton.py -q --no-cov): 1 failed, 7 passed in 5.7 s (the default run with coverage takes about 25 s). The failure is test_two_init_scheduler_calls_start_scheduler_once (line 432, assert 2 == 1 on built): it asserts one BackgroundScheduler instantiation across two init_scheduler() calls, and the standby worker's probe scheduler is a second instantiation by design. The test predates the probe (see Gotchas). Counting start() instead does not fix it: _FakeScheduler.start (line 385) counts every instance and the probe calls _probe_scheduler.start() (line 174), so started is 2 as well (measured 2026-09-11, a copy of the test with only the built assertion removed fails assert 2 == 1 on started). The fix that passes, measured the same day: give the fake an add_job counter, count start() only on the instance that received more than one job (the probe receives exactly one), and assert built == 2 and that count == 1; with that change the copied test passes and the file would be 8 passed.

A new app adds one test the reference lacks, test_leader_demotes_when_lock_connection_is_terminated: acquire the lock through init_scheduler(), read the lock connection's pg_backend_pid(), run SELECT pg_terminate_backend(<pid>) from a second connection, drive the leader tick once, and assert scheduler is None, that pg_locks has no row for 911000001 in the current database (SELECT count(*) FROM pg_locks WHERE locktype = 'advisory' AND objid = 911000001 AND database = (SELECT oid FROM pg_database WHERE datname = current_database())), and that a second _acquire_scheduler_singleton_lock() on a fresh connection returns True. The measurement behind it is in Gotchas.

Known-answer vectors a job recipe adds: the backoff table as a list of seconds, the exact SKIP LOCKED select as SQL in the test, a fake clock (services/service_desk/clock.py is the desk's) so "due" is a fixture and not utcnow(). Hermetic: fake the receiver (the drains take a session and a transport); live: none, no job test may reach a PSP or Telegram. Playwright gate: not applicable to the scheduler; the outbox admin console (F30) gets one when it exists.

Outbox-level tests that already exist and a recipe cites rather than copies: tests/test_contact_linking_job.py, tests/test_contact_cross_tenant.py, tests/test_contact_linking_derived_ws.py (the hourly linker never crosses a workspace).

Deploy & ops

Config keys the Labs env heredoc actually writes (alchemycloud/scripts/distribute-env.sh at b3d34da, the labs block between the EOF markers at lines 360 and 550, opened 2026-09-11): ENVIRONMENT=${LABS_ENVIRONMENT} (line 376), which is what turns the fail-closed branch on when it renders production; TELEGRAM_BOT_TOKEN (line 465); the PRIMA_* seam keys the drains post with (PRIMA_KEY_SYNC_ISSUER, PRIMA_KEY_SYNC_SECRET, PRIMA_INTERNAL_BASE_URL, PRIMA_CREDIT_GRANT_SECRET, PRIMA_CREDITS_MERCHANT_WORKSPACE_ID, PRIMA_CREDITS_TOPUP_URL, lines 484 to 510). It does not write ACCOUNTING_TELEGRAM_BOT_TOKEN, DESK_TELEGRAM_BOT_TOKEN or MONITORING_ENABLED (see Gotchas), so on a redistributed host the two bots are off and monitoring runs on its True default. SCHEDULER_ROLE appears in deploy/infomaniak/docker-compose.infomaniak.yml (the migrate service sets it to migrate, the backend to worker) and is read by nothing; the advisory lock made it unnecessary before it was wired. A migration container never starts a scheduler because migrate.sh runs alembic, not the app.

Migrations order: none for the scheduler itself; each job's stamp column arrives with its domain recipe's migration, and verify_schema_at_head() at boot guarantees the column exists before the first tick.

Restart behaviour: docker compose up -d --force-recreate kills the leader, its lock connection drops, and the new process's first worker wins the election at boot; a standby elsewhere would take over within 60 seconds. Interval jobs restart their countdown at boot, so a 1-hour job deployed twice in an hour runs once in two hours (the warm job's next_run_time is the exception). Cron jobs (06:00, 04:17, 07:00) fire at wall-clock time in the container's timezone.

Argus checks (argus e42e467): apps.scheduler_jobs ages the seven named jobs against docker logs -t (the log window is 26 hours, which is why the daily job's grace is set explicitly to 3600 s and not a second day); scheduler.expected_jobs and scheduler.clock_skew cover host timers and clocks, not APScheduler. A job renamed in add_job(name=...) becomes a phantom CRIT there (the 2026-08-12 fix commit is the branch head).

Rollback: the scheduler has no state to roll back. Rolling an image back while a newer migration stays applied is the general rule from S8 and S13 (the old image refuses to boot off head).

Gotchas

• 2026-09-11 (measured at fd8558f, deploy/infomaniak/docker-compose.infomaniak.yml lines 41 and 61 to 67): the compose template says the scheduler must run in one process and requires "the backend change (guard init_scheduler() behind SCHEDULER_ROLE)"; no file under backend/ reads SCHEDULER_ROLE. The advisory lock is the mechanism that ships. The compose comment describes a plan that was superseded; the wiki's platform/alchemy-cloud.md line 218 (d4f5b95, "web pods x N + one singleton scheduler pod") describes the same plan, and so do deploy/infomaniak/k8s/deployment-scheduler.yaml (a 1-replica scheduler Deployment, SCHEDULER_ROLE at lines 7 and 34), k8s/deployment-web.yaml line 40 and k8s/job-migrate.yaml line 55, none of which is deployed. Code wins: N workers, one lock, no scheduler pod; do not copy the manifests.
• 2026-09-11 (measured at fd8558f): the wiki inventory row S12 says "the 20-job table"; _start_full_scheduler registers 21 jobs (19 unconditional, monitoring_checks and scribe_report_warm conditional), and the w1 branch 8b5361c registers 22. The table above is the measured one.
• 2026-09-11 (measured, tests/v2/test_scheduler_singleton.py line 369): test_two_init_scheduler_calls_start_scheduler_once fails, 2 constructions where it expects 1. The standby probe is a BackgroundScheduler too. The test's docstring (line 372) says "the second returns early", and the _acquire_scheduler_singleton_lock docstring (scheduled_jobs.py line 72) says a second worker "simply does not start the scheduler"; both were written before the probe. Fix the assertion, not the probe, and not by counting start() (Tests says which assertion passes).
• 2026-09-11 (measured, scheduled_jobs.py line 105): the fail-closed switch reads os.environ["ENVIRONMENT"], while every other production check in the codebase reads settings.ENVIRONMENT. A .env file loaded only by pydantic-settings, not exported into the process, leaves the lock check fail-open. In compose env_file exports, so it holds; in a bare uvicorn on a laptop it does not. A new app reads settings.ENVIRONMENT.
• 2026-09-11 (measured at fd8558f, scheduled_jobs.py lines 78 to 89 and 182 to 188, against the test Postgres on 5434 with SQLAlchemy 2.0.43, the repo's own pin): the leader's lock connection runs a bare SELECT pg_try_advisory_lock with no commit, so pg_stat_activity.state for it reads idle in transaction for the whole process lifetime. SELECT pg_terminate_backend(<that pid>) from another connection empties pg_locks of 911000001, a second connection's pg_try_advisory_lock(911000001) then returns true, and the old holder learns nothing until its next use of the connection raises OperationalError, which the reference never triggers because _singleton_probe_tick returns early when scheduler is not None. A Postgres restart, a failover or a managed-Postgres idle_in_transaction_session_timeout therefore promotes a standby within 60 s while the old leader keeps firing, and every job double-fires, the incident this foundation exists to prevent. Measured the same day: the same acquire on a connection with execution_options(isolation_level='AUTOCOMMIT') reads idle, and SELECT 1 FROM pg_locks WHERE locktype = 'advisory' AND objid = 911000001 AND pid = pg_backend_pid() on it returns 1. Application logic steps 2 and 4 say what a new app does; not fixed in the reference.
• 2026-09-11 (opened at alchemycloud b3d34da, scripts/distribute-env.sh): the Labs heredoc writes TELEGRAM_BOT_TOKEN (line 465) and nothing named ACCOUNTING_TELEGRAM_BOT_TOKEN, DESK_TELEGRAM_BOT_TOKEN or MONITORING_ENABLED (a grep for the three prints nothing in the script; env/.env.example line 178 lists the accounting token as a master variable the heredoc never renders). The script's own comment at lines 368 to 373 says whatever it writes is what the app gets, so a fresh redistribute leaves both long-poll bots off with one log line each and monitoring on its default. A host where they run has them by hand-edit. Add the two tokens to the heredoc before relying on either bot; not fixed in the script.
• 2026-09-11 (measured, scheduled_jobs.py lines 1431 to 1491): schedule_post_deploy_checks() registers three DateTrigger jobs at +5 s, +5 min and +60 min after a deploy and has no caller outside its own file and the tests. The post-deploy checks that do run are scripts/postdeploy_smoke.sh (S13). Delete or wire it; do not copy it.
• 2026-09-11 (measured, scheduled_jobs.py lines 1736 to 1772 and 1816 to 1843): run_hourly_jobs calls two functions whose bodies are pass with a TODO; run_daily_jobs calls cleanup_expired_sessions and send_admin_digest, both pass. The Argus token_maintenance and daily_cleanup rows age against these batch names, so the batches must keep running even while half their contents are stubs. A new app registers one job per real piece of work and no batches; the inventory's instruction "stubs to delete" stands.
• 2026-09-11 (measured, scheduled_jobs.py lines 1956 to 1978): ScheduledJobService is a legacy wrapper whose six methods all take a db argument; run_hourly_jobs and run_daily_jobs ignore it, the other four pass it through. scheduled_job_service is the module-level instance. Do not copy.
• 2026-09-11 (measured, scheduled_jobs.py lines 49 to 55 and the test file header): the first version of the singleton also wrapped job bodies in pg_try_advisory_xact_lock and called it per-job idempotency. It was false, because every job commits per row and the xact lock is released at the first commit. The code and the test that pinned the false guarantee were both removed; the reasoning is kept in the comment so nobody re-adds it.
• 2026-09-11 (measured, scheduled_jobs.py lines 345 to 366): sparks_grants.drain_once existed with an inline attempt and nothing scheduled ever called it, so a failed grant stayed pending forever while the module docstring and a seed script claimed the drain would pick it up. An outbox without a registered drain is a bug that looks like a slow network. Every S11 recipe names its drain job id.
• 2026-09-11 (measured at cd2c248, prima/app.py lines 270 to 278): _credit_sweep_loop awaits asyncio.sleep and then calls the synchronous _sweep_stale_reservations_once() directly on the event loop; the sweep's Postgres round trip blocks every request on that replica for its duration, every five minutes. The heartbeat in the same repo (execution.py line 221) does it right with asyncio.to_thread. A new app hands sync DB work from an async loop to a thread.
• 2026-09-11 (measured at cd2c248): three docstrings (app.py line 208, execution.py line 17, spend.py line 1118) call reap_abandoned_jobs(older_than_s=3600) "the hourly reaper" and "the backstop"; nothing in prima/, scripts/ or tests/ calls it. The startup reap by node id is the only reaper that runs. Either schedule it (a per-replica asyncio task is enough, the update is idempotent) or stop calling it a backstop.
• 2026-09-11 (measured at 8b5361c, auto_topup_service.py header item 2, and the W1-fix commits 39c5d61, bcd8574, 6d7d1e5, 60f6175): a claim made only of timestamps expires at the cooldown, and a retry runner that raced the first found a stale last_attempt_at and charged the same card again (verified HIGH, decision D8 list). The claim now carries the Order reference and the amount it asked for; a marker whose claim is younger than the cooldown is another runner's charge in flight. The branch is unmerged; the reference for "claim before send" on the shipping line is subscription_service._convert_one.
• 2026-09-11 (measured at e42e467, argus/checks/apps.py lines 1096 to 1105): Labs runs a MemoryJobStore, so the monitor's only signal is the Running job "<name> log line, keyed on the human name=; the 2026-08-12 phantom-CRIT incident was a name mismatch. apps.scheduler_jobs matches key and match both, so a future database jobstore keyed on id needs no monitor change.
• 2026-09-11 (measured, app_fastapi.py lines 326 to 334): the LinkedIn reminder startup backfill is commented out because its dedup state lived in /tmp, wiped per deploy, so every boot re-sent the reminders to the Telegram group. State a job needs across restarts lives in a table.
• 2026-09-11 (measured, scheduled_jobs.py lines 398 to 414): the Scribe warm job exists because the dashboards polled /stats and /insights every 60 s and each cache miss ran a multi-second aggregation on the request path, exhausting the connection pool. Moving that cost to one serialised run per hour is the fix; the lesson is that a polled read endpoint with an expensive query is a scheduler job, not a cache.

Reference implementation

Branch that ships: alchemy_labs feat/service-desk; the d1a-token-source worktree at fd8558f (2026-09-11) is that line plus two frontend-only commits, so its backend tree is the shipping backend (decision D7 asks for a fast-forward before wave 0). Paths are relative to each repo's root.

• backend/services/scheduled_jobs.py (1978 lines): the key and its rationale (29 to 56), _acquire_scheduler_singleton_lock (65 to 117, the production branch at 101 to 108), _release_scheduler_singleton_lock (119 to 137), init_scheduler (140 to 152), _start_singleton_probe (155 to 179), _singleton_probe_tick (182 to 188), _start_full_scheduler (191 to 486; add_job calls 202 to 458, start() 461, bots 464 to 480), shutdown_scheduler (489 to 504), get_db_session (588 to 596), the drains (631 to 743), run_booking_reminders (1193 to 1262), schedule_post_deploy_checks (1431 to 1491, uncalled), run_hourly_jobs and run_daily_jobs (1736 to 1772), the legacy class (1956 to 1978).
• backend/app_fastapi.py lines 269 to 358: the lifespan; init_scheduler() at 320, shutdown_scheduler() at 356, the disabled backfill at 326 to 334.
• backend/services/commerce/prima_credit_grants.py: the outbox header (1 to 40), _BACKOFF_SECONDS (82), deliver with the 4xx stuck transition (267 to 355), _fail with the max-attempts transition (356 to 383), drain_once with SKIP LOCKED (384 to 428), stuck_grants (463).
• backend/services/primaapi/prima_key_sync.py: the same shape with sync_seq (45 to 48) and a shorter backoff (5, 15, 60, 180, 600) (102).
• backend/services/commerce/subscription_service.py lines 311 to 407: _convert_one, claim before send (the claim itself at 339 to 346).
• backend/services/commerce/auto_topup_service.py (w1 8b5361c): the header contract (1 to 90), charge_if_due (576 onward). Deviation from this recipe: unmerged, and its double-charge history is the reason the recipe insists on a claim with a reference.
• backend/services/service_desk/telegram_bot.py lines 1 to 8 and 215 to 258, backend/services/accounting/telegram_bot_service.py lines 50 to 90: the two long-polls, daemon threads, offset handling, the 5-second sleep on error.
• backend/api/routers/tracking_router.py lines 595 to 640: BackgroundTasks with detached config and run_in_threadpool.
• backend/tests/v2/test_scheduler_singleton.py: the eight tests listed under Tests.
• prima/prima/app.py lines 196 to 320 (cd2c248): the startup reap, the sweep loop, the lifespan that owns and cancels it. prima/prima/runtime/jobs/execution.py lines 170 to 230: the heartbeat through asyncio.to_thread. prima/prima/runtime/spend.py lines 155 to 180: the admission lock namespace and why it is not the scheduler's key.
• argus/argus/checks/apps.py lines 1077 to 1141 (e42e467): SchedulerJob, SCHEDULER_JOBS, the match strings.
• Deviations reference versus recipe: the reference has no jobs table and no admin view (F30 planned); two batch jobs contain stubs; one uncalled post-deploy scheduler; the ENVIRONMENT read through os.environ; the lock connection left idle in transaction and no leader-side ownership check; the deploy script that does not write the two bot tokens; Prima's blocking sweep and uncalled reaper. Each is dated above.

Acceptance checks

1. grep -n "pg_try_advisory_lock\|911_000_001" backend/services/scheduled_jobs.py prints the key at line 56 and the SELECT at line 81, and grep -rn "911_000_001\|911000001" backend --include='*.py' | grep -v tests/ prints five lines of which exactly one is an assignment (scheduled_jobs.py line 56; the other four are docstrings in primalayered/spend.py and primalayered/runner.py). In the prima repo the same grep prints only docstring lines (runtime/spend.py, runtime/runner.py); grep -rn "ADMISSION_LOCK_NAMESPACE = " prima prints one line, runtime/spend.py line 158.
2. With the test Postgres on 5434 up, cd backend && python -m pytest tests/v2/test_scheduler_singleton.py -q --no-cov reports 1 failed, 7 passed, the failure being test_two_init_scheduler_calls_start_scheduler_once with assert 2 == 1 at line 432. Removing only the built assertion still fails, assert 2 == 1 on started; the change described under Tests (count start() on the instance that received more than one add_job, expect built == 2) passes.
3. Boot two workers against one database (uvicorn app_fastapi:app --workers 2 with ENVIRONMENT=production) and read the log: exactly one Scheduler singleton lock acquired (key=911000001) and exactly one Scheduler standby: singleton failover probe started (retry every 60s).
4. In psql on that database, SELECT objid FROM pg_locks WHERE locktype = 'advisory' AND database = (SELECT oid FROM pg_database WHERE datname = current_database()) shows 911000001 once while the app runs and not at all after shutdown_scheduler() (without the database predicate a leader in any other database on the server shows too, Application logic step 4).
5. Kill the leader worker (kill -9 <pid>). With uvicorn 0.37 (S4's pin) the master respawns the dead worker at once and the new process runs the election at boot, so within 60 s the log shows either Scheduler failover: acquired singleton lock from the standby (its probe fired first) or Scheduler singleton lock acquired (key=911000001) from the respawned worker (measured 2026-09-12: the respawned worker won and the failover line never appeared); in both cases SELECT count(*) FROM pg_locks WHERE locktype = 'advisory' AND objid = 911000001 AND database = (SELECT oid FROM pg_database WHERE datname = current_database()) stays 1 and SELECT pid, application_name FROM pg_stat_activity WHERE pid = (SELECT pid FROM pg_locks WHERE locktype = 'advisory' AND objid = 911000001 AND database = (SELECT oid FROM pg_database WHERE datname = current_database())) names a live process (scheduler-leader:<pid> on a new app, step 2). To observe the failover line itself, run two single-worker processes on two ports and kill one; nothing respawns it.
6. docker logs <backend> 2>&1 | grep -c 'Running job "Send booking reminder SMS' grows by one every 15 minutes; renaming the job in add_job and rebuilding makes the count stop growing under the old name, which is the phantom CRIT the monitor would raise.
7. Insert one prima_credit_grants row with status='pending', attempts=0, next_attempt_at=now() and an unreachable receiver; after 6 ticks the row shows attempts=6, next_attempt_at about one hour ahead, status='pending'; after PRIMA_CREDIT_GRANT_MAX_ATTEMPTS ticks it shows status='stuck' and still exists.
8. Run run_accounting_standing_orders() twice in one minute with one due template: exactly one journal entry, next_due_date advanced once (this is Gate C in the test file).
9. grep -rn "SCHEDULER_ROLE" backend --include='*.py' prints nothing; grep -rn "schedule_post_deploy_checks" backend --include='*.py' | grep -v scheduled_jobs.py | grep -v tests/ prints nothing.
10. In the prima repo, grep -rn "reap_abandoned_jobs(" prima scripts --include='*.py' | grep -v "def reap" prints only docstring lines; sed -n 270,278p prima/app.py shows the synchronous call after the await asyncio.sleep.
11. In psql, SELECT pid, state, application_name FROM pg_stat_activity WHERE pid = (SELECT pid FROM pg_locks WHERE locktype = 'advisory' AND objid = 911000001 AND database = (SELECT oid FROM pg_database WHERE datname = current_database())) shows the holder idle in transaction on the reference and idle with scheduler-leader:<pid> on a new app built to step 2. Then SELECT pg_terminate_backend(<that pid>): on the reference the standby logs Scheduler failover: acquired singleton lock within 60 s while the old leader keeps logging Running job lines (two leaders); on a new app built to step 4 (15 s leader tick, the deferred start) the old leader logs its demotion within 15 s and before the standby's first job, and SELECT count(*) FROM pg_locks WHERE locktype = 'advisory' AND objid = 911000001 AND database = (SELECT oid FROM pg_database WHERE datname = current_database()) never exceeds 1 after the standby promotes.