Module kit / Billing and credits
Prima threshold sync
billing-credits/prima-threshold-sync
| Kind | backend |
|---|---|
| Domain | billing-credits |
| Maturity | unmerged |
| Reusable | extract |
| Stack | fastapi-0.118, sqlalchemy-sync, postgres, httpx-0.28, pydantic-settings-2.11, stdlib-hmac-sha256 |
| Depends on | _shared/fastapi-project-layout, _shared/workspace-scoping, _shared/error-envelope, _shared/outbox-pattern, _shared/scheduler-and-background-work, _shared/hmac-seam-pattern, _shared/alembic-migrations |
| Frontends | billing-credits/ui/subscription-manager |
| Reference implementation |
|
| Measured on | alchemy_labs w1-auto-topup af7a35246dd4a009485bc67b3a13fea6e5a688cc 2026-09-12 |
| Version | 1 |
Purpose
A tenant's auto-top-up threshold - the low-balance line Prima compares against balance_micro + allowance_micro inside its own locked debit transaction - is saved in Labs but decided in Prima, so the setting has to be durably pushed across the wire rather than merely written to a Labs table (module docstring, prima_threshold_sync.py:1-10). The invariant that makes it trustworthy is that the obligation to publish the setting (threshold_sync_seq bumped, threshold_sync_status='pending') is written and committed in the same transaction as the setting itself - mark_pending never commits, the caller's single db.commit() does (prima_threshold_sync.py:114-127, primaapi_router.py:1559-1560) - so a Prima outage at save time leaves a visible pending row a drain retries, never a setting that looks saved in Labs and is silently missing on Prima's side.
When to use
Read it before touching prima_auto_topups.threshold_sync_*, before changing what Labs sends to PUT /internal/credits/threshold, or when someone asks "why is auto top-up not firing even though the toggle is on", "what does threshold_sync_status='stuck' mean", or "why did the threshold not reach Prima". Trigger phrases: "threshold sync", "threshold_sync_seq", "set_seq", "PUT /internal/credits/threshold", "prima_threshold_sync", "sync stuck", "MAX_THRESHOLD_MICRO".
Not this recipe: the crossing → charge flow that fires once Prima's threshold is synced and a low-balance event arrives (auto_topup_service.py, run_auto_topup_retry, ADR-0036 §4a - ownership, dunning, prima_balance_events, all a separate recipe); the CHF top-up grant/outbox that credits a balance after a Payrexx payment (prima_credit_grants.py - a sibling outbox with the same shape and the same backoff table, but money entering the ledger rather than a comparand); the HMAC scheme's own mechanics, shared by both outboxes (_shared/hmac-seam-pattern, prima_credit_sig.py); the full tenant-facing PUT/DELETE /credits/auto-topup contract - body validation, the card-on-file guard, the If-Match concurrency check, the stalled-crossing state machine (those live in primaapi_router.py and belong to a broader "auto top-up settings" recipe; this recipe covers only the two calls those handlers make into prima_threshold_sync and what happens after).
Stack assumptions
At af7a352: fastapi==0.118.2, synchronous handlers on a SQLAlchemy Session, httpx==0.28.1 for the outbound PUT (imported locally inside deliver(), prima_threshold_sync.py:143, so a test can monkeypatch the module-level httpx.put freely), pydantic-settings==2.11.0 for core.config.settings, stdlib hmac/hashlib/uuid for the signature (prima_credit_sig.py). No ORM relationship is used; PrimaAutoTopup rows are queried and locked directly.
Foundations assumed: the outbox shape (claim inside the save transaction, deliver inline as an optimisation, drain on a schedule, terminal stuck state) is _shared/outbox-pattern's, and this file's own docstring says its backoff table (_BACKOFF_SECONDS = (5, 15, 60, 300, 900, 3600), prima_threshold_sync.py:76) is "the same schedule, and the same reasoning" as prima_credit_grants._BACKOFF_SECONDS - copied as a literal tuple, not imported (Gotchas). The signature scheme is _shared/hmac-seam-pattern's: signed_headers (prima_credit_sig.py:63-84) builds X-Prima-Timestamp, X-Prima-Nonce, X-Prima-Signature: v1=<hmac> over canonical_string = f"{ts}\n{nonce}\n{METHOD}\n{path}\n{sha256hex(body)}" (prima_credit_sig.py:49-52). The drain rides the existing scheduled job rather than getting its own timer (_shared/scheduler-and-background-work). tenant_id is the workspace id (_shared/workspace-scoping) - prima_auto_topups.tenant_id is a ForeignKey('workspaces.id', ondelete='RESTRICT') (prima_auto_topup.py:135-138). The two Labs-side routes that call into this module wrap their 403s in the Labs envelope (_shared/error-envelope): error.type: permission_error, error.code: AUTHZ_001 (primaapi_router.py:815-817, 856-858).
Environment (core/config.py): PRIMA_INTERNAL_BASE_URL (Optional[str], default None, line 767) - unset means "outbox rows queue as pending and are never delivered", by design, so the failure is visible as a stuck queue rather than a crash. PRIMA_CREDIT_GRANT_SECRET (Optional[SecretStr], default None, line 755) - shared with POST /internal/credits/grant and /reverse; the same credential authorises the threshold write, and what stops one endpoint becoming a lever on the other is the method+path inside the signed string, not a distinct credential (prima_credit_sig.py:38-42, prima_threshold_sync.py:38-42). PRIMA_CREDIT_GRANT_TIMEOUT_SECONDS (float, default 10.0, line 756). PRIMA_CREDIT_GRANT_MAX_ATTEMPTS (int, default 24, line 757) - shared with the grant outbox; there is no threshold-specific attempts setting.
Fail behaviour when PRIMA_INTERNAL_BASE_URL or PRIMA_CREDIT_GRANT_SECRET is unset: deliver() takes a dedicated branch (prima_threshold_sync.py:147-161) that increments threshold_sync_attempts before calling _fail, logs an ERROR naming the tenant, attempt count and resulting status, and returns False; the row stays pending until attempts exhaust into stuck. The comment at those lines records that this ordering is itself a fix: with the counter untouched, an unconfigured Labs deployment logged an ERROR for every tenant on every drain tick forever and never reached the one queryable "a human must look" state.
Data model
The sync columns on prima_auto_topups (models/prima_auto_topup.py:190-199; created by migration prima_auto_topups_001, migrations/versions/20260910_0001_prima_auto_topups.py:45, down_revision = 'subscription_cancel_otp_001' - re-pointed onto the reunified main head by the commit this recipe is measured on, per that commit's own message):
prima_auto_topups (
tenant_id uuid PRIMARY KEY REFERENCES workspaces(id) ON DELETE RESTRICT,
enabled boolean NOT NULL DEFAULT false,
threshold_sparks integer NOT NULL DEFAULT 0, -- display unit; what the tenant typed
-- ... other auto-top-up columns (card, cap, crossing state) belong to a
-- different recipe; not reproduced here.
threshold_sync_seq bigint NOT NULL DEFAULT 0, -- the wire's set_seq; monotonic per tenant
threshold_sync_status varchar(16) NOT NULL DEFAULT 'synced',
threshold_sync_attempts integer NOT NULL DEFAULT 0,
threshold_sync_next_attempt_at timestamptz NULL,
threshold_sync_last_error text NULL, -- a class name or "HTTP nnn", never a response body
CHECK (threshold_sync_status IN ('pending','synced','stuck')) -- constraint 'sync_status'
);
CREATE INDEX ix_prima_auto_topups_sync_due
ON prima_auto_topups (threshold_sync_status, threshold_sync_next_attempt_at);
threshold_sync_status values (prima_auto_topup.py:67-71): pending - not yet reached Prima, a drain retries it; synced - Prima holds this row's threshold_sync_seq; stuck - attempts exhausted, or Prima refused with a 4xx; never deleted, a state a human is meant to look at.
The index ix_prima_auto_topups_sync_due exists specifically for the drain's query (comment at migrations/versions/20260910_0001_prima_auto_topups.py:101-102): every row still owed a push whose backoff has elapsed.
What is NOT stored, anywhere in Labs: the balance, the allowance, or the µCHF threshold value as a number Labs reads back and reasons about - Labs sends a comparand and reads back a status; threshold_sparks (Sparks, the display unit) is the only persisted figure, and µCHF is computed once, at send time (module docstring, prima_threshold_sync.py:44-46, threshold_micro_for, prima_threshold_sync.py:84-106). No signature is stored on the outbox row: every attempt, including a retry, gets a fresh timestamp and nonce and is re-signed, because a stored signature would fail Prima's replay-skew window on a retry an hour later (prima_credit_sig.py:67-71).
API surface
This module has no HTTP routes of its own. It is called from inside two tenant-facing routes on primaapi_router.py (prefix /api/prima/v1, primaapi_router.py:48), and it makes one outbound call to Prima.
Inbound (Labs), both requiring a Labs session (never the ai-scoped inference key) and the workspace owner or an admin, in that order (_require_labs_session then _require_billing_role, primaapi_router.py:1390-1391, 1592-1593):
| Route | Calls into this module | Codes relevant to the sync | Notes |
|---|---|---|---|
PUT /api/prima/v1/credits/auto-topup | mark_pending(row) then, after commit, deliver_for_tenant(db, ctx.tenant_id) (primaapi_router.py:1559, 1563) | 403 session_required (not a session); 403 owner_required (not owner/admin); 422 threshold_out_of_range (enabled with threshold_sparks < 1, checked before any sync call, primaapi_router.py:1410-1417, and again if threshold_micro > MAX_THRESHOLD_MICRO, primaapi_router.py:1436); 200 on success, with threshold_sync_status in the response body | the endpoint's own body/card/If-Match rules are a different recipe |
DELETE /api/prima/v1/credits/auto-topup | same two calls, on a row that has just been cleared (_clear_pending_fields, mark_pending, then deliver_for_tenant, primaapi_router.py:1611, 1614) | same 403s; 200 always (idempotent: configured: false even with no row) | switching off sends Prima null for the threshold (clears the line) |
Outbound (Prima), signed, from deliver() (prima_threshold_sync.py:135-233):
PUT {PRIMA_INTERNAL_BASE_URL}/internal/credits/threshold
Headers: X-Prima-Timestamp, X-Prima-Nonce, X-Prima-Signature: v1=<hmac>, Content-Type: application/json.
Body (compact-serialised JSON, exactly four keys, asserted exhaustively by test_the_wire_carries_exactly_four_keys_and_no_amount_micro):
{"tenant_id": "<uuid>", "threshold_micro": 5000000, "set_seq": 3, "source": "auto_topup"}
threshold_micro is null when the row is disabled (threshold_micro_for, prima_threshold_sync.py:104-106).
Response handling (prima_threshold_sync.py:196-233):
200- applied or "you sent something older than what I have" (both treated as success; Prima's response is expected to echoapplied,stale,fired, logged but not otherwise read,prima_threshold_sync.py:203-211). Row set tosynced.400–499except408/429- terminal: row set tostuck,threshold_sync_last_error = "HTTP {code}", not retried.408,429,5xx, or a transport exception - retryable: row stays/returns topendingwith a scheduledthreshold_sync_next_attempt_atvia_fail(prima_threshold_sync.py:236-253), or goes tostuckoncePRIMA_CREDIT_GRANT_MAX_ATTEMPTS(24) is reached.
deliver() itself never raises on a transport or server-side failure - that is the whole point of the pending row being a queryable state rather than an exception a caller must remember to catch (prima_threshold_sync.py:140-142).
Application logic
CLAIM - mark_pending(row) (prima_threshold_sync.py:114-127), called synchronously inside the tenant endpoint's own transaction, in commit order:
row.threshold_sync_seq = int(row.threshold_sync_seq or 0) + 1.row.threshold_sync_status = "pending".row.threshold_sync_attempts = 0,row.threshold_sync_last_error = None(a fresh obligation resets the retry state of any prior one).row.threshold_sync_next_attempt_at = now()(due immediately).- Returns the new
set_seq. Does not commit - the caller's singledb.commit()(primaapi_router.py:1560or:1612) is what makes the setting and the obligation to publish it durable together.
DELIVER, inline - deliver_for_tenant(db, tenant_id, source="auto_topup") (prima_threshold_sync.py:256-285), called immediately after that commit, never inside it (an HTTP call inside the transaction that is about to persist the tenant's setting is how a Prima timeout becomes a lost setting):
SELECT ... WHERE tenant_id = :tid AND threshold_sync_status = 'pending' FOR UPDATE SKIP LOCKED LIMIT 1.SKIP LOCKEDso a scheduled drain racing this call cannot both send the same row; even if they did, Prima'sset_seqpredicate makes the loser's write a no-op (prima_threshold_sync.py:300-304).- No row found (already synced, or lost the lock race) → return
False, nothing else happens. deliver(db, row, source=source)(below), wrapped in a baretry/exceptthat never lets a delivery failure escape into the endpoint - a saved setting must not become a 500 because Prima is unreachable (prima_threshold_sync.py:278-283).db.commit()- persists whateverdeliver()or the exception handler set on the row.
DELIVER, the attempt itself - deliver(db, row, source=...) (prima_threshold_sync.py:135-233), owns no transaction, does not commit:
- If
PRIMA_INTERNAL_BASE_URLorPRIMA_CREDIT_GRANT_SECRETis unset: bumpthreshold_sync_attempts, call_failwith"ThresholdSyncNotConfigured", log ERROR, returnFalse(:147-161). - Build the payload (
tenant_id,threshold_micro_for(row),set_seq,source); serialise withjson.dumps(..., separators=(",", ":"))so the signed bytes are exactly the sent bytes (:163-173). - calls
signed_headerswith the credential,method="PUT",path=THRESHOLD_PATHandbody=body- fresh timestamp and nonce (:174-177). - Bump
threshold_sync_attempts(before the network call, so a crash mid-call still counts as an attempt on the next read). httpx.put(...). Transport exception →_fail(row, type(exc).__name__, None)(class name only, never a message that might carry a response body or a host), log, returnFalse(:187-194).200→synced, clearnext_attempt_at/last_error, log the echoedapplied/stale/fired, returnTrue(:196-212).400–499except408/429→stuck,last_error = "HTTP {code}",next_attempt_at = None, log ERROR, returnFalse(:214-226).- Anything else (
408,429,5xx) →_fail(row, None, status_code), log WARNING, returnFalse(:228-233).
_fail(row, exception_class, status) (:236-253): if threshold_sync_attempts >= PRIMA_CREDIT_GRANT_MAX_ATTEMPTS (24) → stuck, next_attempt_at = None, log ERROR "is STUCK after N attempts". Else → pending, next_attempt_at = now() + backoff(attempts) where backoff indexes _BACKOFF_SECONDS = (5, 15, 60, 300, 900, 3600) by attempts - 1, clamped (:79-81).
DRAIN - drain_once(db, limit=50) (prima_threshold_sync.py:293-338), the second pass of the scheduled prima_credit_grant_drain job:
SELECT ... WHERE threshold_sync_status = 'pending' AND (next_attempt_at IS NULL OR next_attempt_at <= now()) ORDER BY updated_at ASC LIMIT :limit FOR UPDATE SKIP LOCKED.- Per row (not per batch, so one bad row cannot take the successful sends either side of it back to
pending):deliver(db, row)inside its owntry/except UnexpectedDrainError, thendb.commit()- one commit per row. - Returns
{"due": len(rows), "delivered": delivered}.
stuck_thresholds(db, older_than_minutes=15) (:341-363): every row that is stuck, or pending and updated_at older than the cutoff - "everything a human should look at", each one a tenant whose toggle is on and which Prima will never trigger because it does not know where the line is.
Invariants
set_seqis strictly monotonic per tenant and is bumped once per save, in the same transaction as the setting (mark_pending). Prima applies a write only when the incomingset_seqis strictly greater than what it already stored - the same convergent ruleprima_key_syncs.sync_sequses - so a retried, reordered, or stale delivery can never regress a newer setting (test_mark_pending_advances_the_sequence_monotonically).- The obligation to sync is durable before any network call is made:
mark_pendingruns and is committed inside the tenant endpoint's own transaction;deliver_for_tenantruns only after that commit and is explicitly "an optimisation, never the mechanism" (prima_threshold_sync.py:257-265). deliver()never raises. Every failure path - unconfigured seam, transport exception, any HTTP status - ends in a row mutation and areturn False, never an unhandled exception (test_a_transport_failure_records_the_class_name_and_never_a_body,test_an_unconfigured_seam_leaves_a_pending_row_rather_than_raising).- A 4xx other than
408/429is terminal (stuck) on the first occurrence; it is never retried, because retrying a malformed-body or bad-signature failure changes nothing (test_a_4xx_goes_straight_to_stuck_rather_than_burning_attempts). - The signed string binds the HTTP method and path, so a signature captured for
POST /internal/credits/grantis not a valid signature forPUT /internal/credits/thresholdand vice versa, even though both usePRIMA_CREDIT_GRANT_SECRET(test_a_grant_signature_is_not_valid_for_the_threshold_endpoint). - Nothing Labs ever sends or stores on this path is money entering the ledger: the payload's only numeric field beyond
set_seqisthreshold_micro, a comparand, and the wire carries exactly those four keys (test_the_wire_carries_exactly_four_keys_and_no_amount_micro). - A disabled row's threshold is cleared in Prima (
threshold_micro: null), never left at its last value - an old line left behind would keep firing crossings Labs would answerdisabledforever (test_a_disabled_row_clears_the_threshold_rather_than_leaving_the_old_one). threshold_sync_last_erroris a class name or"HTTP {status}", never a response body or an exception message that might carry a host or a stack fragment (test_a_transport_failure_records_the_class_name_and_never_a_body).
Frontend
The module that renders the state this recipe writes, by id: billing-credits/ui/subscription-manager, one line: it is the surface that shows the tenant's auto-top-up toggle and threshold, and is expected to read threshold_sync_status off GET /credits/auto-topup and show the tenant when a saved setting has not yet reached Prima - the exact shape of that field is not determined by this recipe's scope (see the _auto_topup_shape response in the endpoint recipe, not this one).
Minimum admin/debug surface, all measurable from prima_threshold_sync.py: stuck_thresholds(db) - every stuck row, or pending past 15 minutes, oldest first (:341-363); the raw query it runs is reproducible as SELECT tenant_id, threshold_sync_status, threshold_sync_attempts, threshold_sync_last_error, threshold_sync_next_attempt_at FROM prima_auto_topups WHERE threshold_sync_status = 'stuck' OR (threshold_sync_status = 'pending' AND updated_at < now() - interval '15 minutes') ORDER BY updated_at ASC; the log lines prima threshold: tenant %s is STUCK after %s attempts (%s) and prima threshold: tenant %s REFUSED by Prima with %s - not retryable, marked stuck (:244-247, 221-225).
Security & tenancy
Auth dependency: the two routes that call into this module require ctx.auth_method == "jwt" (a Labs session; the inference key is refused session_required, primaapi_router.py:820-827) and then user_may_change_billing(db, ctx.user.id, ctx.tenant_id) (owner or admin; anyone else refused owner_required, primaapi_router.py:861-868). This module itself is not reachable from any route directly; it is a plain Python call from an already-authorised handler. Workspace derivation: tenant_id comes from ctx.tenant_id (the gateway identity, derived from the session/key, never from the request body) - the same id is what Prima's wire payload and every prima_auto_topups row key on.
Fail direction: mark_pending never fails (a plain attribute assignment); deliver() and deliver_for_tenant() fail closed toward the row's own state, never toward an exception - every branch ends in pending or stuck, and deliver_for_tenant's own try/except is a second backstop so a bug inside deliver() still leaves the row in a valid, queryable state rather than crashing the request that just saved the tenant's setting. drain_once fails closed per row: an unexpected exception during one row's delivery is caught, recorded as UnexpectedDrainError, and does not stop the rows queued behind it.
Secrets at rest: PRIMA_CREDIT_GRANT_SECRET is read from SecretStr (.get_secret_value(), prima_threshold_sync.py:174-176) and never logged; threshold_sync_last_error is constrained to a class name or an HTTP nnn string by construction (no code path assigns a response body or exception message to it). No card, balance, or PII is anywhere on this path - the module docstring is explicit that there is no balance in this file (:44-46).
Rate limits: not determinable from the two source files this recipe is scoped to - neither prima_threshold_sync.py nor prima_credit_sig.py references a limiter, and the two calling routes' rate-limit behaviour belongs to whatever primaapi_router.py applies generally, which is out of this recipe's read scope.
Attacker-controlled inputs on this path: none reach prima_threshold_sync directly - tenant_id is server-derived (ctx.tenant_id), and threshold_sparks is validated and range-checked by the caller (primaapi_router.py:1410-1417, 1436) before mark_pending is ever called. The one input this module's own code bounds is the µCHF ceiling: MAX_THRESHOLD_MICRO = 100_000 * 1_000_000 (prima_threshold_sync.py:71), described in its own comment as "a blast radius against a units mix-up, not a business rule" - but the check against it lives in the router (primaapi_router.py:1436), not in threshold_micro_for or deliver themselves (Gotchas).
Tests
backend/tests/prima_credits/test_auto_topup_client.py, NO DATABASE by design (module docstring, lines 1-13) - the signature scheme and "an unkeyed event never charges" are treated as too important to run only when a test database happens to be configured. Run at af7a352, 2026-09-12: python -m pytest tests/prima_credits/test_auto_topup_client.py -q --no-cov -o addopts="" → 26 passed (22 def test_... functions, one parametrized over 5 HTTP codes: 408, 429, 500, 502, 503).
Known-answer vectors (KAT_SECRET, KAT_TIMESTAMP="1785500000", KAT_NONCE="kat-nonce-0001", a fixed JSON body): test_the_threshold_canonical_string_names_PUT_not_POST asserts the canonical string's method field is PUT, not POST; test_the_threshold_signature_matches_the_prima_vector asserts sign(...) reproduces the expected HMAC; test_a_grant_signature_is_not_valid_for_the_threshold_endpoint asserts the same credential over the same body but a different method+path produces a different signature. The docstring names the vector's twin as prima/tests/ on Prima's side - that repo is outside this recipe's read scope, so whether the two vectors currently agree is not verified by this recipe; it is stated as a fact the source asserts, not one this pass confirmed independently.
Retry state machine, all hermetic (an httpx.put monkeypatch, no database, a MagicMock() session): test_the_body_is_compact_serialised_so_the_signed_bytes_are_the_sent_bytes, test_the_wire_carries_exactly_four_keys_and_no_amount_micro, test_a_4xx_goes_straight_to_stuck_rather_than_burning_attempts, test_a_retryable_status_schedules_a_backoff[408|429|500|502|503], test_a_transport_failure_records_the_class_name_and_never_a_body, test_an_unconfigured_seam_leaves_a_pending_row_rather_than_raising, test_mark_pending_advances_the_sequence_monotonically.
Sparks → µCHF and the zero-threshold rule: test_the_threshold_travels_in_micro_chf_at_one_thousand_per_spark, test_a_disabled_row_clears_the_threshold_rather_than_leaving_the_old_one, test_an_enabled_row_can_never_hand_prima_a_threshold_of_zero.
Live, database-backed tests of the calling routes (not this module's own tests, but the only place mark_pending/deliver_for_tenant are exercised end to end through a real handler): backend/tests/prima_credits/test_auto_topup_endpoints.py, entire module skipped unless PRIMA_CREDITS_TEST_DATABASE_URL is set (pytestmark, lines 34-37) - not run in this pass (no scratch Postgres was stood up for it). It monkeypatches prima_threshold_sync.deliver_for_tenant to lambda *a, **k: False throughout (line 82) so that module's own tests never make an outbound HTTP call; the relevant case is test_a_zero_threshold_while_enabled_is_refused_here_not_by_prima, which asserts the router's own 422 guard fires before mark_pending is ever reached.
Playwright: none found referencing this path in the files read for this recipe; the frontend module that would carry one (billing-credits/ui/subscription-manager) is being authored separately.
Deploy & ops
Config keys (core/config.py): PRIMA_INTERNAL_BASE_URL, PRIMA_CREDIT_GRANT_SECRET, PRIMA_CREDIT_GRANT_TIMEOUT_SECONDS (default 10.0), PRIMA_CREDIT_GRANT_MAX_ATTEMPTS (default 24) - all shared with the prima_credit_grants outbox; there is no threshold-specific config key.
Migrations order: prima_auto_topups_001 (migrations/versions/20260910_0001_prima_auto_topups.py), down_revision = 'subscription_cancel_otp_001' at af7a352 - the commit this recipe is measured on exists specifically to re-chain this migration onto the reunified main head after a rebase (commit message: "Re-point its base so the chain descends from the new main head -> single head"), which is itself evidence this branch had, at least once, diverged into a second Alembic head and is not yet the schema main runs.
Scheduler: no dedicated job. prima_threshold_sync.drain_once rides the existing prima_credit_grant_drain job as its "SECOND PASS" (services/scheduled_jobs.py:723-749) - scheduler.add_job(run_prima_credit_grant_drain, IntervalTrigger(minutes=1), id='prima_credit_grant_drain', coalesce=True, max_instances=1) (services/scheduled_jobs.py:337-344). The threshold pass runs in its own try/except after the grant pass, with an explicit db.rollback() first if the grant pass left the session in a failed transaction - "a broken grant drain must not silently stop threshold delivery (or the reverse)" (services/scheduled_jobs.py:731-749).
Argus-style check: SELECT count(*) FROM prima_auto_topups WHERE threshold_sync_status = 'stuck' should be 0 in steady state; a nonzero count is exactly what stuck_thresholds surfaces. Rollback: nothing deployable beyond the code and the one migration; a stuck row is never deleted by a rollback, only by a human fixing the underlying refusal and the row re-entering pending on the tenant's next save.
Gotchas
- 2026-09-12 (measured at af7a352):
MAX_THRESHOLD_MICRO(prima_threshold_sync.py:71) is defined in this module and exported, but the only place it is actually checked is the router, at save time (primaapi_router.py:1436). Neitherthreshold_micro_fornordeliverre-checks it. A row written directly (a migration, a manualUPDATE, a future caller that does not go through the endpoint) with athreshold_sparksvalue whose µCHF equivalent exceeds the ceiling would be sent to Prima uncapped by this module itself; the "blast radius against a units mix-up" the comment describes is enforced only at one call site, not at the seam. - 2026-09-12 (measured at af7a352): the backoff table (
_BACKOFF_SECONDS,prima_threshold_sync.py:76) is a hand-copied literal, not an import fromprima_credit_grants. The comment says it is "the same schedule, and the same reasoning" - nothing in the code enforces that claim; changing one file's schedule without the other silently desyncs the retry cadence of the two outboxes. - 2026-09-12 (measured at af7a352): the module's own docstring flags that an enabled row is not supposed to reach
deliver()withthreshold_sparks == 0because the tenant endpoint refuses that combination with 422 first (prima_threshold_sync.py:96-102) - but the docstring also says there is deliberately no assertion enforcing this insidedeliver()itself, because the function's job is to make progress for every other tenant queued behind it. A0written by hand into the table is answered by Prima's own 400 and parkedstuck, visibly, rather than raised as an exception here. - 2026-09-12 (measured at af7a352):
test_auto_topup_endpoints.py, the only test module that exercisesmark_pending/deliver_for_tenantthrough a real HTTP handler rather than by callingdeliver()directly, is skipped wheneverPRIMA_CREDITS_TEST_DATABASE_URLis unset. This recipe's own test run only covered the hermetic module; the end-to-end path (endpoint →mark_pending→ commit →deliver_for_tenant) was read, not executed, in this pass.
Reference implementation
Paths relative to the repo root; commit as in measured_on.
alchemy_labs(w1-auto-topupataf7a352):backend/services/commerce/prima_threshold_sync.py(364 lines; this recipe's core):threshold_micro_for(84-106),mark_pending(114-127),deliver(135-233),_fail(236-253),deliver_for_tenant(256-285),drain_once(293-338),stuck_thresholds(341-363).backend/services/commerce/prima_credit_sig.py(85 lines):canonical_string(49-52),sign(55-60),signed_headers(63-84).backend/models/prima_auto_topup.py: thethreshold_sync_*columns (190-199), thesync_statusCHECK (272-275) - the file's other columns (card, cap, crossing state) belong to a different recipe and are not reproduced above.backend/api/routers/primaapi_router.py:AutoTopupRequest(1098-1121),put_auto_topup(1359-1566, the two sync calls at 1559 and 1563),delete_auto_topup(1569-1619, the two sync calls at 1611 and 1614),_require_labs_session(792-827),_require_billing_role(832-868).backend/services/scheduled_jobs.py:run_prima_credit_grant_drain(684-752, the threshold second pass at 723-749), the job registration (337-344).backend/migrations/versions/20260910_0001_prima_auto_topups.py: the table and index (45 onward),down_revision(45).- Tests as listed under Tests.
Acceptance checks
Run from alchemy_labs/backend at af7a352 (a Python environment matching requirements.txt, no database required for the first four). Expected observations are exact.
python -m pytest tests/prima_credits/test_auto_topup_client.py -q --no-cov -o addopts=""prints26 passed(measured 2026-09-12).grep -c "^def test_" tests/prima_credits/test_auto_topup_client.pyprints22.grep -n "MAX_THRESHOLD_MICRO" services/commerce/prima_threshold_sync.py api/routers/primaapi_router.pyshows exactly one definition (prima_threshold_sync.py:71) and exactly one comparison against it (primaapi_router.py:1436) - the Gotcha above.grep -n "down_revision" migrations/versions/20260910_0001_prima_auto_topups.pyprintsdown_revision = 'subscription_cancel_otp_001'.grep -n "sync_status" migrations/versions/20260910_0001_prima_auto_topups.pyshows the CHECKthreshold_sync_status IN ('pending','synced','stuck')and the indexix_prima_auto_topups_sync_dueon(threshold_sync_status, threshold_sync_next_attempt_at).grep -n "id='prima_credit_grant_drain'" services/scheduled_jobs.pyandgrep -n "_threshold_drain" services/scheduled_jobs.pyboth match - confirming the drain rides the existing job rather than a job of its own.- With
PRIMA_CREDITS_TEST_DATABASE_URLset and a scratch database (pertests/prima_credits/conftest.py):python -m pytest tests/prima_credits/test_auto_topup_endpoints.py -k "zero_threshold or enabling_without_a_card" -qpasses; not run in this recipe's own pass (see Tests).
Content synced from the prima-modules repository at 0df2ec8 on 2026-09-14. The recipe file is the source; this page is a rendering of it.