Key-sync outbox, Labs to Prima

Purpose

A key minted in Labs works on Prima seconds later, and a key revoked in Labs stops working on Prima, whatever order the messages arrive in and however many times they are retried. The invariant that makes it trustworthy is that the obligation to tell Prima is a row written in the same transaction as the key, carrying the key's whole desired state and a per-key counter, and Prima applies a message only where prima_api_keys.sync_seq < EXCLUDED.sync_seq, so a replay changes nothing and a stale "active" that arrives after a revocation changes nothing (alchemy_labs/backend/services/primaapi/prima_key_sync.py 42 to 58 and prima/prima/api/keys_sync.py 475 to 504, read 2026-09-12 at fd8558f and cd2c248).

When to use

Use it wherever a credential is minted in one service and validated in another that must not call back: it is S11's outbox and S10's signed seam applied to ai-scoped keys, and it is what closed the 2026-08-04 defect where every self-serve key 401ed on the gateway (ADR-0026). Trigger phrases: "the key works in Labs and not on api.prima.li", "stuck row", "pending for too long", "sync_seq", "a revoked key still works", "superseded", "Prima refused the key with 400", "the drain", "reconcile the pre-seam keys", "PRIMA_KEY_SYNC_SECRET".

Do not use it for the pattern itself (_shared/outbox-pattern, S11, which names this outbox as one of four and fixes the shared shape; _shared/hmac-seam-pattern, S10, which fixes the canonical string and the credential rules), for the mint route (identity-auth/developer-api-keys-and-security-policy, B10, the caller), for how Prima validates the row afterwards (identity-auth/gateway-api-key-auth, B11), for the money seam (billing-credits, the credit-grant outbox with the same shape and a different credential), or for the other direction (Prima reading Labs, B13, a design with a death date).

Stack assumptions

Pinned as measured at fd8558f and cd2c248 (2026-09-12): Labs as S4 (SQLAlchemy sync sessions, httpx for the one outbound call, APScheduler's IntervalTrigger on the leader, S12), Prima as the S7 and S10 reference (an async signature dependency in front of a sync handler so the raw body can be awaited, keys_sync.py 209 to 227). Foundations assumed: S8 for the two migrations, S9 for the hermetic test layout on both sides. The scheme is the money-IN seam's byte for byte (prima_key_sig.py 18 to 29; internal_sig.py 14 to 26).

Environment names and their fail behaviour:

Side	Name	Default	Behaviour
Labs (core/config.py)	PRIMA_KEY_SYNC_ISSUER	None (712)	falls back to JWT_ISSUER (_issuer, prima_key_sync.py 322 to 338), whose default is https://labs.alchemy.zuerich (113 to 116), so the raise at 332 to 337 is reachable only when JWT_ISSUER is set to an empty string; unset on a deployment whose Prima federates a different issuer, every mint queues a row that parks as stuck with HTTP 400 (Gotchas); must equal Prima's PRIMA_JWT_ISSUER byte for byte or Prima answers 400 unknown_issuer
Labs	PRIMA_KEY_SYNC_SECRET	None, SecretStr (719)	unset: deliver records SyncNotConfigured and the row stays pending (374 to 384); never the read-seam value, never the credit-grant value (prima_key_sig.py 31 to 43)
Labs	PRIMA_INTERNAL_BASE_URL	None (752)	one variable for both Labs-to-Prima seams; a loopback address on the estate, never the public ingress, which does not route /internal/*; unset behaves as the line above
Labs	PRIMA_KEY_SYNC_TIMEOUT_SECONDS	10.0 (720)	per attempt
Labs	PRIMA_KEY_SYNC_MAX_ATTEMPTS	24 (721)	after which _fail parks the row as stuck
Prima (prima/config.py)	PRIMA_KEY_SYNC_SECRET	unset, optional (903)	unset: require_labs_signature answers 503 key_sync_not_configured to everyone and logs at ERROR (229 to 243); at least 16 characters when set; refused at boot when equal to PRIMA_LABS_INTERNAL_SECRET or PRIMA_CREDIT_GRANT_SECRET (959 to 989)
Prima	PRIMA_JWT_ISSUER	required	compared with the payload's issuer (353 to 359)

Data model

Labs, the outbox (S11's shape, one row per key, mutated in place):

-- models/prima_key_sync.py 85 to 169; migrations/versions/20260804_0001_prima_key_syncs.py (prima_key_sync_001, revises prima_magic_link_tpl_001).
-- The stored CHECK names carry the ck_<table>_ prefix from core/database.py's naming convention (S11 Gotchas).
prima_key_syncs (
  id              uuid PRIMARY KEY,
  api_key_id      uuid NOT NULL REFERENCES api_keys(id) ON DELETE RESTRICT,   -- fk_prima_key_syncs_api_key; the row is evidence
  key_hash        varchar(64) NOT NULL,          -- sha256 hex, CHECK (key_hash ~ '^[0-9a-f]{64}$'): a plaintext key cannot be inserted
  key_prefix      varchar(32) NOT NULL,          -- api_keys.key_prefix[:32], trailing dots and spaces stripped, 'prima_pk_' when that is empty (199, 277), so Prima's token pattern accepts it
  tenant_id       uuid NOT NULL,                 -- api_keys.workspace_id copied, no FK; used verbatim as prima_tenants.id
  external_sub    varchar(255) NOT NULL,         -- users.id as a string, the JWT sub
  issuer          varchar(255) NOT NULL,
  scopes          jsonb NOT NULL,                -- the sorted, lowercased list; only lists containing "ai" are ever enqueued
  desired_active  boolean NOT NULL DEFAULT true, -- state, not an event
  expires_at      timestamptz NULL,
  sync_seq        bigint NOT NULL DEFAULT 1,     -- CHECK (sync_seq > 0); +1 on every state change
  status          varchar(16) NOT NULL DEFAULT 'pending',   -- CHECK IN ('pending','synced','stuck')
  attempts        integer NOT NULL DEFAULT 0,
  next_attempt_at timestamptz NULL,
  last_error      text NULL,                     -- an exception class name or 'HTTP <n>', never a body
  created_at      timestamptz NOT NULL DEFAULT now(),
  updated_at      timestamptz NOT NULL DEFAULT now(),
  synced_at       timestamptz NULL,
  UNIQUE (api_key_id)                            -- uq_prima_key_syncs_api_key: one row per key, enforced by the database
);
CREATE INDEX ix_prima_key_syncs_key_hash ON prima_key_syncs(key_hash);
CREATE INDEX ix_prima_key_syncs_tenant ON prima_key_syncs(tenant_id);
CREATE INDEX ix_prima_key_syncs_status ON prima_key_syncs(status);
CREATE INDEX ix_prima_key_syncs_next_attempt ON prima_key_syncs(next_attempt_at);
CREATE INDEX ix_prima_key_syncs_due ON prima_key_syncs(status, next_attempt_at);   -- the drain's query

Prima, the target (B11 owns the tables; this seam adds one column):

-- migrations/versions/20260804_0001_key_sync_seq.py (prima_key_sync_001, revises prima_auth_001)
ALTER TABLE prima_api_keys ADD COLUMN sync_seq bigint NOT NULL DEFAULT 0;
-- 0 = written by scripts/backfill_api_key.py before the seam; every seam message starts at 1 and wins.
-- Written by the seam, with ON CONFLICT: prima_subjects (issuer, external_sub) DO NOTHING; prima_tenants (id) DO NOTHING;
-- prima_api_keys (key_hash) DO UPDATE ... WHERE prima_api_keys.sync_seq < EXCLUDED.sync_seq.

The wire payload (_payload, prima_key_sync.py 346 to 357; KeySyncRequest, keys_sync.py 291 to 335), compact JSON with no spaces:

{"issuer": "<token, <=255>", "external_sub": "<token, <=255>", "tenant_id": "<uuid>", "key_hash": "<64 hex>",
 "key_prefix": "<token, <=32>", "scopes": ["<token, <=64>", "..."], "is_active": true, "expires_at": "<iso or null>", "sync_seq": 1}

TOKEN_PATTERN is ^[A-Za-z0-9_.:/@+-]+$ (prima/content_free.py 126); scopes is 1 to 16 entries; sync_seq >= 1. There is no field for a plaintext key, no email, no name, no workspace name (test_the_plaintext_key_has_no_field_to_travel_in). What is not stored on either side: the raw key, a signature (recomputed on every attempt with a fresh nonce and timestamp, prima_key_sig.py 45 to 48), a Prima response body.

API surface

Route	Auth dep	Tenancy	Codes	Notes
POST /internal/keys/sync (Prima, keys_sync.py 338 to 543, mounted unconditionally at prima/app.py 362, off the gateway prefix: INTERNAL_PREFIX = "/internal/keys", 181)	LabsOnly = Depends(require_labs_signature) (209 to 283): X-Prima-Timestamp, X-Prima-Nonce, X-Prima-Signature over the raw body; never CurrentIdentity	tenant_id from the body, inserted verbatim	200 {applied, superseded, key_present, is_active, sync_seq, subject_id, tenant_id}; 503 key_sync_not_configured when the credential is unset; 401 for missing headers, a malformed or out-of-window timestamp (300 s, internal_sig.MAX_SKEW_SECONDS, 75), or a bad signature; 400 unknown_issuer, unrepresentable_scopes, not_a_gateway_key, subject_suspended; 400 invalid_request with detail Request body failed validation. on the Pydantic bounds (Prima maps every RequestValidationError to 400 and never echoes the input, prima/app.py 110 to 128)	off the gateway prefix the wire body is S7's Shape E, {"detail": <message>, "request_id": <hex>} (prima/app.py 72 to 74); the code names above, and the param naming the failed field (126), are on the PrimaError and in the request_failed log, and neither reaches the caller; the tests assert on detail

The caller's contract, deliver (prima_key_sync.py 360 to 448): 200 is success whether applied is true or superseded is true (409 to 425); any 4xx other than 408 and 429 is stuck at once with last_error = 'HTTP <n>' (427 to 441); 408, 429 and every 5xx schedule a retry (443 to 448); a transport exception records its class name and schedules a retry (400 to 407). Signature headers are built by signed_headers (prima_key_sig.py 77 to 96): X-Prima-Timestamp (unix seconds), X-Prima-Nonce (uuid4().hex, 32 characters), X-Prima-Signature (v1= plus HMAC-SHA256 hex over ts\nnonce\nPOST\n/internal/keys/sync\nsha256hex(body)), Content-Type: application/json.

There is no Labs route for this outbox: no admin endpoint lists, resets or drains it, and reconcile_existing_keys and stuck_syncs have no caller outside the module and its tests (grep 2026-09-12 at fd8558f). The admin console the inventory names for it (F30) is planned.

Application logic

Enqueue, inside the caller's transaction, never committing (prima_key_sync.py 134 to 319):

1. enqueue_mint(db, api_key) from create_api_key (developer_router.py 173) and from create_api_only_account (identity.py 358, B14): is_gateway_key (116 to 126) is "ai" in scopes; a key without ai returns None and writes nothing. A gateway key with workspace_id None raises ValueError and fails the mint (155 to 162). A wildcard among the scopes is logged at ERROR and enqueued anyway so Prima's 400 parks it as stuck (164 to 177). db.flush() when api_key.id is still None (179 to 189, the first live mint failed here). Insert the row with sync_seq = 1, status = 'pending', attempts = 0, next_attempt_at = now, under a savepoint; an IntegrityError on the UNIQUE rolls back only the savepoint and returns the existing row (211 to 232).
2. enqueue_revocation(db, api_key) from revoke_api_key (developer_router.py 310): lock the key's row FOR UPDATE (blocking, 251 to 255); with a row, set desired_active = False, sync_seq += 1, and reset status, attempts, last_error, next_attempt_at as one unit (307 to 315); with no row and a gateway key (pre-seam or backfilled), insert a desired_active = False row at sync_seq = 1 under a savepoint (257 to 304); with no row and no ai, return None.
3. The caller commits once; the key row and the outbox row are one atomic unit.

Deliver, after the commit:

4. Inline, deliver_for_key(db, api_key_id) (519 to 547): select the key's pending row FOR UPDATE SKIP LOCKED, deliver, commit; any exception becomes UnexpectedInlineError on the row and never fails the completed mint. Called under a try by both routes (developer_router.py 185 to 191, 316 to 323) and by the API-only signup (identity.py 368).
5. Scheduled, run_prima_key_sync_drain (scheduled_jobs.py 631 to 665) every minute (IntervalTrigger(minutes=1), coalesce=True, max_instances=1, 322 to 329) calls drain_once(db, limit=50) (474 to 516): rows pending with next_attempt_at null or past, oldest first, FOR UPDATE SKIP LOCKED, one deliver and one commit per row so one bad row cannot take its neighbours back to pending.
6. deliver(db, row) (360 to 448): unconfigured is _fail(row, "SyncNotConfigured") before the attempt counter moves (372 to 384, Gotchas); otherwise serialise _payload(row) compactly, sign, attempts += 1, httpx.post with the timeout; then the status table above. _fail (451 to 466) sets last_error, and either stuck with next_attempt_at = NULL when attempts >= PRIMA_KEY_SYNC_MAX_ATTEMPTS, or pending with next_attempt_at = now + _BACKOFF_SECONDS[min(attempts - 1, 4)] over (5, 15, 60, 180, 600) (96 to 107): 24 attempts span about 3.2 hours before stuck (computed 2026-09-12).

Receive, sync_key on Prima (keys_sync.py 338 to 543), one transaction:

7. require_labs_signature: credential unset is 503; the three headers must be present; abs(now - timestamp) <= 300; internal_sig.verify recomputes v1=<hmac> over the raw awaited body and compares with hmac.compare_digest over the whole header value (internal_sig.py 94 to 113), so a v2= prefix fails rather than being read as v1.
8. issuer must equal PRIMA_JWT_ISSUER when that is set (353 to 359): a signature proves who sent the message, not that it may name any issuer.
9. Normalise scopes (sorted, lowercased); refuse an intersection with WILDCARD_SCOPES with 400 unrepresentable_scopes, which in practice catches admin and full (367 to 381); refuse a list without ai with 400 not_a_gateway_key (382 to 396). Both log key_sync_refused at ERROR with the tenant id and the reason. * never reaches this step: it is not a TOKEN_PATTERN character, so the scopes field validator (327 to 335) refuses it first and the RequestValidationError handler answers 400 invalid_request with detail Request body failed validation. and no key_sync_refused event. Both refusals are 400s, and a 400 is terminal for the outbox (step 6), which is the property that matters; which layer caught it is visible only in the wording.
10. Insert the subject (gen_random_uuid(), issuer, external_sub, is_native=false, email=NULL, status='active') ON CONFLICT (issuer, external_sub) DO NOTHING, then read its id and status (402 to 415).
11. FIND-011: is_active true for a subject whose status is not active is 400 subject_suspended (432 to 447); a deactivation always lands.
12. Insert the tenant (tenant_id, issuer, is_internal=false) ON CONFLICT (id) DO NOTHING (454 to 460): the endpoint can create a tenant and can never renumber one.
13. Upsert the key: INSERT ... ON CONFLICT (key_hash) DO UPDATE SET subject_id, tenant_id, key_prefix, scopes, is_active, expires_at, sync_seq = EXCLUDED.* WHERE prima_api_keys.sync_seq < EXCLUDED.sync_seq RETURNING id, sync_seq (475 to 503); applied = row is not None; commit; re-read the current row; log key_sync with ids and applied; answer 200 with applied, superseded = not applied, and the current is_active, sync_seq, subject_id, tenant_id (506 to 543). subject_id is in the SET so the first genuine message re-points a backfilled key from its placeholder subject onto the real one (469 to 474).

Invariants

• The obligation is as durable as the key: enqueue_* never commits and never catches anything but the UNIQUE collision, and the callers write the outbox row before their single commit (developer_router.py 153 to 176; prima_key_sync.py 139 to 148).
• One row per key, enforced by uq_prima_key_syncs_api_key, carrying whole desired state including is_active; there is no second row for a revocation (models/prima_key_sync.py 35 to 55; test_the_payload_carries_state_not_an_event).
• sync_seq is a counter, bumped on every state change, and Prima applies only WHERE sync_seq < EXCLUDED.sync_seq; a replay is applied: false, superseded: true and a 200 (keys_sync.py 490; test_a_superseded_answer_is_a_success; live: test_the_same_message_twice_is_idempotent, test_a_revoked_key_is_never_resurrected_by_a_stale_retry).
• Only ai-bearing keys are enqueued (is_gateway_key; test_only_ai_scoped_keys_are_gateway_keys), and Prima refuses any other with 400 (test_a_key_without_the_ai_scope_is_refused), so a stuck row is always a customer's gateway key and never a Labs read key.
• A 4xx is terminal for the worker and a 5xx is not (deliver 427 to 448; test_a_4xx_goes_straight_to_stuck_rather_than_retrying_forever, test_a_transient_status_is_retried over 408, 429, 500, 502, 503, 504); nothing ever deletes a row (test_exhausting_attempts_parks_the_row_as_stuck_and_never_deletes_it).
• Every attempt is signed afresh; no signature is stored (test_every_attempt_is_signed_afresh); the canonical string is identical on both sides and pinned by a known-answer vector (Tests).
• No key material and no PII cross: key_hash is checked as 64 hex on both sides (the Labs CHECK, Prima's _HASH_PATTERN 190), last_error is a class name or a status (test_last_error_is_a_class_or_a_status_never_a_response_body), and the request model has no field for a name or an email (test_the_plaintext_key_has_no_field_to_travel_in).
• The seam is not on the gateway prefix and a customer's key cannot reach it (test_the_seam_is_not_on_the_gateway_prefix, test_a_customers_gateway_key_is_not_a_way_in); the key path still cannot import Labs after the seam was added (test_the_key_path_still_cannot_reach_labs).
• tenant_id is workspaces.id verbatim, copied onto the outbox row at enqueue and inserted DO NOTHING; the seam never renumbers a tenant (live: test_the_tenant_id_is_never_renumbered).
• A suspended subject's replacement key is refused and its deactivation is not (live: test_a_replacement_key_is_refused_for_a_suspended_subject, test_a_deactivation_still_applies_even_for_a_suspended_subject).

Frontend

The frontend modules that render this layer, by id (the same set as frontend_modules above, 2026-09-12):

identity-auth/ui/api-keys-settings-and-developer-dashboard   interim: the mint and revoke buttons whose outcome this
                                                             outbox propagates; the card shows nothing of the sync
                                                             (no status, no "works on Prima yet"), and a stuck row is
                                                             invisible to the developer who holds the key
billing-credits/ui/billing-outbox-admin-console (planned)    the inventory's frontend for this row (B12 names only
                                                             F30): one table per outbox with status, attempts,
                                                             next_attempt_at, last_error, created_at, the key prefix
                                                             and the tenant; a filter that is exactly stuck_syncs();
                                                             a reset action that is exactly the statement under
                                                             Deploy & ops; read-only for non-admins (S11 Frontend);
                                                             not a file at 58f13a0, so not in the frontmatter yet

A symptom site, not a renderer, and so not in frontend_modules:

identity-auth/ui/developer-portal-keys-page                  the portal mints an ai key at sign-in and then calls
                                                             GET /api/prima/v1/usage on it; a 401 there within the
                                                             first seconds is this outbox's inline delivery having
                                                             failed and the drain not yet run

Minimum admin and debug surface until F30 exists: the stranded-key query from the module docstring (prima_key_sync.py 25 to 34: SELECT s.id, s.api_key_id, s.tenant_id, s.key_prefix, s.desired_active, s.status, s.attempts, s.last_error, s.created_at, s.updated_at, u.email FROM prima_key_syncs s JOIN api_keys k ON k.id = s.api_key_id JOIN users u ON u.id = k.user_id WHERE s.status <> 'synced' ORDER BY s.created_at), stuck_syncs(db, older_than_minutes=5) from a shell (589 to 617), the Labs log lines prima key sync: queued mint ..., ... delivered (seq=.. active=.. applied=..), ... REFUSED by Prima with <n>, ... is STUCK after <n> attempts, SCHEDULER: Prima key sync drain with the due and delivered counts (the source line carries a dash the house rule forbids here), and on Prima the events key_sync (ok), key_sync_refused (ERROR, with reason), key_sync_signature_rejected (ERROR), key_sync_not_configured (ERROR).

Security & tenancy

Auth dependency: on Prima, require_labs_signature, a separate function reading a separate credential from the credit seam's near-twin on purpose (keys_sync.py 220 to 225); on Labs, none (the outbox has no route). Workspace derivation: the outbox row's tenant_id, copied from api_keys.workspace_id at enqueue so a later change in Labs' workspace resolution cannot move a queued key (models/prima_key_sync.py 57 to 64). Fail direction: closed on Prima (unset credential is 503, bad signature 401, unrepresentable key 400, suspended subject 400); on Labs a delivery failure is a visible pending or stuck row, never an exception in the mint, and an enqueue failure is a failed mint, never a silent key.

Blast radius of the credential: PRIMA_KEY_SYNC_SECRET lets its holder write a working ai key for any tenant id it names, which is impersonation of any customer on the gateway (keys_sync.py 34 to 54). That is why it is neither the read-seam value Prima holds nor the credit-grant value, why Prima refuses to boot when two of the three are equal, and why Labs at fd8558f, which has no such boot check (S10 Gotchas), is the side to be careful with in the heredoc. The issuer check (step 8) is the one field a compromised or buggy caller could otherwise use to file a subject where the revocation feed would never find it.

Secrets at rest: the credential in each service's environment; key_hash on the row and on the wire; nothing else. Rate limits: none on the endpoint; the caller is one outbox worker with a 10 s timeout and a 50-row drain.

Attacker-controlled inputs, on the assumption that the endpoint is reachable: the three headers (timestamp parsed as int, skew bounded, nonce bounded by nothing but the canonical string, signature compared constant-time over the whole value); the body (Pydantic bounds on every field; tenant_id is a UUID; scopes are tokens of at most 64 characters; key_hash is 64 hex, so no string that could be a key passes); replay of a captured request within 300 s, which is a no-op at the database because the same sync_seq fails the predicate (internal_sig.py 37 to 50 gives the same argument for the money seam). On the estate the endpoint is bound to a loopback address and the public ingress does not route /internal/* (core/config.py 752).

Tenancy of the message: one key, one tenant, one subject; the subject is (issuer, external_sub) and is never a copy of the Labs user (no email by CHECK); the tenant is the workspace id and a fresh tenant row is created on first sight. Prima's tenant cannot be re-keyed by this seam, and the metering chain depends on that (keys_sync.py 151 to 157).

Tests

Labs (alchemy_labs/backend/tests/prima_keys/test_prima_key_sync.py, run 2026-09-12 at fd8558f with python -m pytest ... -q --no-cov -o addopts="" -p no:cacheprovider): 26 passed in 0.63s, no database. The wire contract: test_the_signature_matches_the_prima_vector (40), test_every_attempt_is_signed_afresh (73), test_the_payload_carries_a_hash_and_never_a_key (107), test_the_payload_carries_state_not_an_event (121); the retry policy: test_backoff_is_front_loaded_and_clamped (137), test_a_retryable_failure_schedules_another_attempt (144), test_exhausting_attempts_parks_the_row_as_stuck_and_never_deletes_it (155), test_last_error_is_a_class_or_a_status_never_a_response_body (173); the delivery outcomes: test_a_200_marks_the_row_synced (233), test_a_superseded_answer_is_a_success (252), test_a_4xx_goes_straight_to_stuck_rather_than_retrying_forever (261), test_a_transient_status_is_retried (275, parametrised), test_a_transport_failure_leaves_a_pending_row_not_an_exception (283), test_unconfigured_stalls_visibly_rather_than_raising (298); the gateway rule: test_only_ai_scoped_keys_are_gateway_keys (332, six cases), test_the_wildcard_set_matches_primas (340).

Prima (prima/tests/, run 2026-09-12 at cd2c248 with .venv/bin/python -m pytest <file> -q -o addopts=""): tests/auth/test_key_sync_seam.py 20 passed in 0.33s, hermetic through the HTTP layer with the credential set on a test settings object: test_unset_secret_refuses_everyone_with_503_not_404 (94), test_an_unsigned_request_is_refused (109), test_a_signature_from_the_wrong_secret_is_refused (115), test_a_signature_over_a_different_body_is_refused (127), test_a_stale_timestamp_is_refused (149), test_a_customers_gateway_key_is_not_a_way_in (156), test_a_wildcard_scope_is_refused_rather_than_narrowed (177, parametrised over *, admin, full), test_a_key_without_the_ai_scope_is_refused (197), test_an_unknown_issuer_is_refused (206), test_a_key_hash_that_is_not_a_sha256_is_refused (217, parametrised), test_a_zero_sync_seq_is_refused (225), test_the_plaintext_key_has_no_field_to_travel_in (233), test_the_canonical_string_matches_the_labs_vector (251), test_the_seam_is_not_on_the_gateway_prefix (285), test_the_key_path_still_cannot_reach_labs (293). tests/test_key_sync_live.py is opt-in on PRIMA_TEST_DATABASE_URL and 8 skipped without it: test_a_mint_creates_subject_tenant_and_key (166), test_the_same_message_twice_is_idempotent (182), test_revocation_propagates (203), test_a_revoked_key_is_never_resurrected_by_a_stale_retry (214), test_a_backfilled_subject_is_healed_rather_than_duplicated (237), test_a_replacement_key_is_refused_for_a_suspended_subject (308), test_a_deactivation_still_applies_even_for_a_suspended_subject (352), test_the_tenant_id_is_never_renumbered (369). Those eight are the ones that prove properties of the ON CONFLICT ... WHERE predicate rather than of an if, and they are the ones a rebuilt seam cannot skip.

Known-answer vector, asserted on both sides (test_prima_key_sync.py 40 to 70; test_key_sync_seam.py 251 to 282; recomputed 2026-09-12): timestamp 1785500000, nonce abc123, method POST, path /internal/keys/sync, body {"hello":"world"}, canonical string 1785500000\nabc123\nPOST\n/internal/keys/sync\n followed by the SHA-256 hex of the body, HMAC key shared-secret, signature v1=2ede58bc4616041070f1df52184ef19317ddefce428af811b23994c00a42ad6b. A rebuilt signer or verifier that prints anything else is a wire break.

Hermetic versus live: every test above is hermetic except the eight opt-in live cases; no test reaches a deployed host. ADR-0026 records the live proof on 2026-08-04 (mint, use, revoke, 401). Playwright gate: none; the seam has no browser.

Deploy & ops

Config keys per app heredoc (S13): Labs PRIMA_KEY_SYNC_ISSUER, PRIMA_KEY_SYNC_SECRET, PRIMA_INTERNAL_BASE_URL, optionally PRIMA_KEY_SYNC_TIMEOUT_SECONDS and PRIMA_KEY_SYNC_MAX_ATTEMPTS; Prima PRIMA_KEY_SYNC_SECRET (the same value as Labs' and different from Prima's other two), PRIMA_JWT_ISSUER (equal to Labs' issuer). Both sides read the values at boot.

Migrations order: Labs 20260804_0001_prima_key_syncs.py after prima_magic_link_tpl_001; Prima 20260804_0001_key_sync_seq.py after prima_auth_001. Prima first, then Labs, then the Labs config: a Labs mint with ai inserts into prima_key_syncs inside the mint transaction, so Labs must have the table before any ai mint, and Prima must have sync_seq before the first delivery or every message is a 500 that retries. The downgrade drops the outbox with every undelivered obligation; do not run it on a database that has minted.

Scheduler jobs (S12): prima_key_sync_drain, every minute on the leader. The day the seam ships, once: reconcile_existing_keys(db) from a shell (550 to 586), which queues a row for every ai key that has none; deliberately not scheduled, so a later non-zero count is a fact a human notices.

Argus checks: none exist at the measured commits (S10 Deploy & ops names the four log events a watcher would key on). The minimum: SELECT count(*) FROM prima_key_syncs WHERE status = 'stuck' OR (status = 'pending' AND created_at < now() - interval '5 minutes') is 0, and Prima's key_sync_not_configured and key_sync_signature_rejected events are absent from the last hour.

Reset and rollback: a stranded row is replayed by UPDATE prima_key_syncs SET status = 'pending', attempts = 0, next_attempt_at = now(), last_error = NULL WHERE id = :id (or WHERE status = 'stuck' AND last_error = 'HTTP 401' after a credential rotation); the next drain sends it and Prima's predicate makes a duplicate harmless. A credential rotation with the drain running parks every in-flight row as stuck with HTTP 401, so rotate both apps in one window and then reset. Rolling back the Labs code to before 24f16ef removes the enqueue; keys minted in that window never reach Prima and reconcile_existing_keys is what catches them up afterwards. Rolling back Prima to before prima_key_sync_001 makes every delivery a 500 that retries until the column returns.

Gotchas

• 2026-09-12 (measured at fd8558f, D8, fix needed; S11's step 7 describes the branch without this consequence): the unconfigured branch of deliver runs before row.attempts += 1 (374 to 384 precede 392), so _fail sees attempts unchanged, never reaches PRIMA_KEY_SYNC_MAX_ATTEMPTS, and reschedules with _backoff(1) = 5 s. On a host with PRIMA_KEY_SYNC_SECRET or PRIMA_INTERNAL_BASE_URL unset, every row retries on every drain forever and none parks as stuck, so stuck_syncs() still finds them only through its five-minute pending clause. The fix is to count the unconfigured attempt.
• 2026-09-12 (measured at fd8558f): reconcile_existing_keys takes the first 500 api_keys rows with a workspace, in no defined order, and only then skips the ones with a row or without ai (569 to 581). A table with more than 500 such rows can hide pre-seam gateway keys past the slice on every run. A rebuilt reconcile filters on the missing outbox row in SQL, or orders and pages.
• 2026-09-12 (measured at fd8558f, D8, fix needed): PRIMA_KEY_SYNC_ISSUER's description at core/config.py 712 promises "Unset = every mint raises rather than queuing a key that can only ever be stuck"; _issuer (prima_key_sync.py 329 to 331) falls back to JWT_ISSUER, which is a str with the default https://labs.alchemy.zuerich (113 to 116), so the raise at 332 to 337 fires only when JWT_ISSUER is set to an empty string. On a deployment whose Prima federates a different issuer, an unset PRIMA_KEY_SYNC_ISSUER produces the outcome the description says it prevents: every mint succeeds, every row is refused with 400 unknown_issuer and parks as stuck with HTTP 400. The fix is either to drop the JWT_ISSUER fallback or to give JWT_ISSUER no default; until then the heredoc sets PRIMA_KEY_SYNC_ISSUER explicitly on every host.
• 2026-09-12 (measured at fd8558f): stuck_syncs and reconcile_existing_keys have no caller outside the module and its tests, and the drain job logs counts, not ids. Until F30 exists, finding a stranded key is a shell or a psql prompt, and the inventory rule "no headless outbox" is unmet here on the shipping line.
• 2026-09-12 (measured at fd8558f): enqueue_mint enqueues a wildcard-bearing ai key on purpose and lets Prima's 400 park it (164 to 177); the 4xx path then writes last_error = 'HTTP 400' without the code Prima raised (unrepresentable_scopes, not_a_gateway_key, unknown_issuer, subject_suspended), because a response body is never copied onto the row. The reason is on Prima's side, in the key_sync_refused event's reason field. No route at fd8558f can mint such a key; the branch is for hand-written rows.
• 2026-09-12 (measured at fd8558f): revocation of a key that was never a gateway key returns None and writes nothing (305), and revocation of a gateway key with workspace_id NULL raises (269 to 272), which fails the DELETE with a 500 after the soft delete is rolled back. Every route-minted ai key has a workspace, so the branch is reachable only through a hand edit.
• 2026-09-12 (measured at cd2c248): tests/auth/test_key_sync_seam.py line 253 names its Labs twin as tests/prima_keys/test_prima_key_sig.py; the Labs file is tests/prima_keys/test_prima_key_sync.py. The vector agrees; the path in the docstring is stale (S10 Gotchas carries the same).
• 2026-09-12 (measured at cd2c248): the 400s and the 503 leave code off the wire because the internal routes are off the gateway prefix (Shape E, prima/app.py 72 to 74). Labs sees HTTP 400 and a human sees the reason only in Prima's log. A rebuilt receiver that wants the code on the wire renders it in Shape E's detail or adds a code key; the outbox must then still write only the status.
• 2026-09-12 (measured at fd8558f and cd2c248): the backoff table is (5, 15, 60, 180, 600) here and (5, 15, 60, 300, 900, 3600) on the credit and Sparks outboxes (S11 Gotchas); with 24 attempts this outbox gives up after about 3.2 hours, the others after about 18.4 hours. Deliberate: the thing waiting here is a customer who cannot make a first call.
• 2026-08-04 (measured, alchemy_labs 24f16ef; S11 carries it): the first live mint failed inside enqueue_mint with a NULL api_key_id because APIKey.id is a Python-side default applied at flush; the db.flush() at 179 to 189 is the fix and the comment is the record.
• 2026-08-04 (ADR-0026): the founder's key was backfilled by hand before the seam existed and its subject row carried a placeholder external_sub; the subject_id in the upsert's SET and sync_seq = 0 on the backfilled row are what let the first real message heal it (keys_sync.py 469 to 474; 20260804_0001_key_sync_seq.py 30 to 38). The live test test_a_backfilled_subject_is_healed_rather_than_duplicated is the regression guard.

Reference implementation

Paths are relative to each repo's root; commits as in measured_on and also_measured_on, read 2026-09-12.

• alchemy_labs (d1a-token-source, fd8558f):
  • backend/services/primaapi/prima_key_sync.py: the docstring with the stranded-key SQL (1 to 64), SYNC_PATH, AI_SCOPE, WILDCARD_SCOPES (84 to 94), _BACKOFF_SECONDS and _backoff (102 to 107), _scopes_of (110 to 113), is_gateway_key (116 to 126), enqueue_mint (134 to 237), enqueue_revocation (240 to 319), _issuer (322 to 338), _payload (346 to 357), deliver (360 to 448), _fail (451 to 466), drain_once (474 to 516), deliver_for_key (519 to 547), reconcile_existing_keys (550 to 586), stuck_syncs (589 to 617).
  • backend/services/primaapi/prima_key_sig.py: the scheme and the credential rule (18 to 48), canonical_string (63 to 66), sign (69 to 74), signed_headers (77 to 96).
  • backend/models/prima_key_sync.py: the design docstring (1 to 65), SYNC_STATUSES (82), PrimaKeySync (85 to 175).
  • backend/migrations/versions/20260804_0001_prima_key_syncs.py: revision, down_revision (55 to 56), the table (62 to 111), the downgrade (114 to 120).
  • backend/services/scheduled_jobs.py: the registration (315 to 329), run_prima_key_sync_drain (631 to 665).
  • backend/core/config.py: PRIMA_KEY_SYNC_ISSUER (712), PRIMA_KEY_SYNC_SECRET (719), PRIMA_KEY_SYNC_TIMEOUT_SECONDS (720), PRIMA_KEY_SYNC_MAX_ATTEMPTS (721), PRIMA_INTERNAL_BASE_URL (752).
  • backend/api/routers/developer_router.py: the enqueue and inline delivery call sites (153 to 191, 299 to 323); backend/services/primaapi/identity.py: the API-only signup's call sites (337 to 368).
  • backend/tests/prima_keys/test_prima_key_sync.py.
• prima (feature/per-model-pricing, cd2c248):
  • prima/api/keys_sync.py: the argument for the second direction and the credential (1 to 158), INTERNAL_PREFIX (181), _HASH_PATTERN (190), require_labs_signature (209 to 283), KeySyncRequest (291 to 335), sync_key (338 to 543).
  • prima/internal_sig.py: MAX_SKEW_SECONDS (75), canonical_string (78 to 81), sign (84 to 91), verify (94 to 113).
  • prima/app.py: the router include (362), Shape E (72 to 74).
  • prima/config.py: key_sync_secret (903), the PRIMA_KEY_SYNC_SECRET map entry (1088), the pairwise refusal (959 to 989).
  • prima/auth/models.py: PrimaApiKey.sync_seq (223 to 232); migrations/versions/20260804_0001_key_sync_seq.py (55 to 74).
  • prima/content_free.py: TOKEN_PATTERN (126).
  • tests/auth/test_key_sync_seam.py, tests/test_key_sync_live.py.
• Wiki philosophers_stone (6322fa3): operations/decisions/0026-labs-to-prima-key-propagation.md (the decision, the live proof), platform/primaapi.md ("Self-serve signup mints a key Prima cannot see"), askedquestions/2026-08-21-prima-user-provisioning-and-spark-grants.md, operations/prima-modules-inventory.md rows B12, F30, S10, S11.

Deviations of the reference from this recipe, all dated 2026-09-12: the unconfigured branch never reaches stuck; the issuer fallback that makes the config description false; the reconcile slice of 500 in no order; no route, job or page reads stuck_syncs; the refusal code stays on Prima's side of the wire; the stale twin-test path in Prima's docstring; no Labs boot check that the three seam credentials differ (S10).

Acceptance checks

Run from alchemy_labs/backend at fd8558f and from prima at cd2c248, or from a new pair built to this recipe. $PRIMA is a running Prima with the credential set, $SECRET_VALUE the shared value (never echoed). Expected observations are exact.

1. Labs: python -m pytest tests/prima_keys/test_prima_key_sync.py -q --no-cov -o addopts="" -p no:cacheprovider prints 26 passed. Prima: python -m pytest tests/auth/test_key_sync_seam.py -q -o addopts="" prints 20 passed; python -m pytest tests/test_key_sync_live.py -q -o addopts="" prints 8 skipped without a database and 8 passed with PRIMA_TEST_DATABASE_URL set to a scratch database.
2. Anywhere with Python 3: python3 -c "import hashlib,hmac; b=b'{\"hello\":\"world\"}'; base='1785500000\nabc123\nPOST\n/internal/keys/sync\n'+hashlib.sha256(b).hexdigest(); print('v1='+hmac.new(b'shared-secret',base.encode(),hashlib.sha256).hexdigest())" prints v1=2ede58bc4616041070f1df52184ef19317ddefce428af811b23994c00a42ad6b.
3. Labs: grep -n "prima_key_sync_drain" services/scheduled_jobs.py | grep -c "id=" prints 1; grep -c "with_for_update(skip_locked=True)" services/primaapi/prima_key_sync.py prints 2 and grep -c "with_for_update()" services/primaapi/prima_key_sync.py prints 1 (the blocking lock in enqueue_revocation).
4. Labs, with the :5434 Postgres: python -c "from services.primaapi.prima_key_sync import _BACKOFF_SECONDS, _backoff; print(_BACKOFF_SECONDS, [_backoff(n).total_seconds() for n in (0,1,2,5,6,99)])" prints (5, 15, 60, 180, 600) [5.0, 5.0, 15.0, 600.0, 600.0, 600.0].
5. Labs, end to end with the app running and B10's checks 2 and 4 done (an ai key minted from a session): psql -c "SELECT status, attempts, sync_seq, desired_active, last_error FROM prima_key_syncs ORDER BY created_at DESC LIMIT 1" prints synced | 1 | 1 | t | when Prima is reachable, or pending | 1 | 1 | t | ConnectError (or another class name) when it is not; never a row with a plaintext key in any column, and SELECT count(*) FROM prima_key_syncs WHERE key_hash !~ '^[0-9a-f]{64}$' prints 0.
6. Labs: revoke that key through DELETE /api/developer/keys/{id}; the same SELECT now prints sync_seq 2, desired_active f, and status synced once delivered; on Prima psql -c "SELECT is_active, sync_seq FROM prima_api_keys WHERE key_hash = :h" prints f | 2.
7. Prima, a replay: re-send the seq-1 message by hand (sign the payload from check 5 with the shared value and POST $PRIMA/internal/keys/sync): the response is 200 with "applied":false,"superseded":true,"is_active":false,"sync_seq":2; the key stays revoked.
8. Prima: curl -s -i -X POST -H 'Content-Type: application/json' -d '{}' $PRIMA/internal/keys/sync shows 401 and "detail":"Missing internal signature headers."; with a timestamp 400 s old and an otherwise valid signature, 401 and "detail":"Internal signature timestamp is outside the window."; with PRIMA_KEY_SYNC_SECRET unset on a scratch instance, 503 and a detail containing not configured; with a valid prima_pk_ key in X-API-Key and no signature, 401.
9. Prima: a signed message whose scopes is ["read"] answers 400 with detail starting A gateway key must carry the 'ai' scope; ["ai", "admin"] (or ["ai", "full"]) answers 400 with detail starting This key carries a scope this service does not define; ["ai", "*"] answers 400 with detail Request body failed validation., because * fails TOKEN_PATTERN in the field validator before the wildcard branch runs (test_a_wildcard_scope_is_refused_rather_than_narrowed accepts either wording for that reason); an issuer other than PRIMA_JWT_ISSUER answers 400 with detail This deployment does not federate that issuer.; sync_seq: 0 answers 400 with detail Request body failed validation. (test_a_zero_sync_seq_is_refused asserts the 400).
10. Labs: grep -rn "reconcile_existing_keys\|stuck_syncs" --include='*.py' . | grep -v "tests/\|services/primaapi/prima_key_sync.py" prints exactly one line at fd8558f, the docstring mention at services/scheduled_jobs.py:651, and no call; on a build with F30 or an admin route it prints a call site as well.