Build Rust replacement for production ZITADEL deployment

## Outcome Deliver a production-ready Rust identity and access management platform that replaces the production `feat/nb-translations` deployment at commit `4877cddfba02c3fed166a35e19927360dd8467fe`. Reuse its PostgreSQL database in place without schema or data conversion, preserve the OIDC and v2 contracts consumed by moeteadm, and provide Askama server-rendered login and administration with narrowly scoped Stimulus for browser-only capabilities. ## Constraints - Treat commit `4877cddfba02c3fed166a35e19927360dd8467fe` and its production database as the pinned compatibility contract for schema, relational writes, immutable audit events, OIDC behavior, the moeteadm v2 API subset, routes, tokens, cookies, configuration, and notifications. - Preserve strict Instance → Organization → Project ownership and authorization boundaries. - Keep domain logic independent of HTTP, templates, protocols, and PostgreSQL through explicit interfaces and transactional command/query boundaries. - Use server-side state and normal HTTP form flows by default; no SPA framework or parallel JSON-only UI implementation. - Preserve security properties for credentials, encryption keys, OAuth2/OIDC authorization-code flow with PKCE, PAT/service authentication, WebAuthn/passkeys, passwords, sessions, CSRF, redirects, headers, rate limits, and tenant isolation. - Use a rehearsed hard cutover with one active writer and no rolling Go/Rust coexistence. Prefer restarting the pinned Go binary against Rust-written compatible data for rollback, retain backup/restore as the emergency fallback, and enforce startup schema fingerprint checks and observability. - Support English (`en`), Norwegian Bokmål (`nb`), and Danish (`da`) across login, administration, validation, and notification content. - This is not an MVP or prototype; delivery may be phased, but each merged slice must be production-quality and converge on the same architecture. - Compatibility, not 1:1 replication: preserve the consumed external contracts, existing database readability, security invariants, and rollback requirements. Do not reproduce Go's internal architecture, every subsystem, or incidental event choreography; prefer simple Rust-native application/domain boundaries and invariant-focused tests. ## Acceptance criteria - A compatibility manifest pins the production commit and database fingerprint and identifies the required OIDC endpoints, moeteadm v2 methods, event/projection effects, login/admin workflows, configuration keys, and explicit exclusions. - The production-compatible database fixture opens and operates without schema or data migration; actual pinned-binary read/write paths are identified, Rust writes remain readable by the pinned Go binary, unsupported drift fails safely, and backup restoration is rehearsed as a fallback. - OIDC discovery, authorization, code/token exchange, JWKS, logout, token validation, PAT-protected v2 operations, multi-tenant authorization, audit history, required administration, invite/reset notifications, and branding meet the compatibility manifest. - Login and administration are accessible Askama SSR flows with complete `en`, `nb`, and `da` catalogs; essential operations work without JavaScript except WebAuthn, and Stimulus enhancements degrade safely. - Unit, property, integration, differential, browser, protocol-conformance, security, migration, concurrency, failure-recovery, and performance suites pass in CI with enforced coverage thresholds and reviewed exclusions. - Production artifacts include hardened container/deployment configuration, health/readiness, metrics/traces/logs with secret redaction, database fingerprint verification, hard-cutover, same-database Go rollback and emergency restore runbooks, and a signed release/SBOM/vulnerability policy. ## Scope decisions - Implementation repository: `/home/mathias/work/zitadel-rust`; `/home/mathias/work/zitadel` remains the read-only compatibility reference. - Baseline: production `feat/nb-translations`, commit `4877cddfba02c3fed166a35e19927360dd8467fe`. - Data transition: hard cutover against the existing database; no schema/data migration is expected and Go/Rust mixed writers are prohibited. - Testing safety restriction: production-data restores/clones are prohibited. Removed the dedicated local restored cluster and two orphaned identity volumes; retained the private dump archive with unchanged checksum. The owner-approved replacement is fully synthetic: 11 fake users, two organizations, fresh credentials/keys, localhost clients, no stored SMTP/SMS providers, and verified provenance guards. Interactive identity/Mailpit/RP run in an egress-denied Docker namespace with loopback-only ingress; native tests use a private PostgreSQL socket. CI bootstraps and destroys synthetic data instead of consuming production-derived secrets. Development serve is synthetic-only until explicit production cutover approval. - Languages: English (`en`), Norwegian Bokmål (`nb`), and Danish (`da`). - Public API: no legacy Admin, Auth, Management, v1, v2beta, or v3alpha compatibility. Implement only the v2 organization/user methods consumed by moeteadm plus its OIDC relying-party surface. - Legacy moeteadm migration/cleanup rake calls to `/admin/v1` and `/management/v1` are not compatibility requirements; any still-needed operation must move to the supported UI or v2 surface before cutover. ## Required external surface - OIDC: discovery, `/oauth/v2/authorize`, `/oauth/v2/token`, `/oauth/v2/keys`, `/oidc/v1/userinfo`, and `/oidc/v1/end_session`; authorization code with PKCE, client-secret authentication, `prompt=select_account`, and `openid email profile` claims. - V2 organizations: `POST /v2/organizations/_search`. - V2 users: `POST /v2/users/new`, `POST /v2/users`, and `GET`, `PATCH`, and `DELETE /v2/users/{user_id}`. Updates must cover profile, username/email, verified phone, and password/current-password checks used by moeteadm. - V2 onboarding: `POST /v2/users/{user_id}/password_reset` and `POST /v2/users/{user_id}/invite_code` preserve consumed Kaukus payloads but implement Kaukus-native setup links, not Go invitation-code UI. Silent passwordless creation; direct-password notification without forced change for changeRequired:false; explicit API or passwordless-login email request; localized two-password form without visible/hidden code fields. No SMS or old Go invitation-code support. - V2 passkeys: list, registration-link creation, registration start, verification, and deletion under `/v2/users/{user_id}/passkeys`. - Authentication/authorization: preserve existing client credentials, PAT bearer token behavior, `x-zitadel-orgid`, response bodies/statuses, IDs, password hashes, signing keys, sessions, and issued token claims used by moeteadm and its decrypt service. ## Current cutover gate Blocked. Core schema/key/PAT compatibility, the consumed OIDC authorization-code surface, durable localized password/SSO/logout flows, the implemented v2 organization/user contracts, and an initial server-rendered administration release are verified on disposable October-fixture clones. The covered Rust-written user, token, authorization-code and session state passes an exclusive-writer pinned-Go restart/readback gate without schema changes, but this is not a production cutover rehearsal. Kaukus-native setup/email implementation is complete and uncommitted, including the owner-clarified username-first login: username-only entry branches directly to check-email or a password-only stage. Signed selection is issuer/request/purpose/expiry-bound, password errors and locale changes retain the selected account, and explicit account switching returns to username/account selection. Formatting, strict Clippy, default tests and all 49 approved synthetic database tests, actual Ruby contracts and pinned-Go restart pass. Login-originated email setup now automatically authenticates the selected account and resumes the original RP state/PKCE request in its bound browser, with CSP allowing only the validated RP origin. Expired requests return to that RP origin without authentication; standalone, cross-browser and unrelated-account links cannot complete another login. Crypto/HTTP boundary tests pass, and Chrome follows actual Mailpit links through validation/locale changes directly back authenticated to the original RP in English and Danish, with no extra sign-in or manual Kaukus-return message. Chrome follows actual Mailpit setup links, verifies localized validation/notifications/direct-password readiness and completes diagnostic RP login/logout/re-authentication in the isolated namespace. Passkey management/login is still unimplemented. Real SMTP configuration/smoke and the existing production cutover controls remain required. Earlier cloned-fixture evidence does not waive the new prohibition on executing against real-data fixtures. The minimum production administration workflow is complete, including compatible organization creation with generated verified domains and creator ownership; exact production fingerprint, exclusive-writer smoke and immediate rollback controls remain mandatory at cutover. The current administration milestone creates projects and public/confidential authorization-code OIDC applications, rotates one-time client secrets using the pinned generator/hash format, and passes restored-fixture concurrency/tenant/schema checks plus pinned-Go confidential code exchange. Sensitive administration now uses explicit same-account password step-up: forms are gated by freshness, successful verification rotates the durable selected session and terminates its predecessor, failures preserve the current session, and return paths are limited to local administration routes. Validation currently passes formatting, strict workspace Clippy, 83 regular tests and 24 explicitly enabled restored-database tests (10 mutation/concurrency invariants, five moeteadm-shaped HTTP contracts, one read-only integration test, and eight lifecycle/real-Ruby-client cases). The shared harness clones an idle loopback-only fixture into disposable databases, cleans them up after assertions or failures, and leaves the source and schema unchanged. The local fixture matches the manifest: PostgreSQL 170011, 2,433 catalog items, schema SHA-256 `c6a5f3f769004cc02397fa884aeb90a045aa5d688cb8073d7b37317e55eed2d0`. A non-optional trusted-runner compatibility CI job now runs restored fixtures, the actual Ruby client and the Go restart gate, but runner provisioning and remote execution are not verified. Local Chrome passes installed-gem HTTPS login, explicit account reuse, logout/re-authentication and en/nb/da forms under script-denying CSP. Full Rails/controller execution and pinned-Go differential/rollback remain unverified. Verified improvement slice: rejected empty/short/malformed or duplicate-cookie CSRF tokens; shared strict bearer parsing between PAT APIs and userinfo; reproduced and fixed double code consumption and double login completion by locking the live request before reading lifecycle state; rechecked expiry after lookup and lock waiting; removed fixed event-sequence assumptions; extracted password complexity into the domain layer and rejected empty/over-72-byte bcrypt inputs instead of silent truncation. Restored-fixture tests cover user reads/credentials after mutations, duplicate-name concurrency, rejected-update rollback, instance-scoped repository isolation, and schema stability. No new endpoints, application schema changes, or Go architecture cloning. Verified current slice: added `identity-application` for reusable user CRUD/search, authorization, policy and password orchestration through repository/crypto interfaces, without HTTP/PostgreSQL/runtime dependencies. Domain command/credential data and HTTP password DTOs use redacted secret types. V2 handlers now translate/authenticate HTTP requests and delegate decisions to the application service; PostgreSQL remains the transactional adapter. Shared fixture cloning supports repository and real-credential HTTP suites. Organization context is resolved from `x-zitadel-orgid` or credential ownership, must exist in the configured instance, and cannot grant permissions or act as a blanket user-ID filter. Request contexts cannot be reused across instances. The HTTP role matrix confirms that explicit creation ownership and actual user ownership control CRUD permissions, even when a valid context header names another organization. Unknown contexts fail with 403 and ambiguous headers with 400. The pinned direct-password helper checks `user.write`; no additional credential-write requirement was introduced. Verified current-password hashes and actual ownership are rechecked under the stable user-write lock before mutations; bcrypt runs outside DB locks. Concurrent changes cannot reuse one proof, stale proofs reject the whole profile/password patch, and worker failures are not counted as bad credentials. HTTP contracts cover implemented search/CRUD/profile/contact/password payloads, success/error envelopes, and expired/invalid PATs. These exercise the Rust service with moeteadm-shaped requests, not the Ruby client itself. No new endpoints, migrations, or additional Go subsystems. The prior hardening slice is committed as `fae91a1`; the current application/contract slice is implemented, validated, and committed as `5c60d89`. Verified local browser harness: `compose.test.yml` builds a non-root, read-only diagnostic public relying party on the fixture-registered `localhost:3000` callback/logout URLs. It uses PKCE, validates ID-token signature/issuer/audience/nonce against JWKS, verifies userinfo subject, keeps raw tokens only in its in-memory session, and performs RP logout. The browser uses the configured public issuer; the container uses an explicit Docker host-gateway issuer for back-channel requests. `.env.local` has the matching local settings and binds the development identity server to `0.0.0.0:18080` for Docker bridge access. An end-to-end local test passed login, code exchange, JWKS/userinfo validation, and logout. It is committed as `b1e0942` and is a developer harness, not a production RP or browser acceptance suite. The OIDC/session implementation and validation harnesses are committed as `5bc68a9` (`feature: Add durable OIDC sessions and token validation`); October-baseline and pinned-Go wire-compatibility fixes are committed as `d414cb5` (`fix: Align OIDC sessions with pinned Go`); the opaque-token diagnostic-client fix is committed as `d485f62` (`fix: Support opaque tokens in diagnostic client`). The working tree is clean. Workspace formatting, strict all-target/all-feature Clippy, all 84 default tests and all 25 restored-fixture tests pass against the October baseline. Real `Zitadel::Client` executes search/CRUD/profile/contact/password/error/deletion contracts against Rust. A bounded loopback HTTPS harness uses the installed OmniAuth gem with process-only CA trust and generated disposable credentials; Chrome passes wrong-password handling, actual PKCE/JWKS/issuer/audience/nonce/userinfo callback, explicit account reuse, logout/re-authentication and en/nb/da forms. Page scripts are denied by CSP; Chrome's global JavaScript preference was not changed. Temporary servers, TLS proxies and disposable databases are cleaned up. The exact pinned-Go restart gate passes against disposable October-fixture clones: Go validates Rust-issued opaque token state and userinfo, reads the Rust-created user through v2 with the current PAT, exchanges a Rust-created authorization code, and terminates the Rust-created session; after Go stops, Rust rejects the terminated token and validates the separately Go-issued token. The schema fingerprint remains unchanged. This is verified readback/rollback compatibility for the covered paths, not a production cutover rehearsal. Verified October compatibility baseline: private dump `zitadel-db-2026-10-08_03-00-01.dump` (SHA-256 `f012a70f517c4aea4d3082d7925bbc7d085c4212f6144c1bf5a44505cd719705`, source PostgreSQL 17.9) restored independently without owner/ACL application. Rust inspection reports PostgreSQL 170011, 2,514 catalog items, 25 required objects, and schema SHA-256 `fddf9ff4035c246fd18a029613815db462d4f88acf6d1fe624ad25a8ad317936`. It contains `instance_features5`/`system_features4`; the March fixture remains obsolete and was not migrated. Current master key decrypts the October key material and the current PAT authenticates in both Rust and pinned Go. Rust-side implementation and local validation can continue, but the runtime rollback gate must remain open. The local token/session slice is implemented: opaque/JWT issuance honors client settings, code consumption atomically persists OIDC events, signatures alone never grant token authority, and both opaque/JWT tokens are checked against stored subject/lifecycle state. Pinned-Go differential testing found and fixed the access-token event encoding: token reason is the JSON string `auth_request`, not numeric enum value `1`. Session/authentication timestamps use Go's RFC3339 wire format. Login policy and bcrypt/proof orchestration use application/domain interfaces; stable locks recheck credentials, user state, policy and expiry. Required MFA fails closed, stale proofs cannot create sessions, and worker/policy failures are not bad-password events. Browser credentials are issuer-bound signed cookies; stored events support SSO across server reconstruction and explicit ownership-checked account selection. Logout terminates the underlying session, clears browser credentials and accepts valid expired ID-token hints. Retained keys verify JWTs while removed keys do not; active signer cache reuse requires matching encrypted/public envelope material. HTTP traces record route patterns without raw paths, queries, headers, cookies or bodies, with regression coverage. Do not introduce schema migrations or mixed writers. Invite/reset/email completion should follow this stage, then passkeys and required SSR administration; full conformance, browser, operational and cutover gates remain open. ## Owner-defined production cutover gate The remaining feature blocker is passkey registration/login/list/delete compatibility. Kaukus-native setup/reset/email implementation is verified on synthetic fixtures. Production SMTP configuration/smoke, explicit approval for production startup, exact production fingerprint and exclusive-writer smoke/rollback controls remain deployment gates; production-data restores remain forbidden locally. The minimum production administration UI is complete. Existing ZITADEL backup and emergency-restore procedures are accepted as the deployment escape hatch. Load/soak/HA work, extended conformance/differential matrices, broader rate-limit and lockout hardening, full key lifecycle rehearsal, SBOM/signing policy, and additional operational automation may follow after deployment and must not be represented as completed. ## Rust-owned bootstrap decision Rust must become able to initialize a brand-new compatible database without invoking the pinned Go server. Replace the Python/shell fixture orchestration with Rust development tooling and declarative container configuration. Bootstrap must refuse non-empty targets, install a reviewed versioned schema and required event-store functions transactionally, generate fresh instance/key/policy/admin/client state in compatible formats, and produce the same safe synthetic development baseline. Ordinary development and compatibility CI must not build or run Go. Keep the attested pinned Go artifact only as a temporary explicit same-database rollback oracle until the production rollback window closes; it is not the schema or fixture authority. ## Cutover-readiness audit findings The current binary cannot start in production: `serve` unconditionally requires the synthetic role/database/provenance marker and unauthenticated loopback SMTP. Add an explicit fail-closed production mode that retains exact schema/key/instance checks, requires an approved production fingerprint and secure PostgreSQL settings, and permits validated STARTTLS SMTP without weakening synthetic guards. Passkeys remain a consumed contract, not optional administration: moeteadm calls all five manifest endpoints, and login has no passkey/WebAuthn flow. Existing passwordless users and required-MFA users need compatible login behavior and existing-credential coverage. Rust also replays failed password attempts but does not enforce the effective ZITADEL lockout policy, and it reads password-age settings only for administration rather than login expiry. Before cutover, implement those policy semantics or verify through a production read-only inventory that they are disabled and record explicit compensating controls; production-grade rate limiting remains required at the service or edge. The release also lacks deployment/runbook artifacts, production SMTP/deployed-client smoke evidence, full Rails/passkey/decrypt-service contracts, production semantic preflight, actionable email/backlog and authentication observability, and a rehearsed traffic switch with pinned-Go same-database rollback. The configured dependency gate is currently red: `cargo deny check` rejects workspace/path/license policy and reports RUSTSEC-2023-0071 for the network-exposed RustCrypto RSA signing path; resolve the policy configuration and replace or formally mitigate the affected signing backend before relying on CI or releasing. Existing Go-issued setup links, in-flight Rust email jobs/links during rollback, key rotation, and the remaining legacy moeteadm migration/cleanup rake calls require explicit operating decisions. Cutover still preserves the already-established non-negotiable controls: exact production schema fingerprint, no migrations, one active writer with Go workers stopped, basic Rust smoke checks, and immediate Go restart or emergency restore on failure. User commands now have reusable application interfaces; remaining login/OIDC orchestration and future SSR operations should extend these boundaries rather than add HTTP-specific business logic. Further hardening should review backend-error/audit secret redaction, actor attribution, and permission revocation/write races. The live matrix is `compatibility/status.md` in the implementation repository. Compatibility research must use the pinned commit explicitly: the reference repository's working-tree HEAD is on a different revision.