Implement simple tag-based Docker deployment with staged engine readiness
Investigate replacement of the current Kamal deployment path while preserving Docker, existing networking/data/secrets/observability, and player restart warnings. User supplied Ithaca deploy log for d9a92377ed6b5e4739e333f7144d25a1d9c5ce6b: job 6m35s; deploy/check step 6m1s; Kamal command 335.8s including buildx build/push 191.020s; target pull 12.579s. Eight sequential dotnet publishes total 136.6s (PFE 55.2, Login 10.6, Chat 10.1, Portal 18.7, ClientWeb 14.6, Market 11.7, CLI 10.1, NavigationSmoke 5.6); COPY source 15.2s after 286.38MB context transfer; export/push 18.9s included in 191s. Eight non-portal roles each wait a fixed seven seconds (56s serial total); engine health verdict is running plus delay, not actual readiness. Login starts first, followed by serial Chat/ClientWeb/Market/PFEs/Portal; each new PFE starts before old same-role PFE stops. Buildx builder recreated but early layers cached: this does not establish a fully cold cache. Kamal time beyond build/push is about 144.8s, including pull/orchestration/hooks/waits/cleanup, not measured player downtime. User confirms Ergo should keep automatic develop deployments with the same staged system; production engine and portal releases use independent tags. Minimum gameplay warning interval is 60 seconds. Distinguish build duration, rollout downtime, and warning interval before setting targets. Desired direction: parallel builds of Playfield/Login/Chat, explicit readiness-gated startup (all required PFEs, then Chat, then Login), immutable engine release tags rather than main-branch deployment, and separate portal release tags with a fast path that never restarts gameplay engines. Reuse current PRK.CLI notifications and existing client tag-release conventions where appropriate. Inspect MarketEngine and ClientWebEngine dependencies so migration does not drop existing roles. Source inspection: root Dockerfile COPYs all source before sequential publishes of PFE, Login, Chat, Portal, ClientWebEngine, Market, CLI and navigation smoke; all ship in one image. Portal-only redeploy still builds that shared image. No measured CI/host phase timings collected. config/deploy*.yml defines four production/two staging PFEs plus Market/ClientWebEngine; only Portal has a Docker HTTP health check. PFE metrics starts before Run/cache initialization, so metrics response is not readiness. LocalPlayfieldInstanceLocator advertises LoginMode.Open in its constructor and publishes states every 30 seconds; Login maintains a heartbeat-populated locator. Gate Login admission on fresh expected-release role identities, not merely running containers or old state. PFE startup runs migrations in each process; migrator lacks a cross-process lock and catches/rolls back SQL errors without rethrowing. Parallel rollout needs singleton/locked, fail-closed migration execution. Existing warning reuse: PRK.CLI announce --notify and restart --alert-only, currently best-effort via live Login container. Graceful save/drain/SIGTERM handling must be verified, not assumed. Portal directly references AOEmu.PlayfieldEngine and uses item/data/providers; independent deployment is possible before extracting narrowly shared code to improve build isolation. Preserve generated Tailwind/three.js publish assets. Selected implementation direction: Forgejo tag workflows, separate immutable per-service Docker images and digest-pinned release manifests, parallel BuildKit builds with deliberate restore/source caching and runner capacity limits; lightweight SSH-invoked host orchestrator plus Compose service definitions, existing Traefik/networking retained. Build/test/push/pre-pull before warnings and disruption. Warn/count down, close old Login admission, gracefully save/stop old gameplay cohort, run migrations once, start required PFEs with bounded startup concurrency and full readiness, then Chat, then Login with fresh discovery gate and public smoke check. Portal separate tag/image/rollout/lock; no gameplay warnings/restarts. Runtime failure keeps Login closed with diagnostics; rollback only for schema-compatible releases. Avoid simultaneous old/new stateful PFEs until proven safe. Confirmed policy: Ergo automatically deploys develop using the same improved staged orchestration; Ithaca uses release tags, with independent portal tags. Guarantee at least 60 seconds between successful warning publication and disruptive gameplay changes, after images/preflight are prepared. Implemented tag names engines_vX.Y.Z and portal_vX.Y.Z. Warning publication requires broker confirms and aborts before disruption on failure; this is not client-display/PFE receipt acknowledgement. Market drains accepted handlers with broker responses kept open and starts after PFEs; Chat and ClientWebEngine follow, then Login. Initial cutover requires maintenance/legacy shutdown and an explicit bootstrap; no live cohort may bypass warnings merely because its journal is missing. Benchmark build/push/pull/startup/readiness/downtime independently, and test partial PFE/migration/chat/discovery failures, stale releases, locking, cancellation and schema-safe rollback on Ergo before production. Implementation authorized: deliver a proper, simple replacement rather than a prototype. Use parallel cached service Docker builds, Compose definitions, one host deployment script, shared engine readiness/lifecycle helpers, production engines_v*/portal_v* tags and automatic develop deployment to Ergo. Preserve existing infrastructure and notifications. User now authorizes preparing and cutting over Ergo, including test-server restarts without a maintenance window. Current morning cutover decisions: user posted a 15-minute maintenance notice and selected exact source a52e28118173bcb1d813446eabf5fa8297e44129 (not main's merge commit and not newer develop). Proposed first release tag is engines_v0.1.0; no tag has been created. User also explicitly requests gracefully restarting the existing MSSQL container during cutover after old gameplay stops, preserving its image/volumes and waiting for SQL readiness before starting the new engines. This is a one-time manual cutover step, not a new automatic DB restart policy. SQL restart may briefly affect portal DB requests and cold caches may extend startup. Keep RabbitMQ/Traefik/VPN up, do not switch the portal release, do not add rehearsal/backup gates, and do not cut over before the notice period and go-ahead. Verified agent-only remote_python access via configured ithaca/mathias and passwordless sudo. Production currently runs eight gameplay roles at d9a92377 and portal at d6c85c7a, all healthy/running; Docker 28.1.1, Compose 2.35.1, Python 3.10.12, x86_64, registry login and required GameData/reporting/key mounts present. Docker disk has 347.1 GiB free and host memory about 42.7 GiB available; load about 19 (no resource-heavy production build should run). At initial inspection the new environment files, managed release state and daemon.json were absent; prerequisite files are now prepared as detailed below, while managed release state remains absent. SSH Ed25519 host public key obtained over the trusted remote channel. Candidate image references from successful Ergo run 5072 at revision 49936b1d2 were identified for cache-only pre-pull, not production selection. No PFE/Market SQL migration changes between currently deployed d9a92377 and candidate 49936b1d2; no engine/portal release tags currently exist. Ithaca non-disruptive preparation is complete. Access uses full hostname ithaca.project-rk.com as mathias with the configured agent and passwordless sudo; the user removed the broken legacy aliases, and no agent/SSH settings were changed by the assistant. Temporary non-secret source-transfer directory was removed after validation. Prepared root-owned mode-0600 /etc/prk/ithaca/engines.env and portal.env from the current running containers, excluding image defaults and managed deployment fields. Both settings sets survived Compose parsing byte-for-byte and both actual Compose definitions rendered successfully on Ithaca. Preserved production Login concurrency=8, public ports/network mapping, current portal DB fallback (SITE_DB is empty, so AOEMU_DB_URL remains its existing site DB), reporting config/uploads and DataProtection key mounts. No secrets printed or copied off-host. Saved private /srv/prk-deploy/ithaca/preparation/legacy-inventory.json containing all nine existing app container IDs/images/start times, infrastructure IDs and Compose definition hashes for recovery. Cached and identity-verified all six runtime images from successful Ergo run 5072 at revision 49936b1d2 in 56.5 seconds. Cache metadata is in validated-ergo-images.json and explicitly says selected_for_production=false; it is not an executable production release manifest. Final revision/tag remains unchosen. No PFE/Market SQL migration-file changes were found between currently running d9a92377 and this candidate; rerun the comparison if a different source is selected. Configured /etc/docker/daemon.json shutdown-timeout=240 with validated live reload; daemon PID and every running container ID stayed unchanged. Extended repository DEPLOY_KNOWN_HOSTS to contain both trusted Ergo and Ithaca Ed25519 pins. Existing registry credentials pulled every new private runtime image successfully. Staged and validated /srv/prk-deploy/ithaca/preparation/notice-compose.json for the cached broker-confirming CLI; it was not run. Use this Compose definition for first-cutover notices so quoted/interpolated env-file values are parsed correctly; do not pass these Compose env files directly to docker run --env-file or source them as shell scripts. Checked SQL backup-history metadata using the existing configured DB credential: nightly full IthacaDb backups are recorded, latest 2026-10-19 02:30:23, about 12.3 GB logical / 3.14 GB compressed, basename ithacadb_19.BAK. Saved redacted metadata in preparation/backup-metadata.json. Initial query failed due to sqlcmd's mutually exclusive -h/-y0 flags (confirmed with a parser-only help diagnostic); corrected flags succeeded with the same credential and no authentication fallback. No backup, restore, restore-verify or migration was run. User explicitly declined an additional backup or restore rehearsal for tomorrow's deployment-only transition and accepts one final legacy deploy progress rollback. Do not add ad-hoc backup/restore verification as a cutover gate. Existing nightly backups remain available. Recheck the selected revision's migration list/ledger; an unexpected schema change or failed startup is a genuine issue to surface, not a reason to silently expand the transition. Final verification: all nine existing app container IDs/start times/images remain unchanged and running; existing portal is healthy, infrastructure IDs unchanged, no new managed production containers or current-release state exist. Public portal home/status/protected API/HTTPS OIDC callback checks pass. Working tree is clean; no release tags created and no deployment or unattended activation scheduled. Agreed transition is deliberately minimal: fresh go-ahead with the selected source, confirm no competing legacy deploy, honor the posted notice and at least a 60-second in-game warning, stop the old gameplay cohort, gracefully restart the existing MSSQL container and await readiness, bootstrap the new system, then verify staged readiness and login. Read-only morning preflight succeeded after user explicitly authorized rerunning an interrupted attempt: all four legacy PFEs/core apps are running, no managed production release or Kamal deploy lock is present, game ledger has 87 entries against 86 current files with zero pending, and Market has 17 entries/files with zero pending. Source a52 has no PFE/Market migration-file changes relative to deployed d9a92377. That preflight was read-only. User subsequently gave cutover go-ahead around 06:23 CEST, requested old-engine shutdown roughly 06:25, and selected a52e28118173bcb1d813446eabf5fa8297e44129. Created/pushed annotated engines_v0.1.0 at exactly that commit. Manually dispatched bootstrap=true on that tag: API run 5280 / UI https://git.project-rk.com/prk/engines/actions/runs/5130; source SHA and input verified. Tag push also triggers its ordinary non-bootstrap run; do not confuse it with 5130. Broker-confirmed in-game announcement and alert-only 60-second countdown completed at 2026-10-20 04:25:51 UTC (06:25:51 CEST), recorded at /srv/prk-deploy/ithaca/preparation/cutover-notice.json. Honored at least 60 seconds before shutdown. Acquired the same engines/deploy.lock as the new orchestrator while preparing the legacy/SQL transition. Stopped the eight old gameplay containers at 04:28:38–04:28:42 UTC, retaining them; portal/RabbitMQ/Traefik were not stopped. Began graceful Docker stop --timeout=-1 of existing engines-db container b4689ac77cc33f8c9db55e0991a2fdde7e5bfc7528715ff878c68cf9da11be64 at 04:28:42 UTC. Its CLI wait exceeded the 90-second subprocess limit. No SIGKILL, new SQL start, backup/restore, or stop retry was issued. Remote script exited with TimeoutExpired (not an SSH/auth failure), releasing the host lock; the Docker daemon stop may still be pending. Progress marker remains at /srv/prk-deploy/ithaca/preparation/cutover-progress.json, stage Gracefully stopping MSSQL. Do not rerun the whole cutover script or automatically retry stop/start. Asked user for permission/help to inspect Docker/SQL shutdown state and logs before further action. Ithaca engine cutover completed successfully. Approved annotated tag engines_v0.1.0 points exactly to a52e28118173bcb1d813446eabf5fa8297e44129. Bootstrap=true run 5280 / UI https://git.project-rk.com/prk/engines/actions/runs/5130 passed prepare, 37 release-gate tests, all six image builds/cache exports and deploy; workflow ran 04:26:52–04:37:39 UTC (10m47s), deploy job 04:36:01–04:37:39 (1m38s). Duplicate automatic tag run 5282 / UI 5132 was cancelled before deploy; an attempted API DELETE returned 405 and no unsupported cancellation endpoint was retried. Final in-game broker-confirmed warning completed 04:25:51 UTC and gameplay shutdown began 04:28:38, exceeding the minimum notice by about 167 seconds. Eight legacy gameplay roles stopped by 04:28:42 and remain retained. Per explicit user request the same engines-db container was gracefully stopped/restarted: Docker stop --timeout=-1 outlived a 90-second CLI wait and the remote script exited without SIGKILL/retry; inspection showed the same DB container had restarted at 04:31:48 and completed IthacaDb recovery by 04:31:56. Existing portal was then stopped to eliminate DB request pressure, IthacaDb/ledger readiness verified (87 entries), and the same portal restarted healthy at 04:33:32. RabbitMQ/Traefik/VPN stayed running; no backup/restore/rehearsal or SQL migration was performed. Progress marker updated to SQL online/portal healthy/bootstrap ready. Managed services started in the required order: four PFEs at 04:36:51 UTC, Market 04:37:13, Chat/ClientWebEngine 04:37:23, Login 04:37:30. Independently verified all eight new gameplay containers healthy, exact manifest digests, release/role identities and zero restarts. Confirmed all four PFEs before Market and Chat/ClientWeb before Login. Public TCP 7000, 8501–8504 and 7005 reachable; Chat greeting received; public portal status HTTP 200. current.json and attempt Release healthy recorded with no command-failure log. Legacy gameplay containers remain stopped/retained, old portal remains running/healthy, and infrastructure IDs remain unchanged. MSSQL is running at the post-restart StartedAt with preserved image/mounts. Actual gameplay downtime from old cohort stop to new Login start was approximately 8m52s, dominated by waiting for the first production image build/tag run after shutdown; deploy itself was 1m38s. Future normal tagged deployments build/pull before warning and disruption. User verified real-player Ithaca login and zoning work after cutover and accepted the first-cutover downtime. Chat and player/pet/mercenary/NPC-trade persistence under production managed restarts remain optional follow-up validation; production portal has now moved to its independent tagged deployment system. User accepts one final legacy progress rollback; do not require an empty server, player-by-player save/logout proof, a rehearsal, an extra backup or restore drill. Keep existing nightly backups and old containers/images as fallback. Leave portal, database, RabbitMQ, Traefik and VPN running, and do not activate anything unattended tonight. User separately authorized a Git host registry-only Nginx change to resolve HTTP 413 on cache-layer uploads. Acceptance: deterministic image/digest release identity; at least 60-second warning before disruption; graceful save/stop; singleton fail-closed migrations; PFEs then Chat then Login with current-release discovery; independent health-gated portal rollout; bounded failures/locking/retained prior releases; meaningful automated tests and build validation. Production/staging rollout and measured latency require an authorized operational validation later. Keep implementation small; no framework, service, Kubernetes or elaborate shared compilation pipeline. Implementation in feature/staged-docker-deployment: deployment/ contains Compose definitions, a parameterized cached Dockerfile, host-local Python orchestrator, small release/upload scripts, test suite and provisioning/cutover README. New release-services workflow replaces the two branch-only Kamal deploy workflows. Shared private readiness and signal lifetime added; PFE locked/fail-closed migrations and awaited save/NPC/pet drain; Login fresh-role/release discovery gate; Market request drain; standalone CLI broker-confirmed warnings. Per-scope host locks, run/attempt high-water ordering, digest manifests and retained bundles; explicit rollback reconciles containers even if successful history matches; portal uses healthy TCP router priorities and preserves a durably committed candidate after interruption. Four patch-scoped static reviews exposed shutdown/recovery defects that were addressed with regression coverage; this is not production-readiness approval. The last fixes have targeted test/build validation but have not had another independent static review. Shared PendingOperations now joins Market requests, entrance restoration and instance creation. PFE shutdown also joins pre-existing disconnect callbacks, retains failed retiring playfields, tracks NPC grants beyond chat-window lifetime, propagates reward-command exceptions, drains queued pet spawns with sleeping instances forced awake, and preserves staged/deferred pet/mercenary snapshots across save retries and disconnect handoff. Required shutdown saves reject staged-trade deferral. Docker stop uses --timeout=-1 with a bounded CLI wait, suppressing restart loops without requesting SIGKILL; a disposable integration test verifies this behavior. Final local validation passed: 17 Python rollout/manifest/failure regressions; 14 focused C# readiness, operation-drain, retained-save, deferred-pet-preparation and actual SIGTERM child-process tests; 352 selected engine regressions including real throwing NPC reward, closed-window work retention, retiring-instance retention, sleeping-instance dispatch and confirmed broker warnings; 27 legacy deployment regressions. All seven per-role production Docker images built successfully, including portal generated-asset assertions and the Playfield native navigation smoke. Both Compose files rendered successfully; Python/workflow YAML/shell syntax and git diff whitespace checks passed. Disposable MSSQL integration passed concurrent singleton migration execution, nonzero failed migration and atomic rollback. Disposable Traefik v3.7.7 TCP test passed starting-candidate exclusion, healthy priority cutover and old-slot fallback. A disposable Docker stop test passed restart-loop suppression and a slow successful drain surviving CLI timeout with no SIGKILL escalation. Generated Avalonia test artifacts were removed. No registry publication or deployment-host contact occurred during local implementation validation. Ergo prerequisites provisioned and verified: /etc/prk/ergo/engines.env created root-owned mode 0600 from the six current gameplay containers, preserving required/shared application settings without exposing values. Compose parsing preserved every value unchanged; the actual deployment Compose definition rendered successfully with this host env file. /etc/prk/ergo and /srv/prk-deploy/ergo directories are private. Existing root private-registry access verified against the running image. /etc/docker/daemon.json now sets shutdown-timeout=240; dockerd config validation and live SIGHUP reload succeeded, and the daemon journal confirms 240 with unchanged daemon PID/container IDs. Repository CI secret DEPLOY_KNOWN_HOSTS created with Ergo's Ed25519 host key obtained over the trusted SSH connection. Existing SSH_PRIVATE_KEY/registry secrets retained; no SSH/agent changes, private keys copied or credentials printed. Host prerequisite preparation did not restart containers. During the subsequently authorized gameplay cutover, the six old gameplay containers were stopped and retained after CLI announcement/in-zone warnings plus a full 60-second wait and a second zero-player check. SQL/RabbitMQ/Traefik and the stale Ergo portal remain running, unchanged. Ithaca has not been contacted or changed, and its host pin/env provisioning remains future scope. Current operational validation: user merged PR #2748 into develop as 6e5bf3bec22044a215d1939e4a6af9e555d94aa7 and cancelled the initial automatic release run (API ID 5201 / UI run 5067). Confirmed no active deploy lock/process and zero players on both PFEs before legacy shutdown. Recovery metadata is retained at /srv/prk-deploy/ergo/legacy-cutover.json; old gameplay containers/images were not deleted. Dispatched bootstrap=true for release-services.yml on develop: API run ID 5202 / UI https://git.project-rk.com/prk/engines/actions/runs/5068, inputs and source SHA verified. Tea's dispatch command reported unexpected end of JSON after the server accepted it; checked the created run and did not dispatch a duplicate. Bootstrap run 5068 failed before deployment: engine_tests failed because ClosingWindowCannotOrphanAnAcceptedGrant constructed PlayerCharacter with a null EngineInstance.Instance, and all six image jobs failed during registry cache export with Nginx HTTP 413. Deploy was skipped. The user cancelled an attempted legacy recovery and explicitly wants Ergo kept off; subsequent read-only inspection confirmed only SQL, RabbitMQ, Traefik and the old Ergo portal running, with no outstanding recovery/deploy process. Do not restore gameplay containers automatically. Fixture correction is local on fix/deployment-validation-and-cache-uploads, based on origin/develop: DeploymentNpcWorkTests now explicitly initializes a mock IServiceProvider before each test, restores previous global service state afterward, and is non-parallelized. Reproduced the failure with that single test in a fresh test host; after correction the isolated test and the exact 37-test release-workflow selection pass. No test gate or cache export was disabled. User explicitly authorized pushing the fixture correction directly to develop. Committed on dedicated fix/deployment-validation-and-cache-uploads as 49936b1d2 (fix: Isolate deployment NPC work test services) and fast-forward pushed HEAD:develop without rewriting history. Working tree is clean. The push can run the ordinary release workflow, but bootstrap=true still requires a manual dispatch; no new bootstrap was dispatched as part of the push. Git registry proxy correction is applied and verified on git-prk-server using remote_python with its existing configured root identity. /etc/nginx/sites-available/forgejo had a 300 MiB vhost limit; added a /v2/-only location with unlimited request size, unbuffered HTTP/1.1 upstream upload and identical forwarded headers. Ordinary web/Git limit remains 300 MiB. Backed up at /root/prk-deployment-backups/forgejo-nginx-20261019T173550Z.conf. nginx -t and live reload passed with the same master PID. A synthetic unauthenticated request declaring 400 MiB without sending a body reproduced 413 before the change; after worker reload the registry returns 401 (authentication still required), while a non-registry request still returns 413. Initial immediate post-reload probe hit an old worker and triggered safe rollback; reapplied with bounded polling, observing [413, 401], and verified both route limits. After publishing fixture fix 49936b1d2, automatic push run 5071 (API 5205) passed engine_tests and all six image/cache-export jobs, confirming the registry fix with actual cache uploads. Its deploy job stopped at the intended first-deployment bootstrap guard; no managed release was created. Cold image jobs took about 5–7 minutes each, with roughly 14 minutes overall to deploy preflight. Ergo is successfully running the new deployment system at source 49936b1d2cf14829206658993c23dfc4a6902d4b. Manual bootstrap=true run 5206 / UI https://git.project-rk.com/prk/engines/actions/runs/5072 passed all gates and deployment. Workflow ran 17:54:57–17:58:35 UTC (3m38s); deploy job took 1m22s. Warm image jobs took 42–62 seconds each. Observed both PFEs healthy before Market started, then Chat/ClientWebEngine healthy before Login started. Independently verified all six containers' exact image digests, release/role readiness identities, healthy status and zero restarts. Public TCP 7000/7002/7003/7005 is reachable, Chat sends its greeting, and public ClientWebEngine /health/ready reports the correct release. SQL/RabbitMQ/Traefik identities unchanged; old gameplay containers remain stopped/retained and the old Ergo portal is untouched. Also replayed the exact successful CI-validated retained bundle directly on Ergo without bootstrap or image rebuild, exercising the normal deployment path. It succeeded in 122.1 seconds: warning at 18:02:39 UTC, Login/drain began 18:03:42 (63 seconds later), migrations 18:03:46, PFEs 18:03:51, Market 18:04:08, Chat/ClientWebEngine 18:04:25, Login 18:04:33, Release healthy 18:04:40. Confirmed again all six containers healthy with correct digests/release and zero restarts, staged start times, public TCP endpoints and public web readiness. This verifies empty-server warning publication, minimum warning duration and graceful managed restart; it does not verify client-visible notification delivery or real character/NPC-trade/pet persistence. No further code changes or production rollout performed. User verified real-client Ergo login is working after cutover. A later, separately tracked Ithaca portal edge incident was resolved without backend/game/VPN restart: the edge reused a stale upstream TCP connection returning 404 while fresh backend connections returned 200. Portal and reporting-upload transports on 5.161.243.214 now disable upstream idle keep-alive (maxIdleConnsPerHost=-1), with exact-version Traefik 3.1.7 fixture and public HTTP/static/auth validation. This edge configuration is a relevant prerequisite for future portal TCP blue/green switching, but the new Ithaca engine/portal release paths remain unactivated. Independent portal release completed successfully. Tagged portal_v0.1.0 at the same approved a52e28118173bcb1d813446eabf5fa8297e44129. Automatic run 5284 / UI 5134 built and exported the initial portal/cache successfully in about 6m54s, then failed only at the intended first-deployment bootstrap guard while the legacy portal stayed online. Per user instruction no player warning was sent. Stopped and retained the healthy legacy portal at 04:48:41 UTC without touching gameplay/infrastructure, then dispatched bootstrap=true run 5285 / UI https://git.project-rk.com/prk/engines/actions/runs/5135. Warm portal build took 39 seconds; full workflow 04:48:47–04:50:30 UTC (1m43s). Managed blue slot started 04:50:11, yielding about 89.6 seconds from old stop to new process start. Independently verified healthy status, exact immutable digest/release/role readiness identity, blue slot, zero restarts, Release healthy state and no command-failure log. Legacy portal remains stopped/retained. Public home, /api/status, protected forms API, OIDC callback, CSS and /health/ready all pass; public readiness reports portal_v0.1.0. All eight managed gameplay containers remained healthy. Edge stale-connection prevention was already active, so external cutover observed the new release correctly. Core implementation and production engine/portal cutovers are complete. Optional remaining validation: client-visible warning delivery and player/pet/mercenary/NPC-trade persistence during a future managed restart, plus representative failure/recovery drills if desired. Do not block normal operation on those deferred checks. User explicitly declined retaining rollback assets and authorized final live cleanup. All stopped legacy app containers, obsolete local PRK images, non-current release bundles, cutover/preparation snapshots and per-host /root/.kamal state have now been removed from Ergo and Ithaca. Only current engine/portal bundles and manifests remain. Current services, infrastructure, volumes, registry credentials and compatibility network remain intact; final public/health validation passed. No remote registry/cache deletion was performed. Do not claim production readiness or measured routine full-CI normal-deploy time from this bootstrap and host-only replay. Regular merged-branch engine CI passed while its client job failed separately; investigate only if relevant to this engine release. Remaining: verify bootstrap success, then verify actual clients see the minimum 60-second warning, character/pet/NPC-trade persistence, readiness ordering and recovery under representative game data/load. Confirm the installed production Traefik version/health filtering and portal cookie/key continuity; measure build/pull/startup and actual player downtime. Ergo host preparation/test cutover is authorized; no maintenance window is required for this test server. Access restored after the user explicitly specified root@ergo.project-rk.com. Use the same configured SSH agent; no SSH configuration or credentials changed. Normal SSH transporting Python on stdin is used because remote_python cannot override the login username. Read-only inspection verified the Ergo hostname/root account, Python 3.10.12, Docker 28.3.2, Compose 2.38.2, curl, existing private registry login, kamal network and GameData mount. Host env file is provisioned; managed release state is not yet recorded during the active bootstrap. Six legacy gameplay containers/images at f60c1d59 are stopped and retained for recovery; an older Ergo portal remains running and is outside this gameplay cutover. SQL, RabbitMQ and Traefik remain untouched. PR #2748 is merged into develop; Ithaca/production activation has not been performed. Keep overall work in progress until operational acceptance is resolved; source implementation and local validation are ready for user review. Pull request: #2748 — https://git.project-rk.com/prk/engines/pulls/2748 Title: feature: Replace Kamal with staged Docker releases Head/base: feature/staged-docker-deployment → develop (recorded repository default). Published two session-only commits without rewriting history: 84991aaeb (engine readiness and graceful drains), 16d5bb6d2 (staged Docker release orchestration). Working tree clean; the branch is three existing develop commits behind the fetched base with no overlapping changed files. Python rollout/legacy suites and shell/whitespace checks rerun before commit. PR is open for review, not a production activation; task status remains unchanged.
Checklist
9 of 9 complete- Implement and test real engine readiness, safe migration execution, and graceful Playfield shutdown.
- Implement Compose and host orchestration with warning window, staged startup, locking, failure diagnostics, and independent portal replacement.
- Add parallel cached Docker builds and tag/develop workflows with immutable release manifests.
- Run builds, deployment regressions, review, and document safe provisioning/cutover and validation limitations.
- Validate Forgejo execution and Ergo bootstrap/normal redeployment: image identities, readiness order, public endpoints, minimum warning duration, graceful restart and measured timings.
- After separate authorization, provision and validate the independent Ithaca portal release, cookie/key continuity and failure recovery before replacing the retained legacy portal.
- Prepare Ithaca non-disruptively: private env/config validation, trusted CI host pin, runtime-image cache, rollback inventory, notice configuration and backup-history check; verify production services unchanged.
- Provision and validate Ithaca engine deployment: release tag, player warning, legacy stop, requested MSSQL restart, staged bootstrap, immutable image/readiness verification and public endpoint checks.
- Verify real-player Ithaca login and zoning under the managed release.
- Area
- Project Rubi-Ka
- Project
- engines
- Kind
- Feature
- Task ID
tsk_ah9pynd33g- Updated