Spencer Morris スペンサー・モリス← the streetLet's talk
SMSL

One Box

FIELD REPORT · 18 MIN
Execution / field report

Moving Storage Levers off one server

For two years one rented server ran the pricing, phones, scheduled jobs and staging for my storage pricing company. On March 16, I moved all of it to Kubernetes in its own AWS account, using a team of AI coding agents to do the work in parallel. Production stayed up. This report covers the design, how I ran the move, and why each choice was made.

ExecutionMar 15 to 16, 2026EKS · Argo · Kafka · OpenTofuThe product ↗
Argo CD showing 24 applications, all Healthy and Synced
Argo CD the morning after: 24 applications, all Healthy and Synced.
preflight to verified cutover
server errors after the switch
cloud resources inventoried in git
phone gap, scheduled in advance

At 18:44:24 UTC on March 16, the health check for fastapi.storagelevers.com returned a deployment_id of fastapi-79cfc4c67d-m4znl, the name of a Kubernetes pod. Until then, every response from that address came from one rented Linode server. That server priced every unit our clients rent, answered their phones, ran 46 cron jobs, hosted staging, and also hosted five side projects for other businesses. Its disk was 88% full.

Users did not see the switch. The server's reverse proxy started forwarding traffic to a load balancer in AWS, and requests already in progress finished on the server. An hour earlier, the new cluster ran only a read-only copy of production. The night before, the cluster did not exist.

I am the only engineer at the company. I designed the target platform and the cutover, wrote the rules the work had to follow, and split the build across several AI coding agents running in parallel overnight. The next day I ran the cutover step by step, with an agent executing each step and reporting to me as it went. The hard part was not Kubernetes. It was moving a system that had grown without a complete record of what it ran.

1 One box

The server

The server ran Traefik as its reverse proxy. Behind it ran the production API as two Docker Compose stacks, blue and green, which a 600-line Python script swapped on each release. Next to the API ran a pricing service, a data explorer, Redis, a cron container with 46 crontab lines, a worker that sends the daily summary email, staging, a shared development environment with its own Postgres, and the side projects. The host also had 2 lines in root's crontab, 11 files in /etc/cron.d and 15 systemd units.

The rest ran in AWS, in the organization's management account, which also runs a second business. That account held the AI phone agent on Fargate, a marketplace worker, an ads worker, a review scraper and 14 scheduled data fetches. All of it used one access key that never expired. My laptop, the agents and production all used that key.

The immediate problem was CI cost. In five days we ran 70 releases and were billed about 1,980 GitHub Actions minutes. A 28-minute release billed 55 minutes, because it ran 14 jobs and GitHub rounds each job up to a whole minute. More than half the runs failed, many after 20 minutes, so most releases were paid for twice.

I wanted the setup used by teams whose systems cannot go down: deploys driven from git, signed images, streaming data pipelines, a separate identity for each workload, and central logs and metrics. Some of it is more than a company our size strictly needs. But the server was a single point of failure with a nearly full disk, and I could not list everything it ran. I set two constraints: production stays up, and the Neon database and Vercel frontends do not move.

The platform before and after the migrationBEFORE · 15 MARONE RENTED SERVER · 6 vCPU · 16 GB · DISK 88%traefikevery hostnameprod api, blue/greena 600-line scriptcron container46 crontab linespricing · explorer+ redisstaging · shared devown postgres5 side projectsother businesseshost crontab · systemd timers · local filesstate that existed only on this diskAWS MANAGEMENT ACCOUNT, SHAREDECS: phones, workers14 scheduled fetchesone admin key: me, agents, prodAFTER · 16 MARDNS → load balancer + WAF → EKSno server of mine in the request pathEKS · ITS OWN AWS ACCOUNT · 5 NODESapi, canaryargo rolloutsworkers · phonespublic voice node47 cronjobsone per old linestaging · shared dev+ side projectskafka → parquetchange data capturemlflow · workflowspricing calibrationargo cd · kyverno · grafanagit is the record; pods get roles, not keysEFS · S3 · Secrets Managerstate that used to live on one diskold server: idle, kept for rollbackThe frontends (Vercel) and the database (Neon) did not move.
Before and after. Everything on the right is defined in git.
◇ One box

The design

Before any agent started, I wrote down the target and the reasons for it. Eight principles decided almost every choice that followed.

Separate the blast radius. Storage Levers gets its own AWS account, so nothing it runs can touch the other business in the organization. Every workload gets its own IAM role through Pod Identity, and nothing in the cluster holds a long-lived key.

Buy the parts where failure is expensive. AWS runs the Kubernetes control plane, the load balancer, TLS and rate limiting. Neon keeps the database and Vercel keeps the frontends, because both already worked. I run only what is specific to this product.

Make git the only way in. OpenTofu defines the account, Kustomize defines the cluster and Argo CD applies it. CI builds and signs images but never holds cluster credentials, so every running version traces back to a commit.

Keep the old judgment. The blue/green script's read checks encode two years of knowledge about what a healthy release looks like for this product. They became the canary's analysis steps unchanged.

Keep state off the machines. The server's files go to EFS, Kafka and Redis keep their data on volumes that outlive their pods, and everything durable goes to S3. Any pod can be replaced on any node.

Build for the data we will need. Debezium streams database changes through Kafka into Parquet, so pricing models train on data that can be reproduced. Kafka runs on one dedicated node through Strimzi: a broker restart only delays change capture, and a managed cluster would cost about six times as much.

Pay for observability, not for operating it. Grafana Cloud stores metrics, logs and traces, and a small agent on each node ships them. There is no monitoring stack of our own to keep alive.

Make the cutover boring. A read-only shadow copy proves the new platform before it serves anyone, an inventory in git proves nothing is left behind, and DNS moves last, so every step can be undone in seconds.

These choices cost more than the server. At on-demand rates, the new account would cost about $855 a month, against about $450 for the server and the cloud services it replaced. After a month of stable running, I bought a one-year Savings Plan, bringing the estimated monthly cost to about $710. I accepted that for a platform where no single disk, process or credential can take the company down.

Monthly hosting cost before and afterMONTHLY HOSTING, USD · FRONTENDS AND DATABASE EXCLUDEDbefore: server + shared ECS$450after: on-demand rate$855after: 1-year plan in place$710floor: no Kafka, ARM nodes$580
Monthly cost estimates. The one-year Savings Plan brought the estimate to $710; $855 shows on-demand pricing.
2 One box

The build

I wrote one shared document for the agents. It listed my hard rules first: do not change the server, do not change the running cloud services, treat the production database as read-only except for one replication role, never commit a credential, and never poll more often than every 30 seconds. Then I split the work: one agent for the platform, one for each group of workloads, one for CI, one for data, and one to merge their work, with me reviewing the merged result.

By morning the platform I had specified existed: a new AWS member account, built with OpenTofu, running EKS on Kubernetes 1.35. It has three general nodes that can scale to five, one node reserved for Kafka, and later a small node in a public subnet for the phone gateway. No pod has an access key. Each workload has its own IAM role through EKS Pod Identity. Workloads that still read buckets and queues in the old account assume one limited cross-account role. Secrets stay in AWS Secrets Manager, and External Secrets copies them into the cluster.

Argo CD deploys everything from git. Production follows a branch named k8s-release, which is main plus one commit that sets image tags. Releasing means rebuilding that branch. Argo Rollouts replaced the blue/green script with a canary deploy that runs the same read checks. Kyverno checks image signatures and reports violations without blocking them yet. Grafana Cloud collects logs, metrics and traces. Strimzi runs Kafka, and Debezium streams 18 production tables into S3 as Parquet files, which gives the pricing models a training set that can be reproduced. Argo Workflows runs the pricing calibration and records each candidate in MLflow. A candidate can change live prices only through a pull request.

I also restructured CI to cut the bill. Cheap checks run first, so a lint error fails in one minute. Passed checks are recorded in S3 under a hash of the code, so a release that does not change backend code skips backend tests that already passed. Images are built once, signed, and copied to the new registry.

Storage Levers / after the move

Production, data and delivery

Storage Levers AWS account Existing external service
01

Production traffic

  1. Managed edgeVercel + API GatewayDashboard, sites, marketplace REST + WebSocket.
  2. AWS / public subnetsLoad balancer + WAFACM certificates, host/path routing, request limits.
  3. EKS / private subnetsAPI ServicesFastAPI + marketplace API. Stable Services in sl-prod and sl-workers.
  4. Managed databaseNeon PostgresExisting production database, unchanged.
Phone media takes a separate path
  1. Phone networkCallers via GoToSIP registration + WebRTC media for store phone lines.
  2. EKS / public subnetDedicated voice nodePublic IP + host network. Media bypasses the ALB and NAT.
  3. EKS / private subnetsSame FastAPI serviceBusiness API calls only. Audio stays on the public voice node.
Outbound work: CronJobs and workers reach property systems through the NAT gateway. Legacy S3, SQS and secrets access uses a limited cross-account role.
02

Data and model candidates

  1. Change sourceNeon18 tables via logical replication.
  2. EKS / Kafka nodeDebezium + KafkaKafka Connect + 3 KRaft brokers, managed by Strimzi.
  3. AWS / object storageS3 ParquetReproducible training datasets.
  4. EKS / platformArgo WorkflowsRefit + evaluate pricing parameters.
  5. EKS + S3 artifactsMLflowParameters, metrics + model artifacts.
Review gate: A candidate changes live prices only through a pull request and the release path below. Kafka is outside the customer request path.
03

Code to production

  1. CIGitHub ActionsCached checks, build once, sign with cosign.
  2. AWS / image registryECRSigned container images.
  3. GitHub / k8s-releasePin image in gitmain + one image-tag commit.
  4. EKS / platformArgo CDPull git, apply manifests, reconcile drift.
  5. EKS / appsArgo RolloutsCanary + analysis Jobs. Promote or abort.
Two separate pulls: Argo CD reads manifests from git; pods pull images from ECR. Customers stay on the stable Service until analysis passes. CI holds no cluster credentials.
Shared foundationOne AWS member account, defined in OpenTofu
Compute and environments
General pool: 3 to 5 m6i.xlarge nodes. One m6i.large for Kafka; one public t3.medium for voice. Staging, shared dev and side projects have separate namespaces.
Identity, secrets and state
One IAM role per workload via Pod Identity. External Secrets reads Secrets Manager. EFS holds server files, EBS holds Kafka and Redis data, S3 holds artifacts.
Policy and observability
Kyverno audits image signatures; Policy Reporter shows results. Alloy sends metrics, logs and traces from the cluster to Grafana Cloud.
Read each path in arrow order. Public and private labels show network placement; dashed boxes mark existing services outside the new account. Vercel and Neon stayed in place.

Select a layer below to see what it does, what it replaced and why it is there.

Account & identity

AWS member accountOpenTofuEKS Pod Identityone cross-account role
WHAT
Storage Levers got its own AWS account inside the organization, built entirely by OpenTofu with state in S3. Every workload runs as its own IAM role through Pod Identity; the few that still read buckets and queues in the old account assume one narrow cross-account role.
REPLACED
Workloads in the organization's management account, all authenticated by one admin key that never expired and was shared by me, my agents and production.
WHY
A leaked key now exposes one workload's permissions in one account, not admin access to two businesses.

Shadow mode

The rule I relied on to make the overnight build safe was shadow mode. Each production namespace ran an exact copy of the image the server was running, connected to the real database, with every write path turned off. CronJobs were created suspended. Queue consumers ran at zero replicas. The phone gateway also ran at zero, because a second copy takes over the SIP registration and calls stop reaching the first one. Ingress was internal only.

Shadow mode needed one switch in the application itself. On startup the API normally starts its queue consumers and runs a few idempotent DDL statements. I added FASTAPI_BACKGROUND_WORKERS=off to skip both, with a test, so the copy could boot against production without touching it. Removing that variable is the step that turns the shadow into production.

The rest of the night went to the comparison I had asked for. The agents recomputed both pricing engines on the cluster for every client company and compared them with the server. They matched except in one case: for one company the cluster counted 3,676 move-ins and the server counted 3,663 from the same inputs. A new pod running the server's own image also returned 3,676. The server's pricing process had been running for two days and was holding stale data in memory.

⇄ One box

Blue/green and canary

The old release process ran on the production server over SSH. The script waited for running cron jobs to finish, then started a second full copy of the API, called green, next to the live copy, called blue. It ran read checks against green and warmed the pricing cache through it. Then it raised green's Traefik router priority by 200 so the proxy sent traffic there. Blue drained for up to 960 seconds and then stopped. If a step failed before the switch, the script deleted green.

The script worked. The problem was where it ran: the release tool, the proxy and production shared one disk and one pool of memory. Starting green meant running two copies of the API on a server that was already short of both, and the release process needed its own disk cleanup step.

How a release reaches users, before and afterBEFORE · BLUE/GREEN ON ONE SERVER (blue_green.py over SSH)1wait for cronlet runningjobs finish2start greena 2nd full stackon the same RAM3verify reads+ pricing prewarmagainst green4flip traefikrouter priority+200 over blue5drain blueup to 960 s,then stopblue 100%green 0%blue 0%green 100%A failure before step 4 deletes green. Both stacks share one disk, one memory pool and one Redis;the release script, the proxy and production are the same machine, so a full disk stops all three.AFTER · CANARY ON EKS (Argo CD + Argo Rollouts)1CI builds oncetests, sign,push to ECR2pin in gitk8s-release =main + 1 commit3argo syncsthe Rollout seesa new image4canary podready only when theALB says healthy5analysis Jobssame read checks,same prewarm6promotenew becomes stable,old pods go away✕or abortcanary deleted,nothing movedpassfailusers during steps 4 and 5stable service 100%canary 0%The canary has its own Service. The analysis Jobs read through it;the load balancer only knows the stable one, so no customer requestreaches code that has not passed its checks.after promotionold 0%new 100%Same checks as before, now Kubernetes Jobs. Git records every version; Argo CD reverts drift on its own;a rollback is one click on the previous revision. No SSH, and the release runner never touches production.
The release path before and after. The same read checks and pricing prewarm decide whether a version is healthy. They now run outside the production server, and a failed check no longer affects the running version.

I designed the canary to keep the same checks and run everything else off the production machine. CI builds and signs an image once. A release updates k8s-release to that image. Argo CD applies the change, and the Rollout starts one canary pod next to the stable pod. The canary counts as ready only after the load balancer reports it healthy. Two analysis steps then run as Kubernetes Jobs against the canary's own Service: the read checks and the pricing prewarm. The load balancer sends users only to the stable Service, so no user request reaches the canary. If both checks pass, the canary becomes the stable version. If either fails, the canary is deleted.

3 One box

The phone gateway

The phone gateway joins calls over WebRTC. It asks a STUN server for its public address and tells the other side to send audio there. On Fargate the gateway had its own public IP, so that address was correct. In the cluster, pods reach the internet through a NAT gateway. To check whether that would affect calls, I had a STUN test run from a pod, asking two servers from the same local port. Google's server saw port 32,598. Cloudflare's saw port 18,787.

So the NAT assigns a different public port for each destination. The gateway would give the phone system's media server the wrong address, and calls would connect with no audio. The fix was a small node in a public subnet, with the gateway on the host network and source port randomization turned off. The same test then returned port 40,000 for every server.

4 One box

The inventory

The next morning I reviewed the build and decided I was not ready to cut over. I could not be sure nothing would be lost, because nobody had a complete list of what the server ran.

So I started a second phase with an inventory kept in git. One agent recorded every container, crontab line, cron file and non-default systemd unit on the server: 31 containers, 15 units, 11 cron files, 2 host cron lines and 46 lines in the cron container. Each entry names its replacement on the cluster or the reason it needs none. Another agent did the same for the AWS account and found 113 resources that belonged to us. Under my rule that every workload moves, 27 were marked to move, 48 to stay as AWS-native services and 38 to retire. Then I turned both lists into checks. Preflight reads the live server and the live account and fails if it finds anything not on the list. We tested this by deleting entries and confirming that the check failed.

The inventory found three problems before the cutover. A review scraper ran every morning from an EventBridge rule that started a Step Functions state machine, which started a Fargate task. The cutover script did not handle it, so reviews would have been scraped twice. A connection pooler was running that nothing used. And a call-quality job in root's crontab had failed on every run for more than three months, because its image had been deleted.

5 One box

The cutover

I started the cutover early that afternoon Eastern, with time set aside to watch every step. Two other agent sessions were releasing websites at the time. The release pipeline checks that main has not changed during a release, so I held the cutover until both finished.

I built the cutover as a script with seven steps. Each one refuses to start until the previous one has recorded success:

  1. Preflight runs 40 checks.
  2. Freeze stops the server's background work while its API keeps serving.
  3. The cloud step turns off the old services and schedules.
  4. Final sync copies the server's files and the Redis cache one last time.
  5. Activate commits the production settings to git for Argo to apply.
  6. Route sends traffic to the cluster.
  7. Verify checks that each job and service runs in the new place and not the old one.

Each job stops in its old location before it starts in the new one, so nothing runs twice.

Cutover timeline, March 16, 2026, 17:53 to 18:45 UTCTHE CUTOVER · 16 MAR · UTC17:53preflight 40/4017:57freeze17:58old workers off17:59final sync starts18:26activate18:34canary checks pass18:44first request on EKS18:45verify passes
Preflight to verify took 52 minutes. Users saw no interruption.

Preflight passed all 40 checks. Freeze stopped the server's cron container, its background worker and the host cron lines while the API kept serving requests. The cloud step scaled the old ECS services to zero and disabled 14 schedules in under a minute. Final sync copied the server's files to EFS and its 1,572 Redis keys to the cluster, then re-synced the shared development database and routed staging, shared development and the side projects through the server's proxy to the cluster.

Every step is idempotent, and every step records how to undo itself before it acts. Before stopping a container, for example, the script records the container's restart policy, and rollback restores exactly what was stopped. Any step can be rerun safely, and rollback is one command.

6 One box

Activation and routing

Activate turns the shadow into production with one commit. It unsuspends the CronJobs, scales the workers up, turns on the cache prewarm and removes FASTAPI_BACKGROUND_WORKERS=off. Argo applied the commit, the canary passed both analysis steps, and the queue consumers on the cluster logged that they were ready. The consumers on the server could overlap with them for a few minutes, which is safe because every job is claimed with a FOR UPDATE SKIP LOCKED lease.

Route was designed to avoid touching DNS. It started a small nginx container on the server with copies of the API's five Traefik routing rules, at a priority 900,000,000 higher than the originals. Traefik picked up the container and started forwarding requests to the AWS load balancer, and requests already in progress finished on the server. If the new path had failed its checks, the step would have removed the container and traffic would have returned to the server within seconds. The first request was served by the cluster at 18:44:24, and verify passed at 18:45:47. Over the next few minutes the load balancer handled about 100 requests per minute with no server errors.

Argo Rollouts showing the API canary at step three of three, weight 100, analysis passed
The Argo Rollouts canary that replaced the blue/green script. Each revision keeps a Rollback button.

I moved the phone gateway that afternoon, at a time I chose. Only one gateway can hold the phone registration, so the move requires a short gap. We watched the old gateway's log until a [summary] line followed the last [call] answered line, which meant no call was in progress, and then switched. The old gateway stopped at 19:26:17 and the new one registered at 19:27:40, a 94-second gap during business hours. No call was dropped.

7 One box

Remaining services

By evening, production ran on the cluster. I decided the remaining smaller services had to move too, or the old platform could never be turned off.

The marketplace API had no source code in any repository. It ran as two Lambda functions, and the only copy was the deployment package AWS was running. I had it committed unchanged. It found four AWS SDK packages the code used that came from the Lambda runtime and were never declared, declared them, built a container image and pointed API Gateway's proxy integration at the cluster. Callers kept the same URLs, which was my requirement. We copied 5,196 Redis keys in 14 seconds and tested the switch end to end: a refresh request went to the new worker, and a WebSocket subscription received 13 progress messages ending in completed.

Testing also found a bug that Lambda had been hiding. When the API refuses a request with a 403, the route keeps running and tries to respond again in its error handler, which raises an unhandled promise rejection. On Lambda that ends one invocation. On a long-running Node 22 server it would end the process, so I added a handler that logs the rejection instead.

One operational detail changed with the new pipeline. The release workflow reports success as soon as it sets the new image in git. Argo checks git every three minutes and the canary runs after that, so a successful workflow run means the release was requested, not that it finished. The Rollout shows whether it finished.

8 One box

DNS

The last step was moving DNS so traffic no longer passed through the server. The registrar's API does not accept keys from accounts below a certain size. For most of the day the server stayed in place as a proxy, forwarding each hostname over a connection that only it could open, which let each hostname move separately.

The first hostname failed with an error I had not seen before. Amazon's certificate authority refused to issue a certificate, with CAA_ERROR. The domain's DNS provider publishes CAA records that list which certificate authorities may issue for the domain, and Amazon was not on the list. After one CAA record was added, the certificate was issued in about a minute. I entered the remaining records by hand, one hostname at a time.

The WAF ran in counting mode while the server still forwarded traffic, because every forwarded request arrives from the server's address. Once no hostname pointed at the server, I switched it to blocking. Before that, I checked how many requests the rules would have blocked: none of about 24,000 in six hours. In the 30 minutes after the last change, the load balancer served about 2,900 requests with no server errors and nothing blocked.

Argo CD sl-prod application, Healthy and Synced to k8s-release
Production on the evening of the cutover: Healthy and Synced to k8s-release.
⇄ Route map

How the pieces connect

147 CronJobs2DNS3Load balancer + WAF4API canary5Phone gateway6Postgres7Kafka8S3 Parquet9Calibration
1. 47 CronJobs. and workers
2. DNS. four hostnames
3. Load balancer + WAF. one certificate per domain
4. API canary. Argo Rollouts
5. Phone gateway. public voice node
6. Postgres. Neon, unchanged
7. Kafka. Debezium CDC
8. S3 Parquet. training data
9. Calibration. Workflows + MLflow
✓ Tickets

Why these choices

Each choice had to pass two tests: it removes a way production can fail, and one engineer can run it. The table lists the failures it addresses. The cards give the reasoning for each choice.

Failure on the old setupWhat prevents it now
A full disk stops everythingState on EFS, EBS and S3; no node holds data
A batch job uses up the API memorySeparate pods with their own memory limits
A bad release reaches usersThe canary is checked before traffic moves
Nobody knows what is runningGit records it; Argo reverts drift
One leaked key exposes two businessesOne role per workload and no keys
A cron job fails without anyone noticingInventory check and per-job metrics
Debugging requires SSHLogs, metrics and traces in Grafana
DECISION01

Kubernetes instead of a bigger server

CHOSE
EKS with three general nodes that scale to five
OVER
A larger server, Docker Compose on two servers, or ECS
WHY
A bigger server still fails as one machine. Compose cannot move work to another machine or run canary deploys. ECS would need custom work for scheduled jobs, canaries, GitOps and policy. Kubernetes has all four built in.
DECISION02

Git decides what runs

CHOSE
Argo CD applying a k8s-release branch, which is main plus one commit that sets image tags
OVER
CI pushing deploys into the cluster
WHY
The cluster pulls changes, so CI never holds production credentials. Argo reverts manual changes, and the running version is always a specific commit.
DECISION03

Canary instead of blue/green

CHOSE
One canary pod behind its own Service, checked by the old read checks running as Jobs, then promoted
OVER
Two full copies of the API and a proxy switch
WHY
The checks are the same and it uses half the resources. The release no longer runs on the server being released, and users only reach versions that passed.
DECISION04

IAM roles instead of keys

CHOSE
A separate AWS account and one IAM role per workload through Pod Identity
OVER
One shared access key in an environment file
WHY
A leaked credential exposes one workload instead of two businesses, and there are no keys to rotate.
DECISION05

Managed load balancer

CHOSE
An AWS load balancer with ACM certificates and a WAF in front of the cluster
OVER
An ingress proxy running inside the cluster
WHY
AWS handles TLS, certificate renewal and rate limits, and the load balancer keeps working if the cluster has problems.
DECISION06

State off the nodes

CHOSE
EFS for the old server files, encrypted gp3 volumes for Kafka and Redis, S3 for everything else
OVER
Disks attached to individual nodes
WHY
A pod can restart on any node and keep its data, and no single disk can fill up and stop production.
DECISION07

Stream changes instead of exporting them

CHOSE
Debezium reading the database change log into Kafka, written to S3 as Parquet
OVER
Nightly exports
WHY
Every change arrives in order within minutes, with no extra queries against production, so training data can be reproduced. The application code did not change.
DECISION08

Kafka on one node

CHOSE
Strimzi with three KRaft brokers on one dedicated node
OVER
A managed three-broker cluster at about six times the price
WHY
Change data capture can wait through a broker restart. The money goes to parts whose failures customers would notice.
DECISION09

Phone gateway on a public node

CHOSE
A small node in a public subnet, with the gateway on the host network and source ports preserved
OVER
Running the WebRTC gateway behind the NAT like other pods
WHY
A STUN test showed the NAT gives each destination a different port, which produces calls with no audio.
DECISION10

Shadow copy, inventory, DNS last

CHOSE
A read-only copy of production, a cutover that fails on anything not inventoried, and the old proxy forwarding until DNS moved
OVER
Testing an empty cluster and changing DNS on the day
WHY
Each one prevents a different failure: wrong numbers, a missed job and a slow rollback. Rollback stayed one command that takes seconds.
9 One box

What changed

After the move, a full disk cannot stop production, because no single disk holds it. The nightly pricing batch cannot use up the API's memory. A release cannot leave the system half deployed, because the canary is either promoted or deleted. A key in a leaked transcript no longer exposes two businesses, because production uses no keys. I can see which version is running, whether it is healthy and what it logged without opening a terminal.

Policy Reporter dashboard with image-signature results per namespace
Kyverno results in audit mode. Most failures are images built before signing was added.

These practices apply to any system that has to keep running while it changes, especially one that collects data from the field and needs a supply chain that can be audited:

  • Build a read-only copy of production first, and compare it with the real system before it serves anyone.
  • Keep the inventory in git and make the cutover fail on anything not listed.
  • Make every cutover step idempotent, and record how to undo it before it acts.
  • Let git be the only way changes reach the cluster.
  • Measure NAT behavior at the edge before relying on a network diagram.
  • Move DNS last, one hostname at a time, after the new path has carried real traffic.

The build took one night because I could split it across agents running in parallel. The inventory of what the server did took most of the next day. Every step was logged with its exact error, so this report could be written from those notes. I kept the old server running with no traffic for a few weeks as a rollback option, then shut it down.