Deploy Postmortem — Vultr k3s Bring-up
The first real end-to-end deploy to Vultr k3s surfaced a chain of bugs. Most
were latent — an earlier tofu apply → destroy was called "proven" but never
actually verified k3s ran (it couldn't fetch the kubeconfig), so the cluster-level
bugs sat undetected. This page records every bug, its root cause, the fix, and the
lesson, so we don't burn hours re-diagnosing them.
Meta-lesson: a deploy is not "proven" until
kubectl get nodesshowsReady, system pods are healthy, workloads areRunning, and you have curled the app and gotten real content back. VM-booted ≠ cluster-working. See the verification checklist at the bottom.
Outcome of the session: 4/5 apps live and verified (storefront, blog CMS, API, docs) on a single k3s node with TLS wiring; Medusa root-caused + fixed (pending an image rebuild).
A. Cluster provisioning (OpenTofu / cloud-init)
1. cloud-init double base64 → k3s never installed
- Symptom: cloud-init reports
done~28s after boot, but/usr/local/bin/k3sis absent,systemctl is-active k3s=inactive, and there's no k3s output in/var/log/cloud-init-output.log./var/lib/cloud/instance/user-data.txtcontains base64 text, not a#!script. - Root cause:
k3s.tfuseduser_data = base64encode(templatefile(...)). The Vultr Terraform provider base64-encodesuser_dataitself, so it was encoded twice. cloud-init decoded once → got base64 text (no#!shebang) → didn't recognize it as a script → ran nothing. - Fix:
user_data = templatefile(...)— nobase64encode(). (infra/tofu/k3s.tf) - Lesson: the Vultr provider encodes
user_datafor you. Never pre-encode. Always confirm k3s installed, not just that the VM booted.
2. VPC CIDR ↔ k3s pod CIDR collision → all pod networking broken
- Symptom: coredns
CrashLoopBackOffwithdial tcp 10.43.0.1:443: connect: no route to host; worker nodes can't reach the control-plane over the VPC (100% packet loss to the control-plane's10.42.x.x), so they never join. - Root cause:
vpc_cidrwas10.42.0.0/16— identical to k3s's default pod CIDR10.42.0.0/16. Pods got10.42.x.xaddresses overlapping the VPC network, so a pod could not route to the node's VPC IP (where the API server endpoint lives) → "no route to host" for everything, including thekubernetesservice (10.43.0.1). - Fix:
vpc_cidr→10.20.0.0/16. (infra/tofu/variables.tf) - Lesson: the VPC CIDR must not overlap k3s's pod CIDR (
10.42.0.0/16) or service CIDR (10.43.0.0/16). This is the single most destructive and hardest-to-spot networking bug — coredns crashlooping right after install is the tell.
3. Kubernetes API (6443) unreachable from outside
- Symptom:
kubectltimes out to<node>:6443(packets dropped, not refused), even though SSH (22) works from the same machine. - Root causes (two):
- The firewall allowed only 22/80/443 — not 6443.
deploy.shrunskubectl/helmexternally, so it needs the API port open. - Some office/ISP networks block outbound to 6443 entirely.
- The firewall allowed only 22/80/443 — not 6443.
- Fix:
- Added a 6443 firewall rule scoped to
admin_ssh_cidr(infra/tofu/network.tf). - For networks that block outbound 6443: SSH-tunnel it —
ssh -N -L 6443:127.0.0.1:6443 root@<node>and point the kubeconfig at127.0.0.1:6443(k3s's API cert already includes127.0.0.1, so TLS validates).
- Added a 6443 firewall rule scoped to
- Lesson: the API port must be reachable from wherever
kubectl/helmrun. The SSH tunnel over port 22 is the reliable fallback.
B. Chart / deploy config
4. awx.image helper ignored image.registry → wrong GHCR owner
- Symptom: pods
ImagePullBackOfftrying to pullghcr.io/utlsolar/*(which doesn't exist) despite--set image.registry=ghcr.io/<real-owner>. - Root cause: the
awx.imagetemplate used.image.repositoryverbatim (hardcodedghcr.io/utlsolar/*), so theimage.registryoverride was a no-op. - Fix: build the ref as
<registry>/<base(repository)>:<tag>so--set image.registryre-points every app. (infra/helm/templates/_helpers.tpl) - Lesson: an override flag is only useful if it actually flows into the value.
5. GHCR images are private → pull 403 Forbidden
- Symptom:
ImagePullBackOff,403 Forbiddenfromghcr.io/.../manifests/latest. - Root cause: GHCR packages are private by default, and the token used
(the
.envPAT) had noread:packagesscope. - Fix: a
read:packagestoken →ghcr-pulldocker-registry secret;deploy.shmints it on CI runners from the runner'sGITHUB_TOKEN(packages: readadded todeploy.yml) and wires it viaimagePullSecrets. - Lesson: the cluster needs a pull secret with
read:packages, or the packages must be made public.
6. Vultr CCM crash → LoadBalancer never provisioned
- Symptom: ingress-nginx LB
EXTERNAL-IP <pending>forever;vultr-ccmcrashloops:failed to listen on 0.0.0.0:10258: bind: address already in use. - Root cause: k3s runs its own cloud-controller-manager on port 10258; the external Vultr CCM conflicts.
- Fix:
- Single-node / test: put ingress-nginx on the node's hostPort 80/443 and point DNS at the node IP — no Vultr LB needed.
- Real Vultr LB: add
--disable-cloud-controller(and kubelet--cloud-provider=external) to the k3s server args so the Vultr CCM can bind 10258. (Follow-up — not yet incloud-init.sh.tftpl.)
- Lesson: to use an external CCM with k3s you must disable k3s's built-in one.
C. App runtime
7. Mongo readiness probe false-negative
- Symptom: the mongo pod sits at
0/1(NotReady) though it's running; Payload and the API can't reach it (a NotReady pod is dropped from its Service's endpoints). - Root cause: the readiness probe ran
mongosh --eval ...withtimeoutSeconds: 1, butmongoshtakes ~2s just to start → the probe always timed out. - Fix:
tcpSocketprobe on 27017 (instant, reliable). (infra/helm/templates/databases.yaml) - Lesson: don't put a slow CLI (
mongosh) in a 1s exec probe. For "is it listening",tcpSocketis faster and correct.
8. Docs (Docusaurus) OOMKilled
- Symptom:
docspodCrashLoopBackOff, reasonOOMKilled. - Root cause: memory limit
128Miis too low for the Docusaurus server. - Fix: limit →
512Mi. (infra/helm/values.yaml) - Lesson: even a "static" Node server needs >128Mi. Set realistic limits.
9. Medusa db:migrate deadlock (the hard one)
- Symptom:
medusa db:migratecreates themikro_orm_migrationstable, then hangs forever — Medusa never starts (4/5 apps come up). The Postgres connection isidle(ClientRead), the Node JS stack is empty, and the debug log cyclesknex:tx trxN: Starting top level transactionevery 60s. - Root cause: Medusa/MikroORM's default knex/tarn pool deadlocks inside the
container — the migration transaction can never acquire a connection
(auto-commit
SELECT/CREATE TABLEwork fine), so tarn times out after 60s (acquireTimeoutMillis) and retries forever. Reproduces only in-container, not on a dev host. - Fix: set an explicit pool:
databaseDriverOptions: { connection: { ssl: false }, pool: { min: 0, max: 10 } }. (apps/medusa/medusa-config.ts) - How it was diagnosed (reuse this method):
- Node diagnostic report — run with
NODE_OPTIONS=--report-on-signal, thenkill -SIGUSR2 <pid>→ emptyjavascriptStack+libuvactive handles (only a Postgres socket + a timer) ⇒ idle event loop awaiting an async op. - Component debug logging —
DEBUG=knex:*,mikro-orm:*→ showed theStarting top level transactionretry-every-60s, pinpointing the pool.
- Node diagnostic report — run with
- Ruled out (don't re-chase these): advisory locks (0 held), telemetry (egress works, disabling didn't help), the Redis event bus (removed it → still hung), DNS/musl (used the Postgres ClusterIP directly → still hung), resources (node at 5% CPU).
- Lesson: when a Node app "hangs" in a container, get a diagnostic report + component debug logs immediately instead of guessing. And always set an explicit DB pool for services running migrations.
D. Image-build bugs (earlier, before the cluster deploy)
| Bug | Fix |
|---|---|
apps/blog-cms was an accidental embedded git repo (gitlink) → its files (incl. Dockerfile) weren't in the build context → "Dockerfile not found" | removed the nested .git, tracked the files |
blog-cms npm ci failed — committed package-lock.json out of sync with package.json (a @payloadcms/richtext-lexical transitive tree that a dev box can't reproduce) | Dockerfile deps stage uses npm install |
blog-cms Dockerfile COPY .../public failed — no public/ dir; also missing output: 'standalone' | added public/.gitkeep + output: 'standalone' |
medusa build failed on TS errors in seed-solar.ts (query.graph returns nullable Maybe<StoreCurrency>; casts to non-overlapping types) | drop annotations + optional-chain; filter nulls before .map; cast via unknown |
Verification checklist
Run these every deploy — each one catches a specific bug above before it costs an hour:
- k3s actually installed —
ssh <node> systemctl is-active k3sreturnsactive(catches bug #1). Not "the VM booted". - API reachable —
kubectl get nodesreturnsReady(use the SSH tunnel if 6443 is blocked — bug #3). - Pod networking healthy —
kubectl -n kube-system get podsshows coredns1/1with no restarts (catches the CIDR collision, bug #2). - Images pull —
kubectl -n <tenant-ns> get podsshows allRunning, noImagePullBackOff(catches registry/owner + private-pull, bugs #4/#5). - LoadBalancer/ingress reachable — the ingress has an address (or the node's 80/443 answer) (catches the CCM conflict, bug #6).
- Apps actually serve —
curl -k --resolve <host>:443:<ip> https://<host>/returns real content (200 + expected<title>/health JSON), not just a scheduled pod. - Migrations completed — for Medusa, confirm tables exist:
psql -c "select count(*) from information_schema.tables where table_schema='public'"climbs well past 1 (catches the migrate deadlock, #9).