Skip to main content

Deploy Postmortem — Vultr k3s Bring-up

The first real end-to-end deploy to Vultr k3s surfaced a chain of bugs. Most were latent — an earlier tofu apply → destroy was called "proven" but never actually verified k3s ran (it couldn't fetch the kubeconfig), so the cluster-level bugs sat undetected. This page records every bug, its root cause, the fix, and the lesson, so we don't burn hours re-diagnosing them.

Meta-lesson: a deploy is not "proven" until kubectl get nodes shows Ready, system pods are healthy, workloads are Running, and you have curled the app and gotten real content back. VM-booted ≠ cluster-working. See the verification checklist at the bottom.

Outcome of the session: 4/5 apps live and verified (storefront, blog CMS, API, docs) on a single k3s node with TLS wiring; Medusa root-caused + fixed (pending an image rebuild).


A. Cluster provisioning (OpenTofu / cloud-init)​

1. cloud-init double base64 → k3s never installed​

  • Symptom: cloud-init reports done ~28s after boot, but /usr/local/bin/k3s is absent, systemctl is-active k3s = inactive, and there's no k3s output in /var/log/cloud-init-output.log. /var/lib/cloud/instance/user-data.txt contains base64 text, not a #! script.
  • Root cause: k3s.tf used user_data = base64encode(templatefile(...)). The Vultr Terraform provider base64-encodes user_data itself, so it was encoded twice. cloud-init decoded once → got base64 text (no #! shebang) → didn't recognize it as a script → ran nothing.
  • Fix: user_data = templatefile(...) — no base64encode(). (infra/tofu/k3s.tf)
  • Lesson: the Vultr provider encodes user_data for you. Never pre-encode. Always confirm k3s installed, not just that the VM booted.

2. VPC CIDR ↔ k3s pod CIDR collision → all pod networking broken​

  • Symptom: coredns CrashLoopBackOff with dial tcp 10.43.0.1:443: connect: no route to host; worker nodes can't reach the control-plane over the VPC (100% packet loss to the control-plane's 10.42.x.x), so they never join.
  • Root cause: vpc_cidr was 10.42.0.0/16 — identical to k3s's default pod CIDR 10.42.0.0/16. Pods got 10.42.x.x addresses overlapping the VPC network, so a pod could not route to the node's VPC IP (where the API server endpoint lives) → "no route to host" for everything, including the kubernetes service (10.43.0.1).
  • Fix: vpc_cidr → 10.20.0.0/16. (infra/tofu/variables.tf)
  • Lesson: the VPC CIDR must not overlap k3s's pod CIDR (10.42.0.0/16) or service CIDR (10.43.0.0/16). This is the single most destructive and hardest-to-spot networking bug — coredns crashlooping right after install is the tell.

3. Kubernetes API (6443) unreachable from outside​

  • Symptom: kubectl times out to <node>:6443 (packets dropped, not refused), even though SSH (22) works from the same machine.
  • Root causes (two):
    1. The firewall allowed only 22/80/443 — not 6443. deploy.sh runs kubectl/helm externally, so it needs the API port open.
    2. Some office/ISP networks block outbound to 6443 entirely.
  • Fix:
    • Added a 6443 firewall rule scoped to admin_ssh_cidr (infra/tofu/network.tf).
    • For networks that block outbound 6443: SSH-tunnel it — ssh -N -L 6443:127.0.0.1:6443 root@<node> and point the kubeconfig at 127.0.0.1:6443 (k3s's API cert already includes 127.0.0.1, so TLS validates).
  • Lesson: the API port must be reachable from wherever kubectl/helm run. The SSH tunnel over port 22 is the reliable fallback.

B. Chart / deploy config​

4. awx.image helper ignored image.registry → wrong GHCR owner​

  • Symptom: pods ImagePullBackOff trying to pull ghcr.io/utlsolar/* (which doesn't exist) despite --set image.registry=ghcr.io/<real-owner>.
  • Root cause: the awx.image template used .image.repository verbatim (hardcoded ghcr.io/utlsolar/*), so the image.registry override was a no-op.
  • Fix: build the ref as <registry>/<base(repository)>:<tag> so --set image.registry re-points every app. (infra/helm/templates/_helpers.tpl)
  • Lesson: an override flag is only useful if it actually flows into the value.

5. GHCR images are private → pull 403 Forbidden​

  • Symptom: ImagePullBackOff, 403 Forbidden from ghcr.io/.../manifests/latest.
  • Root cause: GHCR packages are private by default, and the token used (the .env PAT) had no read:packages scope.
  • Fix: a read:packages token → ghcr-pull docker-registry secret; deploy.sh mints it on CI runners from the runner's GITHUB_TOKEN (packages: read added to deploy.yml) and wires it via imagePullSecrets.
  • Lesson: the cluster needs a pull secret with read:packages, or the packages must be made public.

6. Vultr CCM crash → LoadBalancer never provisioned​

  • Symptom: ingress-nginx LB EXTERNAL-IP <pending> forever; vultr-ccm crashloops: failed to listen on 0.0.0.0:10258: bind: address already in use.
  • Root cause: k3s runs its own cloud-controller-manager on port 10258; the external Vultr CCM conflicts.
  • Fix:
    • Single-node / test: put ingress-nginx on the node's hostPort 80/443 and point DNS at the node IP — no Vultr LB needed.
    • Real Vultr LB: add --disable-cloud-controller (and kubelet --cloud-provider=external) to the k3s server args so the Vultr CCM can bind 10258. (Follow-up — not yet in cloud-init.sh.tftpl.)
  • Lesson: to use an external CCM with k3s you must disable k3s's built-in one.

C. App runtime​

7. Mongo readiness probe false-negative​

  • Symptom: the mongo pod sits at 0/1 (NotReady) though it's running; Payload and the API can't reach it (a NotReady pod is dropped from its Service's endpoints).
  • Root cause: the readiness probe ran mongosh --eval ... with timeoutSeconds: 1, but mongosh takes ~2s just to start → the probe always timed out.
  • Fix: tcpSocket probe on 27017 (instant, reliable). (infra/helm/templates/databases.yaml)
  • Lesson: don't put a slow CLI (mongosh) in a 1s exec probe. For "is it listening", tcpSocket is faster and correct.

8. Docs (Docusaurus) OOMKilled​

  • Symptom: docs pod CrashLoopBackOff, reason OOMKilled.
  • Root cause: memory limit 128Mi is too low for the Docusaurus server.
  • Fix: limit → 512Mi. (infra/helm/values.yaml)
  • Lesson: even a "static" Node server needs >128Mi. Set realistic limits.

9. Medusa db:migrate deadlock (the hard one)​

  • Symptom: medusa db:migrate creates the mikro_orm_migrations table, then hangs forever — Medusa never starts (4/5 apps come up). The Postgres connection is idle (ClientRead), the Node JS stack is empty, and the debug log cycles knex:tx trxN: Starting top level transaction every 60s.
  • Root cause: Medusa/MikroORM's default knex/tarn pool deadlocks inside the container — the migration transaction can never acquire a connection (auto-commit SELECT/CREATE TABLE work fine), so tarn times out after 60s (acquireTimeoutMillis) and retries forever. Reproduces only in-container, not on a dev host.
  • Fix: set an explicit pool: databaseDriverOptions: { connection: { ssl: false }, pool: { min: 0, max: 10 } }. (apps/medusa/medusa-config.ts)
  • How it was diagnosed (reuse this method):
    1. Node diagnostic report — run with NODE_OPTIONS=--report-on-signal, then kill -SIGUSR2 <pid> → empty javascriptStack + libuv active handles (only a Postgres socket + a timer) ⇒ idle event loop awaiting an async op.
    2. Component debug logging — DEBUG=knex:*,mikro-orm:* → showed the Starting top level transaction retry-every-60s, pinpointing the pool.
  • Ruled out (don't re-chase these): advisory locks (0 held), telemetry (egress works, disabling didn't help), the Redis event bus (removed it → still hung), DNS/musl (used the Postgres ClusterIP directly → still hung), resources (node at 5% CPU).
  • Lesson: when a Node app "hangs" in a container, get a diagnostic report + component debug logs immediately instead of guessing. And always set an explicit DB pool for services running migrations.

D. Image-build bugs (earlier, before the cluster deploy)​

BugFix
apps/blog-cms was an accidental embedded git repo (gitlink) → its files (incl. Dockerfile) weren't in the build context → "Dockerfile not found"removed the nested .git, tracked the files
blog-cms npm ci failed — committed package-lock.json out of sync with package.json (a @payloadcms/richtext-lexical transitive tree that a dev box can't reproduce)Dockerfile deps stage uses npm install
blog-cms Dockerfile COPY .../public failed — no public/ dir; also missing output: 'standalone'added public/.gitkeep + output: 'standalone'
medusa build failed on TS errors in seed-solar.ts (query.graph returns nullable Maybe<StoreCurrency>; casts to non-overlapping types)drop annotations + optional-chain; filter nulls before .map; cast via unknown

Verification checklist​

Run these every deploy — each one catches a specific bug above before it costs an hour:

  1. k3s actually installed — ssh <node> systemctl is-active k3s returns active (catches bug #1). Not "the VM booted".
  2. API reachable — kubectl get nodes returns Ready (use the SSH tunnel if 6443 is blocked — bug #3).
  3. Pod networking healthy — kubectl -n kube-system get pods shows coredns 1/1 with no restarts (catches the CIDR collision, bug #2).
  4. Images pull — kubectl -n <tenant-ns> get pods shows all Running, no ImagePullBackOff (catches registry/owner + private-pull, bugs #4/#5).
  5. LoadBalancer/ingress reachable — the ingress has an address (or the node's 80/443 answer) (catches the CCM conflict, bug #6).
  6. Apps actually serve — curl -k --resolve <host>:443:<ip> https://<host>/ returns real content (200 + expected <title>/health JSON), not just a scheduled pod.
  7. Migrations completed — for Medusa, confirm tables exist: psql -c "select count(*) from information_schema.tables where table_schema='public'" climbs well past 1 (catches the migrate deadlock, #9).