Harbor, Gitea, Argo CD, Jenkins, Vault, Grafana and vmui now answer on
their deployshed.com names alone. Each was already serving both while the
move was proved out; this removes the nip.io half.
The dual-hostname workarounds go with it. Jenkins' secondaryingress existed
only because its chart's primary ingress takes one hostName and a
certificate could not span both names — the real domain moves onto the
primary with jenkins-tls, which it already holds. Argo CD gets extraTls
rather than ingress.tls, because the boolean hardcodes secretName
argocd-server-tls and would request a second certificate for a name that
already has a valid one in argocd-deployshed-tls.
Harbor also changes in two ways beyond the hostname:
- externalURL moves to https://harbor.infra.deployshed.com. Harbor hands
this to docker clients in its own API responses and builds the push
commands shown in its UI from it, so a stale value is what makes a
correctly-configured registry still advertise the old address.
- updateStrategy is now Recreate. Its jobservice and registry volumes are
standard-rwo (ReadWriteOnce), and a RollingUpdate starts the new pod
before the old one releases the disk, so the replacement hangs forever
on Multi-Attach. The cluster was sitting in exactly that state, old pods
serving while new ones stayed in ContainerCreating. The chart's own
comment on this value recommends Recreate when RWM is unavailable. The
cost is a brief outage during upgrades, which beats a rollout that
cannot complete.
The private registry CA is not removed yet. Apps deployed before this move
recorded nip.io image references that only change when each is rebuilt, so
the old hostname stays served by a standalone Ingress until then.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
Disk, not memory. The volume sits at 1.4G of 2.9G at seven days, and every
extra day is charged against the project's shared 250GB SSD quota. Three
days still spans a weekend, which is the window that matters for working
out what happened to something overnight.
Worth being explicit about what this does not do, since it followed an
OOM-kill and could easily be mistaken for the fix: retention has no
bearing on the memory this needs. That tracks active time series and cache
size, neither of which depends on how long data is kept — which is why the
limit had to be raised rather than this lowered when it started
OOM-killing on restart. Lowering this now would not let that be undone.
The lever for memory, if it is ever needed, is scrape cardinality. Several
targets here emit more than forty labels per series, which VictoriaMetrics
warns about in its own log; dropping labels there reduces series count
directly, and with it the memory.
Old partitions are removed on VictoriaMetrics' own cycle rather than at
startup, so the disk figure falls over the following hours, not
immediately.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
It had run for weeks at 128Mi/512Mi and then OOM-killed in a loop the
first time anything restarted it — exit 137 about fifty seconds after a
clean start, repeatedly. Nothing about its configuration had changed. The
dataset had grown to 3.14 billion rows, and the memory needed to resume
ingestion no longer fit in the limit.
That failure mode is worth naming: a long-lived pod can sit comfortably
past the limit it would need in order to start again, so the problem stays
invisible until something restarts it — here, an unrelated sync adding a
hostname. The limit was not wrong when it was written; it was outgrown.
Memory tracks active time series rather than disk, so shortening
retentionPeriod would not have helped — the same targets are scraped
either way, and several carry more than forty labels.
Affordable: memory requests across the three nodes sit at 62%, 18% and
47%. CPU is what is scarce here, and this costs none.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
Each of these now answers on <name>.infra.deployshed.com alongside the
nip.io name it already had. Both are kept on purpose: nothing that
currently references the old name breaks, and the new one can be proved
before anything depends on it. Removing nip.io is a separate step, and a
larger one, because Harbor's name is embedded in every running app's image
reference.
TLS covers the real domain only. Let's Encrypt cannot issue for nip.io —
it is not on the public suffix list and every *.nip.io certificate shares
one rate limit — so a tls block naming both would request one certificate
spanning them and receive nothing for either. Each tls block therefore
lists exactly the one new hostname, which is why they are written out
rather than derived from the host list beside them.
The charts disagree about how to express a second host, so each is done
the way its own chart supports:
gitea, grafana, vault, victoria-metrics-single take host lists, so the
new name joins the existing one on a single Ingress.
jenkins' primary ingress accepts exactly one hostName, so the new name
goes on secondaryingress — a whole second Ingress object at the same
backend. paths must be set explicitly there; left at the chart's default
of [] it renders zero routes and the hostname answers nothing.
argo-cd takes extraHosts natively, but its ingress.tls is a boolean bound
to one fixed secret covering every host at once. Turning it on would
request a certificate including nip.io and fail, and there is no extraTls
to scope it. So ArgoCD gains the hostname now and its certificate when
nip.io goes.
Harbor is untouched here. It has no multi-host mechanism at all, so its
second hostname needs a standalone Ingress, and its externalURL is what
docker clients are handed — both deserve their own change rather than
riding along with a hostname tidy-up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
Gitea refused every push notification with "webhook can only call allowed
HTTP servers": toolshed builds an app's webhook target from the hostname
the dashboard was browsed on, so an app registered through the public URL
gets a public target, and this cluster's allowlist only permitted private
ranges.
Adding that one hostname rather than reverting to the homelab's "*". The
difference matters: "*" would also permit the node metadata server, which
is why it was narrowed in the first place. This permits exactly one host,
our own load balancer.
It is a stopgap and labelled as one in the file. The callback now hairpins
out to the load balancer and back in, which is precisely what the
cluster-DNS rule exists to avoid. The real fix is a configurable webhook
base URL in toolshed pointing at
toolshed-api.toolshed.svc.cluster.local:8080; this entry should be removed
when that lands.
Verified in the render: the value reaches Gitea's inline config, and
Recreate, standard-rwo, sqlite and disabled registration are untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
Removes the cause of the diff failures that ServerSideDiff worked around:
v2.13.8 diffs against a Kubernetes schema compiled into its own binary,
and this cluster is newer than that schema.
Three things needed real changes, none of them mechanical:
- applicationSet.enabled no longer exists, and there is no replacement
gate — unlike dex and notifications, the ApplicationSet controller's
Deployment carries no conditional at all. Helm ignores unknown keys, so
carrying the old value forward would have quietly started a controller
nothing here uses. replicas: 0 is the only lever.
- The image tag pin is dropped rather than moved to v3.5.2. The chart's
appVersion governs, so image and chart cannot drift; a pin outliving its
chart is close to the failure being fixed here.
- server.insecure moved from server.extraArgs to configs.params, which is
what the chart renders into argocd-cmd-params-cm. Caught by rendering:
an earlier version of this commit deleted the extra arg on the strength
of a comment claiming configs.params already set it, which it did not —
Argo CD would have served its own TLS behind Contour and produced a
redirect loop.
Verified in the rendered output: image v3.5.2 and no v2.13.8 anywhere,
server.insecure true, dex and notifications absent, ApplicationSet at zero
replicas, both HPAs intact with replicas omitted, ingress on Contour, repo
Secrets on cluster DNS.
Behaviour changes in v3 that apply here, none needing a values change:
logs RBAC is now enforced (jenkins-ci only syncs), update/delete no longer
inherit to sub-resources, and resource tracking moves from labels to
annotations, so the first sync re-stamps every managed resource.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
Ports the rest of the homelab's stack: external-secrets, harbor, jenkins,
postgresql, redis, victoria-metrics-single, vmagent, node-exporter and
grafana. All nine verified with helm template.
Most are the homelab's values with storage moved from local-path to
standard-rwo and hostnames rebuilt on the reserved ingress IP. vmagent and
node-exporter are unchanged outright — everything in them is addressed by
cluster DNS, namespace or container port, none of which differs here.
Harbor is the substantial one. The homelab serves it over plain HTTP and
makes containerd accept that by hand-editing hosts.toml on the node; GKE
nodes are managed and replaced, so that edit cannot survive. Instead the
node pool was told at creation to trust a private CA for exactly this
hostname, and cert-manager now signs Harbor's certificate from that same
CA via an ingress-shim annotation. certSource is "secret" rather than the
chart's "auto", which would self-sign a certificate nothing trusts.
externalURL moves to https to match, since Harbor hands that URL to docker
clients and a mismatch surfaces as registry errors.
Jenkins drops secondaryingress, which exists in the homelab only to serve
its Tailscale hostname. Its plugin pins are carried over deliberately: each
fixes a failure whose symptom points somewhere else, above all the
kubernetes/kubernetes-client-api pairing, without which agents never come
online and builds hang at "Still waiting to schedule task".
Postgres and Redis keep the homelab's deliberately small memory settings.
Those were chosen for an 8GB node under pressure, and while this cluster
has room, a bigger cache buys nothing for a handful of small tools.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
Same chart (0.34.1) and same shape as the homelab — production mode,
file storage, standalone, no HA, injector off — with one substantial
difference: this Vault unseals itself.
The homelab unseals with 3 of 5 Shamir keys after every restart, which
was fine on an always-on VM. These nodes are spot and can be reclaimed
at any hour, and a sealed Vault means every secret in the cluster is
unavailable until someone notices. The trade is named rather than
buried: unsealing now depends on GCP IAM rather than on people holding
key shares.
The seal stanza's values come from terraform output vault_seal, so
Terraform and this file cannot disagree about which key is used, and no
credential appears in either: the pod authenticates to KMS as its
Workload Identity. That binding names exactly vault/vault, so
serviceAccount.name is pinned and the GSA annotation set — miss either
and Vault starts, fails to reach KMS, and stays sealed with an error
that never mentions Workload Identity.
Verified against the live project: the rendered seal block matches the
existing key ring and key, which grant encrypt/decrypt to that service
account and nothing else.
Storage is 10Gi on standard-rwo. The homelab's 5Gi exists only because
local-path cannot expand a bound volume; this class can.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
The homelab binds Envoy to node ports 80/443 with hostPort, because
VMware bridging over Wi-Fi never made a LoadBalancer IP reachable
(claude.md issue #6). Here it is a real cloud load balancer, and since
the nodes have no public IPs it is the single inbound path to the
cluster.
envoy.service.loadBalancerIP pins it to the address Terraform reserved,
which is what makes DNS work: every hostname here is
<name>.35.238.248.203.nip.io, so an unpinned Service would take a fresh
ephemeral address and point every hostname at nothing.
That field is deprecated upstream (Kubernetes 1.24). GKE's replacement
annotation is not a drop-in — it takes the address resource's name, and
on an external Service also requires spec.loadBalancerClass:
networking.gke.io/l4-regional-external, changing which controller
programs the load balancer. GKE still honours the field, so this is the
smaller change.
contour.ingressClass.name is pinned to "contour" because every Ingress
in this cluster names that class; the chart default derives a name from
the release instead. Envoy stays a DaemonSet, which is what makes the
chart's default externalTrafficPolicy: Local (real client IPs) correct.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
A new cluster directory rather than edits to k8s-admin-prd-ase1, so no
homelab value is ever reused for GCP by accident. Charts are the ones
already vendored here (gitea 12.7.0, argo-cd 7.7.23, cert-manager
v1.20.1); only the values are new. Verified with helm template.
What differs from the homelab, and why:
- gitea: storageClass standard-rwo, and Recreate for a different reason
than the homelab's LevelDB lock — three nodes and a ReadWriteOnce disk
mean a rolling update's new pod waits forever on Multi-Attach. The
admin password comes from a Secret created at bootstrap instead of the
chart's published default, which would otherwise be live on a public
IP. Registration is disabled and webhooks are limited to private
ranges, for the same reason.
- argocd: single ingress host (no Tailscale), and the homelab's Ingress
health override is dropped, since Contour writes real load balancer
status here. server and repoServer autoscale 1-3 on CPU; the chart
omits replicas when autoscaling is on, so the HPA and ArgoCD's own
self-management do not fight over the count. Memory is deliberately
not a scaling metric: Go does not return memory promptly, so a memory
target scales up and never back down.
- cert-manager: written fresh, not copied. The homelab file was never
adapted from the fleet — it pulls from a private Meesho registry and
pins pods to a node pool that does not exist here. The chart's own
values.yaml carries that registry too, so imageRegistry and
imageNamespace are overridden back to upstream's quay.io/jetstack.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
