A standalone Ingress because Harbor's chart cannot express a second host:
expose.ingress.hosts.core is a single scalar, with no extraHosts like
argo-cd and no secondaryingress like jenkins. The chart's own Ingress is
untouched and keeps serving the nip.io name with its private-CA
certificate; both names reach the same backends.
The path split is load-bearing and was copied from this cluster's own
generated Ingress rather than written from memory: /api/, /service/, /v2/
and /c/ go to harbor-core, everything else to harbor-portal. harbor-core
is the API, auth and registry backend; harbor-portal is only the web UI.
Sending /v2/ to the portal breaks every image pull, and it fails looking
like an authentication problem rather than a routing one.
What this deliberately does not do is make the new name primary. Harbor
still advertises the nip.io address: externalURL is a single value handed
to docker clients in Harbor's own API responses, so the UI may redirect
there and a docker login against this hostname is issued a token endpoint
pointing at the old one. Both resolve, so it works. Moving externalURL
means moving every image reference in the cluster at the same time —
running deployments, the dockerconfigjson auths key, toolshed's registry
settings and its stored connection — which is its own change, not a line
in this one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
The argo-cd chart cannot express this. Its server.ingress.tls is a
boolean covering the primary hostname plus every extraHost from one fixed
secret, with no extraTls to scope it. Enabling it while nip.io is still
served would request a single certificate spanning both names, which
Let's Encrypt refuses — leaving neither name with TLS rather than the one
it could have served.
So the chart keeps serving the nip.io hostname over plain HTTP through its
own Ingress, and this object serves the real domain over TLS. Same
backend, port 80 rather than 443 because this deployment runs
argocd-server with server.insecure: true — TLS terminates at Envoy, and
pointing at 443 would have Envoy attempt TLS to a backend that does not
speak it.
When nip.io is retired this file should be deleted and the chart's own
tls: true turned on instead: one hostname, one certificate, and no second
object to remember.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
The deployment has a domain of its own now, which is the first time a
public certificate has been possible here at all. The registry issuer
beside this one explains why: nip.io is not on the public suffix list and
every *.nip.io certificate shares one rate limit, so Let's Encrypt could
never serve the addresses this cluster has been using.
DNS-01 rather than HTTP-01, because every deployed app lives at
<app>.apps.<domain> and only a DNS-01 challenge can issue the wildcard
that covers all of them. HTTP-01 would mean a certificate per app,
requested the moment each one is created.
Two issuers, staging and production. Production allows five duplicate
certificates a week and a failing solver spends that allowance without
issuing anything, which can lock a domain out of certificates for days.
Staging has no meaningful limit and issues from an untrusted root, so a
browser warning is the signal that the plumbing works. Move to the
production issuer once a staging certificate appears.
The token reaches cert-manager the same way every other credential here
does: Vault, through External Secrets. It wants Zone -> DNS -> Edit on the
one zone and nothing else — enough to write the TXT record a challenge
needs, and no more. Until `vault kv put secret/cloudflare/dns-token` has
run, the ExternalSecret stays unfulfilled and the issuers cannot register.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
Node trust covers image pulls, which containerd performs on the node. A
docker push from a build pod is a different client with its own trust
store and knows nothing about that, so without this a push fails TLS
verification while a pull of the same image succeeds — which reads like a
broken registry rather than a missing trust anchor.
Committed in the clear rather than routed through Vault: this is the CA's
public half, the same certificate every node already fetches from Secret
Manager. The private key stays in Terraform state and reaches the cluster
only as the cert-manager Secret that signs.
Consumers mount it at
/etc/docker/certs.d/harbor.35.238.248.203.nip.io/ca.crt — the directory
name must be the registry hostname exactly, since dockerd looks the path
up by host and a mismatch silently does nothing. First consumer is the
base-images mirror pod; devops-lib-gcp's build pod follows.
Verified the embedded PEM round-trips: its SHA-256 fingerprint matches
terraform output registry_ca_cert_pem exactly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
The GCP repos now live under ~/Documents/localgit/gcp/ so they cannot be
confused with their homelab namesakes, which push to a different Gitea.
This file documents the one-off kubectl that creates the registry-ca
Secret from terraform output, and that path moved with them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
Adds nine Applications — external-secrets, harbor, jenkins, postgresql,
redis, victoria-metrics-single, vmagent, node-exporter, grafana — bringing
the cluster to the homelab's full set. Every nameOverride is pinned, which
is load-bearing rather than cosmetic in at least three cases:
external-secrets (its service account name is what vault-backend's
serviceAccountRef and Vault's auth role bind to), victoria-metrics-single
(its Service renders as <release>-server, the address vmagent and Grafana
both hardcode) and harbor (the pipeline refers to its object names).
extra-manifests gains the ClusterIssuer that makes cert-manager useful
here: a CA issuer over the registry-ca Secret, which is created by hand
from terraform output because the CA's private key exists only in
Terraform state and nothing can reconcile it from git.
Two homelab-only things removed rather than left to confuse: the Tailscale
Harbor Ingress, and the homelab cluster's own entry point and values file,
which have no meaning in a GCP-only repo. app-of-secretstores and
app-of-extra-manifests now point at this cluster's Gitea over cluster DNS,
and harbor-robot-dockerconfig is keyed to this cluster's registry hostname
— docker matches stored credentials by exact host, so the homelab's would
have silently pushed unauthenticated.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
toolshed can now run a one-off command against an app — a migration, a
seed, a backfill — in a throwaway Job built from the app's own image with
its own configuration (internal/deploy.Client.RunCommand). It exists
because a user with a provisioned database otherwise has no way to reach
it: the operator can exec into the Postgres pod, a user cannot, and the
premise is that nobody needs kubectl.
Without this the deployer gets "forbidden" the moment anyone runs one —
the two-repositories drift internal/deploy/kubernetes.go's own package doc
warns about, and the third feature in a row to need both copies changed
together.
Delete is included because the Job is removed once its output has been
captured into the run record; without it they would accumulate one per
command forever. Notably absent is pods/exec, which an exec-into-the-
running-pod design would have required — a far more dangerous grant on the
one credential here whose compromise means the cluster, and one reason the
Job approach was chosen over it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
toolshed deploys an app that asked for a persistent volume as a StatefulSet
with a volumeClaimTemplate rather than a Deployment
(internal/deploy.Client.ensureStatefulSet). Without this the deployer gets
"forbidden" the moment anyone creates one — the two-repositories drift
internal/deploy/kubernetes.go's own package doc warns about, and the same
way the custom-domains Ingress rights were missed until after that feature
shipped.
Delete is included deliberately, not for tidiness: a Deployment and a
StatefulSet share the app's selector, so switching an app between stateless
and stateful must remove whichever controller it no longer is, or both stay
alive fighting over the same pods.
persistentvolumeclaims is read-only. The claims are created by the
StatefulSet's own volumeClaimTemplates, never directly by toolshed, and
deleting one would destroy an app's data — so there is no reason for this
credential to be able to.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Overdue: this belongs with toolshed's custom-domains feature, which has
already merged and shipped. Without it, deployer gets "forbidden" the
moment anyone actually adds a custom domain — the exact failure mode
internal/deploy/kubernetes.go's own package doc warns about, since its
resource list and this file are unsynchronised copies in two repositories.
Scoped as a namespaced Role/RoleBinding rather than widening the
ClusterRole above it. A custom domain's Ingress and TLS secret live in the
gateway's own namespace and never in an app's, so this only ever targets
one fixed namespace — unlike the per-app namespaces the ClusterRole
necessarily spans. Secrets are get/list/watch/delete only: cert-manager
writes them, deployer just reads readiness and cleans up on removal.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Companion change to toolshed's new multi-replica/autoscaling feature
(internal/deploy.Client.ensureAutoscaler in the toolshed repo) — without
this, deployer's own attempt to create a HorizontalPodAutoscaler for any
app with autoscaling enabled fails with "forbidden" the first time
someone actually uses the feature, exactly the failure mode
internal/deploy/kubernetes.go's own package doc comment warns about
for these two unsynchronized copies of deployer's permission list.
Kept in sync with toolshed's own deploy/helm/toolshed/templates/rbac.yaml,
which received the identical addition.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Environment variables are delivered through a Secret so values never appear
in the pod spec, but the ClusterRole was never given the resource — so every
deploy with configuration failed on a forbidden error, minutes after the
change looked fine.
Granted without list or watch. Kubernetes RBAC cannot scope a ClusterRole to
a namespace pattern, so this necessarily covers every namespace; withholding
list at least stops deployer enumerating the cluster's secrets, leaving only
access by a name it already knows. That narrows the blast radius rather than
removing it, and is called out in the manifest.
The proper fix, once there are tenants who are not the operator, is a
RoleBinding created per app namespace. That requires deployer to be able to
create RoleBindings, which is its own escalation path and wants deciding
deliberately rather than being slipped in here.
Scoped to the resource kinds toolshed creates for an app — not
cluster-admin, not a wildcard. This is the one credential in the system
whose compromise means the cluster, so what it can do should fit on one
screen.
Here rather than in toolshed's chart because these are cluster-scoped and
that Application runs in the webapp project, which permits only Namespace.
Widening webapp would have given every demo app in it the ability to create
cluster-wide RBAC.
extra-manifests/priorityclass-high-priority.yaml: cluster-side
PriorityClass backing the app chart's hardcoded priorityClassName.
secretstores/argocd-jenkins-ci-token.yaml: ExternalSecret for the
scoped ArgoCD account devops-lib's syncArgoApp stage authenticates
with.
secretstores/harbor-robot-dockerconfig.yaml: ExternalSecret building a
dockerconfigjson from the Harbor robot account for image pulls.
