Argo CD and Harbor each had a standalone Ingress serving the real domain
while their charts still served nip.io. Both charts now own their real
hostname directly, so the standalone objects are removed — extra-manifests
syncs the directory with prune, so deleting the files deletes the objects.
harbor-ingress-legacy-nipio.yaml is the inverse of the file it replaces.
Harbor's chart now serves harbor.infra.deployshed.com with a Let's Encrypt
certificate, and this object serves the old nip.io name with the private CA
the chart used to use. It has to exist: apps deployed before the move
recorded harbor.35.238.248.203.nip.io image references in toolshed's
database and in already-rendered pod specs, and nothing in git can rewrite
those — they change only when each app is rebuilt. Until then a pod that
restarts, reschedules or scales up pulls the old reference, and if the name
stops resolving that pull fails looking like a broken registry rather than a
retired hostname. It retires together with the registry-ca manifests and
Terraform's node-pool trust.
harbor-robot-dockerconfig lists both hostnames. Docker matches stored
credentials by exact host and nothing else, so a single-value swap breaks
whichever side lands second: change the credential first and the pipeline
pushes unauthenticated, change the pipeline first and the same happens in
reverse. Two entries make the order irrelevant — it is one robot account on
one Harbor reached by two names.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
A standalone Ingress because Harbor's chart cannot express a second host:
expose.ingress.hosts.core is a single scalar, with no extraHosts like
argo-cd and no secondaryingress like jenkins. The chart's own Ingress is
untouched and keeps serving the nip.io name with its private-CA
certificate; both names reach the same backends.
The path split is load-bearing and was copied from this cluster's own
generated Ingress rather than written from memory: /api/, /service/, /v2/
and /c/ go to harbor-core, everything else to harbor-portal. harbor-core
is the API, auth and registry backend; harbor-portal is only the web UI.
Sending /v2/ to the portal breaks every image pull, and it fails looking
like an authentication problem rather than a routing one.
What this deliberately does not do is make the new name primary. Harbor
still advertises the nip.io address: externalURL is a single value handed
to docker clients in Harbor's own API responses, so the UI may redirect
there and a docker login against this hostname is issued a token endpoint
pointing at the old one. Both resolve, so it works. Moving externalURL
means moving every image reference in the cluster at the same time —
running deployments, the dockerconfigjson auths key, toolshed's registry
settings and its stored connection — which is its own change, not a line
in this one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
The argo-cd chart cannot express this. Its server.ingress.tls is a
boolean covering the primary hostname plus every extraHost from one fixed
secret, with no extraTls to scope it. Enabling it while nip.io is still
served would request a single certificate spanning both names, which
Let's Encrypt refuses — leaving neither name with TLS rather than the one
it could have served.
So the chart keeps serving the nip.io hostname over plain HTTP through its
own Ingress, and this object serves the real domain over TLS. Same
backend, port 80 rather than 443 because this deployment runs
argocd-server with server.insecure: true — TLS terminates at Envoy, and
pointing at 443 would have Envoy attempt TLS to a backend that does not
speak it.
When nip.io is retired this file should be deleted and the chart's own
tls: true turned on instead: one hostname, one certificate, and no second
object to remember.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
The deployment has a domain of its own now, which is the first time a
public certificate has been possible here at all. The registry issuer
beside this one explains why: nip.io is not on the public suffix list and
every *.nip.io certificate shares one rate limit, so Let's Encrypt could
never serve the addresses this cluster has been using.
DNS-01 rather than HTTP-01, because every deployed app lives at
<app>.apps.<domain> and only a DNS-01 challenge can issue the wildcard
that covers all of them. HTTP-01 would mean a certificate per app,
requested the moment each one is created.
Two issuers, staging and production. Production allows five duplicate
certificates a week and a failing solver spends that allowance without
issuing anything, which can lock a domain out of certificates for days.
Staging has no meaningful limit and issues from an untrusted root, so a
browser warning is the signal that the plumbing works. Move to the
production issuer once a staging certificate appears.
The token reaches cert-manager the same way every other credential here
does: Vault, through External Secrets. It wants Zone -> DNS -> Edit on the
one zone and nothing else — enough to write the TXT record a challenge
needs, and no more. Until `vault kv put secret/cloudflare/dns-token` has
run, the ExternalSecret stays unfulfilled and the issuers cannot register.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
Node trust covers image pulls, which containerd performs on the node. A
docker push from a build pod is a different client with its own trust
store and knows nothing about that, so without this a push fails TLS
verification while a pull of the same image succeeds — which reads like a
broken registry rather than a missing trust anchor.
Committed in the clear rather than routed through Vault: this is the CA's
public half, the same certificate every node already fetches from Secret
Manager. The private key stays in Terraform state and reaches the cluster
only as the cert-manager Secret that signs.
Consumers mount it at
/etc/docker/certs.d/harbor.35.238.248.203.nip.io/ca.crt — the directory
name must be the registry hostname exactly, since dockerd looks the path
up by host and a mismatch silently does nothing. First consumer is the
base-images mirror pod; devops-lib-gcp's build pod follows.
Verified the embedded PEM round-trips: its SHA-256 fingerprint matches
terraform output registry_ca_cert_pem exactly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
The GCP repos now live under ~/Documents/localgit/gcp/ so they cannot be
confused with their homelab namesakes, which push to a different Gitea.
This file documents the one-off kubectl that creates the registry-ca
Secret from terraform output, and that path moved with them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
Argo CD v3's own CRDs (applications, applicationsets, appprojects) embed
large schemas that exceed the 256KiB last-applied-configuration annotation
a client-side apply writes — the same reason cert-manager and
external-secrets already use this. Without it the upgrade sync fails on
the CRDs rather than on anything in the release.
ServerSideDiff comes with it for the same reason as the other three, and
becomes belt-and-braces once the upgraded Argo CD's schema matches the
cluster.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
contour, cert-manager and external-secrets could not be compared or
synced:
error calculating structured merge diff: error building typed value
from live resource: .status.terminatingReplicas: field not declared
in schema
The ServerSideApply sync option makes Argo CD compute the diff locally
against a Kubernetes schema compiled into its own binary. Argo CD v2.13
is older than this cluster: GKE runs 1.35, and Deployments there carry
status.terminatingReplicas, which went beta and on-by-default in 1.33.
Argo CD's schema has never heard of the field, so the diff aborts before
any sync can happen. Nothing is wrong with the manifests, and only the
three apps using SSA are affected.
ServerSideDiff asks the API server to compute the diff via a dry-run
apply, so the schema in use is the cluster's own. Beta since v2.10 and
supported on the running version.
The generic chart could not express this — Application metadata had no
annotations block at all — so it gains an optional per-entry
compareOptions list rather than the annotation being hardcoded.
This is a workaround for an out-of-date Argo CD, not a fix. The fix is
upgrading to a build whose bundled schema matches the cluster; every
value key this repo relies on already exists in chart 10.8.4 (Argo CD
v3.5.2), so that upgrade is mostly a vendoring exercise plus the 3.0
breaking changes (logs RBAC now enforced, fine-grained RBAC inheritance,
resource tracking moving from labels to annotations).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
Adds nine Applications — external-secrets, harbor, jenkins, postgresql,
redis, victoria-metrics-single, vmagent, node-exporter, grafana — bringing
the cluster to the homelab's full set. Every nameOverride is pinned, which
is load-bearing rather than cosmetic in at least three cases:
external-secrets (its service account name is what vault-backend's
serviceAccountRef and Vault's auth role bind to), victoria-metrics-single
(its Service renders as <release>-server, the address vmagent and Grafana
both hardcode) and harbor (the pipeline refers to its object names).
extra-manifests gains the ClusterIssuer that makes cert-manager useful
here: a CA issuer over the registry-ca Secret, which is created by hand
from terraform output because the CA's private key exists only in
Terraform state and nothing can reconcile it from git.
Two homelab-only things removed rather than left to confuse: the Tailscale
Harbor Ingress, and the homelab cluster's own entry point and values file,
which have no meaning in a GCP-only repo. app-of-secretstores and
app-of-extra-manifests now point at this cluster's Gitea over cluster DNS,
and harbor-robot-dockerconfig is keyed to this cluster's registry hostname
— docker matches stored credentials by exact host, so the homelab's would
have silently pushed unauthenticated.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
Fresh install, unlike the homelab's adoption of a running Vault, so
there is no live state to diff against. nameOverride pins the release to
"vault" in namespace "vault" because Terraform's Workload Identity
binding names that exact pair; a different release name renames the
service account and leaves Vault sealed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
Contour is installed by ArgoCD rather than by hand, which is only
possible because nothing in the bootstrap path needed an ingress:
ArgoCD reaches Gitea over cluster DNS. Its HTTPProxy CRD embeds a large
schema, so the entry uses ServerSideApply for the same reason
cert-manager's does.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
Adds the GCP counterpart of the homelab's cluster entry point, alongside
it rather than replacing it, so the two clusters never share a values
file:
- values/incubator-infra-gke-toolshed-prd-usc1-values.yaml, starting with
only argocd, gitea and cert-manager. Every appSpec entry's nameOverride
equals the Helm release name each is installed under by hand, which is
what makes ArgoCD adopt those releases instead of rendering a second
copy of each.
- incubator/incubator-infra-gke-toolshed-prd-usc1.yaml, the parent
Application, applied once by hand.
- app-of-projects.yaml and the devops AppProject now point at this
cluster's Gitea over cluster DNS, not the homelab's LAN hostname.
Sources are reached at gitea-http.gitea.svc.cluster.local:3000 rather
than through an Ingress: ArgoCD has to read them before Contour exists,
since Contour is itself one of the Applications it installs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
toolshed can now run a one-off command against an app — a migration, a
seed, a backfill — in a throwaway Job built from the app's own image with
its own configuration (internal/deploy.Client.RunCommand). It exists
because a user with a provisioned database otherwise has no way to reach
it: the operator can exec into the Postgres pod, a user cannot, and the
premise is that nobody needs kubectl.
Without this the deployer gets "forbidden" the moment anyone runs one —
the two-repositories drift internal/deploy/kubernetes.go's own package doc
warns about, and the third feature in a row to need both copies changed
together.
Delete is included because the Job is removed once its output has been
captured into the run record; without it they would accumulate one per
command forever. Notably absent is pods/exec, which an exec-into-the-
running-pod design would have required — a far more dangerous grant on the
one credential here whose compromise means the cluster, and one reason the
Job approach was chosen over it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
toolshed deploys an app that asked for a persistent volume as a StatefulSet
with a volumeClaimTemplate rather than a Deployment
(internal/deploy.Client.ensureStatefulSet). Without this the deployer gets
"forbidden" the moment anyone creates one — the two-repositories drift
internal/deploy/kubernetes.go's own package doc warns about, and the same
way the custom-domains Ingress rights were missed until after that feature
shipped.
Delete is included deliberately, not for tidiness: a Deployment and a
StatefulSet share the app's selector, so switching an app between stateless
and stateful must remove whichever controller it no longer is, or both stay
alive fighting over the same pods.
persistentvolumeclaims is read-only. The claims are created by the
StatefulSet's own volumeClaimTemplates, never directly by toolshed, and
deleting one would destroy an app's data — so there is no reason for this
credential to be able to.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Registers the new hand-written redis chart (devops-infra-helm-charts,
separate commit) and the ExternalSecret feeding its admin password from
Vault. Own namespace, addressed over cluster DNS like every other platform
component here:
redis.redis.svc.cluster.local:6379
Only one consumer for the credential, unlike the Postgres one next door:
the server itself, to seed its ACL file on first boot. toolshed's api gets
it from the connection an operator configures in the dashboard, encrypted
in toolshed's own database — so there is deliberately no second
ExternalSecret into the toolshed namespace.
Order matters: put the password in Vault at secret/toolshed/redis before
syncing, or the init container sits in CreateContainerConfigError. The
exact command, the reason the password must be alphanumeric (it is written
into an ACL directive where a space or quote would split it), and the
manual rotation procedure are all recorded in the ExternalSecret's own
header.
Nothing here needs to change for Postgres: toolshed's managed database
add-on points at the existing postgresql.postgres.svc.cluster.local, whose
POSTGRES_USER is the initdb superuser and so already has the CREATEDB and
CREATEROLE that provisioning needs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Overdue: this belongs with toolshed's custom-domains feature, which has
already merged and shipped. Without it, deployer gets "forbidden" the
moment anyone actually adds a custom domain — the exact failure mode
internal/deploy/kubernetes.go's own package doc warns about, since its
resource list and this file are unsynchronised copies in two repositories.
Scoped as a namespaced Role/RoleBinding rather than widening the
ClusterRole above it. A custom domain's Ingress and TLS secret live in the
gateway's own namespace and never in an app's, so this only ever targets
one fixed namespace — unlike the per-app namespaces the ClusterRole
necessarily spans. Secrets are get/list/watch/delete only: cert-manager
writes them, deployer just reads readiness and cleans up on removal.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Companion change to toolshed's new multi-replica/autoscaling feature
(internal/deploy.Client.ensureAutoscaler in the toolshed repo) — without
this, deployer's own attempt to create a HorizontalPodAutoscaler for any
app with autoscaling enabled fails with "forbidden" the first time
someone actually uses the feature, exactly the failure mode
internal/deploy/kubernetes.go's own package doc comment warns about
for these two unsynchronized copies of deployer's permission list.
Kept in sync with toolshed's own deploy/helm/toolshed/templates/rbac.yaml,
which received the identical addition.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Deliberately separate from harbor-robot-dockerconfig, which is mounted
into build pods running arbitrary user Dockerfiles and is scoped to
push+pull only. This one is held by builder itself — a trusted platform
service that never executes user code directly — and carries a
permission the other should never have: delete. A leaked build-pod
credential can never delete anything this way, and a leaked cleanup
credential can never push.
Used by toolshed's cleanupImage (internal/builder/builder.go) to remove
an app's Harbor repository when the app itself is deleted. Nothing
destructive happens until the Vault path this pulls from is actually
populated — builder logs and skips that step otherwise, per its own
code comment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Matches the chart-side addition in devops-infra-helm-charts. The
ExternalSecret needs its Vault path populated before either this or
the grafana Application syncs — see that file's own header for the
exact vault kv put command, same requirement postgres and every other
admin credential in this repo already has.
app-of-secretstores.yaml is automated (prune: true), so this new
secretstore file needs no separate manual sync of its own — only that
app-of-apps wrapper, same as any other new file under secretstores/.
Verified with `helm template` against generic-argo-apps-chart and this
values file: 12 Applications render, grafana present and correctly
formed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Matches the chart-side replacement in devops-infra-helm-charts. Three
Applications now where there was one, since victoria-metrics-single
bundles no scraper or exporter the way the Prometheus chart did.
Sync order matters here specifically: victoria-metrics-single needs to
exist before vmagent's remote_write target resolves, though ArgoCD sync
failures are naturally retried so this only affects how quickly it
converges, not whether it does.
Verified with `helm template` against generic-argo-apps-chart and this
values file: 11 Applications render, the three new ones present and
correctly formed, nothing left pointing at the removed prometheus
chartDir.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Points at the chart and values just added to devops-infra-helm-charts.
nameOverride pinned to "prometheus" so the Helm release name matches
what the chart's server Service naming depends on
("<release-name>-server") — see that repo's commit for the full
reasoning, since this file is the one place that decision has to be
made correctly for toolshed's already-seeded PROMETHEUS_URL to resolve.
Needs the usual two syncs to actually deploy: this values file only
takes effect once incubator-infra-k8s-admin-prd-ase1 is synced (which
creates/updates the child "prometheus" Application object), and that
child Application then needs its own sync to actually create anything
in the cluster.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Environment variables are delivered through a Secret so values never appear
in the pod spec, but the ClusterRole was never given the resource — so every
deploy with configuration failed on a forbidden error, minutes after the
change looked fine.
Granted without list or watch. Kubernetes RBAC cannot scope a ClusterRole to
a namespace pattern, so this necessarily covers every namespace; withholding
list at least stops deployer enumerating the cluster's secrets, leaving only
access by a name it already knows. That narrows the blast radius rather than
removing it, and is called out in the manifest.
The proper fix, once there are tenants who are not the operator, is a
RoleBinding created per app namespace. That requires deployer to be able to
create RoleBindings, which is its own escalation path and wants deciding
deliberately rather than being slipped in here.
Scoped to the resource kinds toolshed creates for an app — not
cluster-admin, not a wildcard. This is the one credential in the system
whose compromise means the cluster, so what it can do should fit on one
screen.
Here rather than in toolshed's chart because these are cluster-scoped and
that Application runs in the webapp project, which permits only Namespace.
Widening webapp would have given every demo app in it the ability to create
cluster-wide RBAC.
A dedicated Jenkins user with an API token, not the admin account. Needs
Job/Create as well as Build and Read, because builder creates an app's job
on first deploy — which is what guarantees the SCM polling trigger exists
and so that a webhook can ever find it (issue #30).
Only api gets the private half. The gateway is given the public half in
plain Helm values, and that asymmetry is the point: the gateway
terminates traffic for every deployed app, so holding only a verification
key means compromising it cannot forge a session for anyone.
The two halves must be installed together — a private key here that does
not match the public key in the gateway's values means every session api
issues is rejected and every app redirects to login forever.
One Vault path (secret/toolshed/postgres) into two namespaces: the server
needs the credential to initialise, api needs it to connect. Kept in one
file because it is one credential with two consumers — split, they drift,
and drift surfaces as an authentication error that looks nothing like a
config mistake.
The api-side Secret also carries a composed libpq URL, so host, port and
database name live in one place instead of across several environment
variables that can disagree.
Postgres gets its own namespace rather than living inside toolshed, so it
is addressed over cluster DNS like any other platform component and
outlives whatever consumes it.
No longer needed now that anonymous pull works — pairs with
devops-helm-charts#efc2c4f dropping the corresponding pullSecret
reference from demo-go-app's values. app-of-secretstores has automated
prune, so this deletion applies on its own on next sync.
Both needed for the registry hostname switch to
harbor.192.168.1.7.nip.io (devops-lib#3ffa2d8):
- harbor-robot-dockerconfig's auths key was hardcoded to
harbor-core.harbor.svc.cluster.local — docker matches stored
credentials by exact registry hostname, so this had to move with
the image reference or push would go back to unauthorized.
- New demo-go-app-image-pull-secret, in the demo-go-app namespace
(imagePullSecrets must be same-namespace as the pod), for the
Deployment's own pull — separate from harbor-robot-dockerconfig,
which only ever lived in the jenkins namespace for the build pod's
push. Same underlying robot account/Vault path.
extra-manifests/priorityclass-high-priority.yaml: cluster-side
PriorityClass backing the app chart's hardcoded priorityClassName.
secretstores/argocd-jenkins-ci-token.yaml: ExternalSecret for the
scoped ArgoCD account devops-lib's syncArgoApp stage authenticates
with.
secretstores/harbor-robot-dockerconfig.yaml: ExternalSecret building a
dockerconfigjson from the Harbor robot account for image pulls.
