Fold the deployshed ingresses into their charts, keep nip.io for Harbor

Argo CD and Harbor each had a standalone Ingress serving the real domain
while their charts still served nip.io. Both charts now own their real
hostname directly, so the standalone objects are removed — extra-manifests
syncs the directory with prune, so deleting the files deletes the objects.

harbor-ingress-legacy-nipio.yaml is the inverse of the file it replaces.
Harbor's chart now serves harbor.infra.deployshed.com with a Let's Encrypt
certificate, and this object serves the old nip.io name with the private CA
the chart used to use. It has to exist: apps deployed before the move
recorded harbor.35.238.248.203.nip.io image references in toolshed's
database and in already-rendered pod specs, and nothing in git can rewrite
those — they change only when each app is rebuilt. Until then a pod that
restarts, reschedules or scales up pulls the old reference, and if the name
stops resolving that pull fails looking like a broken registry rather than a
retired hostname. It retires together with the registry-ca manifests and
Terraform's node-pool trust.

harbor-robot-dockerconfig lists both hostnames. Docker matches stored
credentials by exact host and nothing else, so a single-value swap breaks
whichever side lands second: change the credential first and the pipeline
pushes unauthenticated, change the pipeline first and the same happens in
reverse. Two entries make the order irrelevant — it is one robot account on
one Harbor reached by two names.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N

app-of-extra-manifests.yaml

@@ -16,7 +16,7 @@ metadata:
 spec:
   project: default
   source:
-    repoURL: 'http://gitea.192.168.1.7.nip.io/mukul/devops-infra-argo-config.git'
+    repoURL: 'http://gitea-http.gitea.svc.cluster.local:3000/gitadmin/devops-infra-argo-config-gcp.git'
     targetRevision: main
     path: extra-manifests
     directory:

app-of-secretstores.yaml

@@ -18,7 +18,7 @@ metadata:
 spec:
   project: default
   source:
-    repoURL: 'http://gitea.192.168.1.7.nip.io/mukul/devops-infra-argo-config.git'
+    repoURL: 'http://gitea-http.gitea.svc.cluster.local:3000/gitadmin/devops-infra-argo-config-gcp.git'
     targetRevision: main
     path: secretstores
     directory:

extra-manifests/harbor-ingress-legacy-nipio.yaml

@@ -0,0 +1,92 @@
+# Harbor's OLD hostname, kept alive on the private CA.
+#
+# This is the inverse of what this file used to be. Harbor's own chart now
+# serves harbor.infra.deployshed.com with a Let's Encrypt certificate, and
+# this standalone object serves the legacy nip.io name that the chart used
+# to own.
+#
+# WHY IT STILL EXISTS, and when it can go. Apps deployed before the move
+# recorded their image reference — harbor.35.238.248.203.nip.io/... — in
+# toolshed's database (apps.current_image) and in already-rendered pod
+# specs. Nothing in git can rewrite those; they change only when each app is
+# rebuilt and redeployed. Until then a running pod that restarts, reschedules
+# or scales up pulls the old reference, and if this name stops resolving that
+# pull fails. The failure would look like a broken registry rather than a
+# retired hostname.
+#
+# Delete this file, registry-ca-clusterissuer.yaml and registry-ca-configmap.yaml
+# together, once no image reference anywhere names the old host. Check with:
+#
+#   kubectl get pods -A -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}' | grep nip.io
+#   (and toolshed's own: select current_image from apps)
+#
+# The node pool's private_registry_access_config in Terraform retires at the
+# same time, but that one recycles nodes, so it belongs in a maintenance
+# window rather than in a cleanup commit.
+#
+# THE PATH SPLIT IS LOAD-BEARING. harbor-core is the API, auth and registry
+# backend; harbor-portal is only the web UI. /api/, /service/, /v2/ and /c/
+# go to core, everything else to the portal. Sending /v2/ to the portal
+# instead breaks every image pull, and it fails looking like an
+# authentication problem rather than a routing one.
+#
+# Contour matches the longest prefix, so the trailing / rule cannot shadow
+# the four above it regardless of order — they are written first anyway, so
+# the intent is obvious to whoever reads this next.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: harbor-ingress-legacy-nipio
+  namespace: harbor
+  annotations:
+    # The private CA, not Let's Encrypt, which cannot issue for a nip.io
+    # name: it is not on the public suffix list, and every *.nip.io
+    # certificate shares one rate limit. The GKE node pool was told to trust
+    # this CA for exactly this hostname when it was created, which is what
+    # makes the old image references still pullable.
+    cert-manager.io/cluster-issuer: registry-ca-issuer
+spec:
+  ingressClassName: contour
+  tls:
+    - hosts:
+        - harbor.35.238.248.203.nip.io
+      secretName: harbor-tls
+  rules:
+    - host: harbor.35.238.248.203.nip.io
+      http:
+        paths:
+          - path: /api/
+            pathType: Prefix
+            backend:
+              service:
+                name: harbor-core
+                port:
+                  number: 80
+          - path: /service/
+            pathType: Prefix
+            backend:
+              service:
+                name: harbor-core
+                port:
+                  number: 80
+          - path: /v2/
+            pathType: Prefix
+            backend:
+              service:
+                name: harbor-core
+                port:
+                  number: 80
+          - path: /c/
+            pathType: Prefix
+            backend:
+              service:
+                name: harbor-core
+                port:
+                  number: 80
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: harbor-portal
+                port:
+                  number: 80

extra-manifests/harbor-ingress-tailscale.yaml

@@ -1,56 +0,0 @@
-# Harbor's chart has no multi-host mechanism at all (checked against the
-# actual template — no extraHosts like argo-cd, no secondaryingress like
-# jenkins), so this is a standalone second Ingress for the Tailscale
-# hostname. Correction from an earlier version of this file: it routed
-# everything to harbor-core alone. That's wrong — harbor-core is the
-# API/auth backend; the actual web UI (the portal you log into) is served
-# by harbor-portal. Mirrors the chart's own generated Ingress path split
-# exactly (confirmed against its real rendered spec, not assumed this
-# time): /api/, /service/, /v2/, /c/ go to harbor-core, everything else
-# goes to harbor-portal.
-apiVersion: networking.k8s.io/v1
-kind: Ingress
-metadata:
-  name: harbor-ingress-tailscale
-  namespace: harbor
-spec:
-  ingressClassName: contour
-  rules:
-    - host: "harbor.100.90.248.118.nip.io"
-      http:
-        paths:
-          - path: /api/
-            pathType: Prefix
-            backend:
-              service:
-                name: harbor-core
-                port:
-                  number: 80
-          - path: /service/
-            pathType: Prefix
-            backend:
-              service:
-                name: harbor-core
-                port:
-                  number: 80
-          - path: /v2/
-            pathType: Prefix
-            backend:
-              service:
-                name: harbor-core
-                port:
-                  number: 80
-          - path: /c/
-            pathType: Prefix
-            backend:
-              service:
-                name: harbor-core
-                port:
-                  number: 80
-          - path: /
-            pathType: Prefix
-            backend:
-              service:
-                name: harbor-portal
-                port:
-                  number: 80

extra-manifests/letsencrypt-clusterissuer.yaml

@@ -0,0 +1,72 @@
+# The issuer for this deployment's public certificates, from Let's Encrypt.
+#
+# Two of them: staging first, production second. Let's Encrypt's production
+# endpoint allows five duplicate certificates per week and a failed solver
+# burns that allowance without issuing anything, so a misconfiguration can
+# lock the real domain out of certificates for days. The staging endpoint
+# has no meaningful limit and issues from an untrusted root — a browser will
+# warn, which is exactly what proves the plumbing works before anything
+# depends on it.
+#
+# DNS-01, not HTTP-01, because every deployed app lives at
+# <app>.apps.<domain> and only a DNS-01 challenge can issue the wildcard
+# that covers all of them. HTTP-01 would need a certificate per app,
+# requested the moment each one is created.
+#
+# This is what a real domain buys. The registry issuer beside this file
+# explains why nip.io could never have it: nip.io is not on the public
+# suffix list, and every *.nip.io certificate shares one rate limit.
+#
+# BEFORE THIS WORKS, three things must be true:
+#
+#  1. The zone's records point straight at the load balancer, NOT through
+#     Cloudflare's proxy. A proxied record answers from Cloudflare's own
+#     addresses, so the certificate would protect traffic that never
+#     reaches this cluster.
+#  2. The `cloudflare-api-token` Secret exists in the cert-manager
+#     namespace, created by Terraform from var.cloudflare_api_token. Its
+#     key must be `api-token`; cert-manager reports a mismatch only when a
+#     challenge fails, long after everything else looked fine.
+#  3. The email below is filled in. Let's Encrypt requires one for expiry
+#     notices, and leaving the placeholder makes registration fail.
+---
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-staging
+spec:
+  acme:
+    server: https://acme-staging-v02.api.letsencrypt.org/directory
+    # REPLACE ME — Let's Encrypt registers this address and sends expiry
+    # warnings to it. It is given to a third party, so it is deliberately
+    # not filled in from anyone's account details.
+    email: mukul.sharma909.ms@gmail.com
+    privateKeySecretRef:
+      # cert-manager's own ACME account key, which it creates. Nothing
+      # supplies this; it must differ between the two issuers or they share
+      # an account registration across two different endpoints.
+      name: letsencrypt-staging-account-key
+    solvers:
+      - dns01:
+          cloudflare:
+            apiTokenSecretRef:
+              name: cloudflare-api-token
+              key: api-token
+---
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-prod
+spec:
+  acme:
+    server: https://acme-v02.api.letsencrypt.org/directory
+    # REPLACE ME — see above.
+    email: mukul.sharma909.ms@gmail.com
+    privateKeySecretRef:
+      name: letsencrypt-prod-account-key
+    solvers:
+      - dns01:
+          cloudflare:
+            apiTokenSecretRef:
+              name: cloudflare-api-token
+              key: api-token

extra-manifests/registry-ca-clusterissuer.yaml

@@ -0,0 +1,31 @@
+# The issuer that signs Harbor's certificate, from the private CA the GKE
+# node pool was told to trust.
+#
+# Terraform generates that CA and puts its PUBLIC key in Secret Manager,
+# which is what every node fetches to verify image pulls. cert-manager needs
+# both halves in order to sign, so the key pair is created as the
+# `registry-ca` Secret in the cert-manager namespace by hand, from
+# `terraform output` — the private key exists nowhere else, so nothing can
+# reconcile it from git:
+#
+#   TF=~/Documents/localgit/gcp/toolshed-gke-infra/envs/prod/10-infra
+#   kubectl -n cert-manager create secret tls registry-ca \
+#     --cert=<(terraform -chdir=$TF output -raw registry_ca_cert_pem) \
+#     --key=<(terraform -chdir=$TF output -raw registry_ca_key_pem)
+#
+# A ClusterIssuer, not an Issuer, so Harbor's namespace can request a
+# certificate without the CA's private key being copied there. Note that a
+# ClusterIssuer always reads its secret from cert-manager's own namespace,
+# which is what makes that possible.
+#
+# This is the whole reason cert-manager is installed at all: nothing else in
+# this cluster uses TLS. Let's Encrypt cannot help here, because nip.io is
+# not on the public suffix list and every *.nip.io certificate shares one
+# rate limit.
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: registry-ca-issuer
+spec:
+  ca:
+    secretName: registry-ca

extra-manifests/registry-ca-configmap.yaml

@@ -0,0 +1,67 @@
+# The registry CA's PUBLIC certificate, for Docker clients that push to
+# Harbor from inside the cluster.
+#
+# Not a secret, which is why it is committed here in plain sight rather than
+# going through Vault: it is the same certificate every node already fetches
+# from Secret Manager, and publishing a CA's public half is what a CA is for.
+# The private key is a different matter entirely — it exists only in
+# Terraform state and reaches the cluster as the `registry-ca` Secret in the
+# cert-manager namespace, which cert-manager uses to sign.
+#
+# Why this exists at all: the node pool trusting the CA covers image PULLS,
+# performed by containerd on the node. A `docker push` from a build pod is a
+# separate client with its own trust store, and it has no idea the node
+# trusts anything. Without this, pushes fail TLS verification while pulls of
+# the very same image work — which reads like a broken registry rather than a
+# missing trust anchor.
+#
+# Mount it in whichever container runs dockerd, at
+#   /etc/docker/certs.d/harbor.35.238.248.203.nip.io/ca.crt
+# The directory name must be the registry hostname exactly; dockerd looks the
+# path up by host, so a mismatch silently does nothing. Consumers so far: the
+# base-images mirror pod (devops-base-images-gcp) and devops-lib-gcp's build
+# pod template.
+#
+# The homelab needs none of this — its Harbor is plain HTTP and its dind
+# passes --insecure-registry instead.
+#
+# If the CA is ever regenerated in Terraform, this file must be updated to
+# match, along with the cert-manager Secret and the node pool's Secret
+# Manager version.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: registry-ca
+  namespace: jenkins
+data:
+  ca.crt: |
+    -----BEGIN CERTIFICATE-----
+    MIIFMTCCAxmgAwIBAgIRAPf5gf3756sdf3nlZBTnlREwDQYJKoZIhvcNAQELBQAw
+    MjERMA8GA1UEChMIdG9vbHNoZWQxHTAbBgNVBAMTFHRvb2xzaGVkLXJlZ2lzdHJ5
+    LWNhMB4XDTI2MDkxMTA0MDIxNloXDTM2MDkwODA0MDIxNlowMjERMA8GA1UEChMI
+    dG9vbHNoZWQxHTAbBgNVBAMTFHRvb2xzaGVkLXJlZ2lzdHJ5LWNhMIICIjANBgkq
+    hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA5GS3vpS2iG+Gl9YQx/NwlIXfMLhKVfDM
+    YixjPzYQv93Md9iR8c7CDFPijw8kf7200L7Z6Cp4hrCseL7QZZyvGnrmAt8lfhmz
+    Fj1Q9RP/WbKam2GHXanz3sWQ7XtUQiEg0yHkBchglgHQYQhdbePMi3zXPx8KPhis
+    QcNtOAAsLdisIaCz+NrA8qgPMel9LcdZTejaxhBRkOGPHO47xDslNl2JBnermbLc
+    XqWMh1exyWcscylaE8/e8brufRGWAXMfA+JElDH92VLIGaTQwI2KcTFr9SM7UwYN
+    tdcggT3BzPOmV/DaICJu+81RplZiEj+rU6ZQ1rwYUMPcEelWaDjL5e+27z5bFTJk
+    JCGc2x4HjfwD65961d0Mn/BqG8VzwS4lck9QvHcMNlw8o7AKr/q0ZWCcslMFZq7E
+    fmv6lZLl2Jw0go6dsh2kCRRGjoG4CoQ5YelRYgjzmNPDp4cN8rwKy/Oy80OGjvXt
+    +AgZvpjuBYGEEvSIM3QttU+6OOuHKxseAY5sbWzuhru0xASq3P4e2pQk8EacnTJc
+    OrMMDXd39vTazhEb1INJzOeD0fiBNTscQD/vbO0vffrNQiTF6NaKZErRhadZWDCq
+    3w5SslAEuHMF3op8KhrCggWG3HlQiHyt0GTKagz8DAKum8/ZMyuoIwp1UIrgp/q7
+    fZeqdSDqS5UCAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgGGMA8GA1UdEwEB/wQFMAMB
+    Af8wHQYDVR0OBBYEFEjYpg0hI2j/UC88pR/aG1oeY4VnMA0GCSqGSIb3DQEBCwUA
+    A4ICAQDNIXgdYnKyDjI95dYSiqsdc4Je7hdFA62/9HXYkjtlsycmm3GM33DcX94N
+    9siAtnir6Br3yzGKdwQ8ZB5iZw6aIZTfgwgcnJA/Zrhsx0nfIyIlgZQHOalHXvXt
+    keE2XoMzF8vzN3YsMpEHT7gRrRQ7Rvn+wufgql5oiO0iaf6XYLypoyXKrBsg+gLh
+    8Ek29TuWry9Ip04wtfNCff0+o/vrSpsDVIA34ayEpDqKQqjEgxA8EF3MfXaA3aYb
+    JWcz+BR2eX3xACkkjCNMuVJFPdsdJyDPmWe8hESpdDY0ATBgyOiZtlcpK+3Uo/WN
+    AZX856Axi4vJjkNe8ve8+B3oCCbBVZHaPjoi9jEWt/FmO1mtM1On2SWKgHGM4olz
+    9nQUL6G+4qypWOCzgu1xtCGIOP184oWR6TMplEHafBUWfamRmNsMxzzWCujGCtFh
+    O43WajhM2fo2T3egtpCX3orzuOVIisT2trP8pgClB5IKqvoCKc7cyKEb8Va1MVur
+    pqu+4wNmvZ+M7xRAJhd7FljkeOUTGb3ziuVd8YHsx/GTmJV9QXN1CZsVaRJCZdTt
+    DcHPaQDiwfSdr+p6ILpyGPH0l+2A+NXSjDI/tCK9CIzeleCPy4m1UhffSqKOaEKs
+    Gn1E5oROiqzjN5SWKHJRShbxKMq+v3VBN0OZJxQAMshK9aC+eg==
+    -----END CERTIFICATE-----

generic-argo-apps-chart/templates/genericTemplate.yaml

@@ -24,11 +24,26 @@
 
 {{ $argoAppNamespace := $top.Values.argocdSpec.namespace }}
 
+{{- $compareOptions := $config.compareOptions | default list -}}
+
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
   name: '{{- if $appNameOverride }}{{ $appNameOverride }}{{- else }}{{ printf "%s-%s-%s" $appName $cluster $env }}{{- end }}'
   namespace: {{ $argoAppNamespace }}
+{{- if $compareOptions }}
+  annotations:
+    # Opt-in per appSpec entry, as a list of Argo CD compare options.
+    #
+    # The one that matters here is ServerSideDiff=true, which is the escape
+    # hatch for "field not declared in schema" diff failures. Argo CD
+    # normally computes the diff locally against a Kubernetes schema baked
+    # into its own binary, so a cluster newer than Argo CD has fields Argo
+    # CD has never heard of and the diff aborts. This option asks the API
+    # server to compute the diff instead (a dry-run apply), and the API
+    # server necessarily knows its own fields.
+    argocd.argoproj.io/compare-options: {{ join "," $compareOptions | quote }}
+{{- end }}
   labels:
 {{ toYaml $labels | indent 4 }}
   finalizers:

incubator/incubator-infra-k8s-admin-prd-ase1.yaml

@@ -1,19 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: incubator-infra-k8s-admin-prd-ase1
-  namespace: argocd
-  finalizers:
-    - resources-finalizer.argocd.argoproj.io
-spec:
-  project: default
-  source:
-    repoURL: 'http://gitea.192.168.1.7.nip.io/mukul/devops-infra-argo-config.git'
-    targetRevision: main
-    path: generic-argo-apps-chart
-    helm:
-      valueFiles:
-      - ../values/incubator-infra-k8s-admin-prd-ase1-values.yaml
-  destination:
-    name: in-cluster
-    namespace: argocd

secretstores/cloudflare-dns-token.yaml

@@ -0,0 +1,43 @@
+# The Cloudflare API token cert-manager answers DNS-01 challenges with.
+#
+# DNS-01 rather than HTTP-01 because every deployed app lives at
+# <app>.apps.<domain>, and only a DNS-01 challenge can issue the wildcard
+# that covers all of them at once. HTTP-01 would mean a certificate per app,
+# requested the moment each one is created.
+#
+# The token wants Zone -> DNS -> Edit on the one zone and nothing else. It
+# can create and delete TXT records in that zone, which is all the challenge
+# needs; anything wider is a credential in a cluster that did not have to be.
+#
+# Put the value in Vault first — this only copies it, and an ExternalSecret
+# pointing at a path that does not exist stays unfulfilled with the Secret
+# never created:
+#
+#   vault kv put secret/cloudflare/dns-token token='<the token>'
+#
+# The key below MUST stay "api-token": letsencrypt-clusterissuer.yaml in
+# extra-manifests/ names it in apiTokenSecretRef, and cert-manager reports a
+# mismatch only when a challenge fails — long after everything else looked
+# like it had applied cleanly.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: cloudflare-dns-token
+  # cert-manager's own namespace, because a ClusterIssuer always reads its
+  # secrets from there regardless of which namespace asked for the
+  # certificate. That is what lets one issuer serve every namespace without
+  # the token being copied into any of them.
+  namespace: cert-manager
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: vault-backend
+    kind: ClusterSecretStore
+  target:
+    name: cloudflare-api-token
+    creationPolicy: Owner
+  data:
+    - secretKey: api-token
+      remoteRef:
+        key: cloudflare/dns-token
+        property: token

secretstores/harbor-robot-dockerconfig.yaml

@@ -28,13 +28,27 @@ spec:
       type: kubernetes.io/dockerconfigjson
       data:
         # Keyed by exact registry hostname — docker matches credentials
-        # against the host portion of the image reference. Was
+        # against the host portion of the image reference, so this must be
-        # harbor-core.harbor.svc.cluster.local; switched to the Contour
+        # the same hostname the pipeline pushes to and Harbor's own
-        # ingress hostname alongside buildDocker.groovy and dind-pod.yaml
+        # externalURL uses. On GKE that is the Contour ingress host, never
-        # so push/pull share one consistent, resolvable-from-anywhere
+        # harbor-core.harbor.svc.cluster.local: cluster DNS resolves from a
-        # reference.
+        # pod but not from the node's containerd doing the actual image pull.
+        #
+        # BOTH hostnames are listed, deliberately, while the deployment moves
+        # off nip.io. Because the match is by exact host and nothing else, a
+        # single-value swap breaks whichever side lands second: change this
+        # first and the pipeline (still pushing to nip.io) pushes
+        # unauthenticated, which Harbor rejects as "unauthorized"; change the
+        # pipeline first and the same happens in reverse. Two entries make
+        # the order irrelevant — the credential is identical either way,
+        # since it is one robot account on one Harbor reached by two names.
+        #
+        # The nip.io entry comes out once no image reference anywhere names
+        # it: the pipeline's push target, the fallback Dockerfiles, the
+        # mirrored base images, and every app already deployed from an image
+        # tagged under the old name.
         .dockerconfigjson: |
-          {"auths":{"harbor.192.168.1.7.nip.io":{"username":"{{ .username }}","password":"{{ .password }}","auth":"{{ printf "%s:%s" .username .password | b64enc }}"}}}
+          {"auths":{"harbor.infra.deployshed.com":{"username":"{{ .username }}","password":"{{ .password }}","auth":"{{ printf "%s:%s" .username .password | b64enc }}"},"harbor.35.238.248.203.nip.io":{"username":"{{ .username }}","password":"{{ .password }}","auth":"{{ printf "%s:%s" .username .password | b64enc }}"}}}
   data:
     - secretKey: username
       remoteRef:

values/incubator-infra-gke-toolshed-prd-usc1-values.yaml

@@ -34,11 +34,57 @@ appSpec:
     namespace: argocd
     chartDir: argo-cd
     valuesDir: argocd-admin-prd
+    # Argo CD v3's own CRDs (applications, applicationsets, appprojects)
+    # embed large schemas, past the 256KiB last-applied-configuration limit
+    # a client-side apply writes — the same reason cert-manager and
+    # external-secrets use this.
+    serverSideApply: true
+    # And therefore the same stale-schema diff problem: see the contour
+    # entry. Once the upgrade lands, Argo CD's schema matches this cluster
+    # and this becomes belt-and-braces rather than load-bearing.
+    compareOptions:
+      - ServerSideDiff=true
   - name: gitea
     nameOverride: gitea
     namespace: gitea
     chartDir: gitea
     valuesDir: gitea
+  - name: contour
+    # The cluster's only inbound path, pinned to the reserved IP that every
+    # deployshed.com record resolves to. Installed by ArgoCD rather than by
+    # hand, which is only possible because nothing in the bootstrap needed
+    # an ingress: ArgoCD reaches Gitea over cluster DNS.
+    nameOverride: contour
+    namespace: projectcontour
+    chartDir: contour
+    valuesDir: contour
+    # Contour's HTTPProxy CRD embeds a large schema, same 256KiB
+    # last-applied-configuration problem as cert-manager's.
+    serverSideApply: true
+    # Required alongside serverSideApply on this cluster. That sync option
+    # makes Argo CD compute the diff with the Kubernetes schema compiled
+    # into its own binary, and Argo CD v2.13 predates this cluster: live
+    # Deployments here carry status.terminatingReplicas, beta and enabled by
+    # default since Kubernetes 1.33, which that schema has never heard of.
+    # The diff then aborts with "field not declared in schema" and the app
+    # can neither be compared nor synced. ServerSideDiff hands the diff to
+    # the API server, which necessarily knows its own fields.
+    #
+    # This is a workaround for an out-of-date Argo CD, not a fix. The fix is
+    # upgrading Argo CD to a build whose schema matches the cluster.
+    compareOptions:
+      - ServerSideDiff=true
+  - name: vault
+    # Fresh install here, unlike the homelab's adoption of a running Vault.
+    # nameOverride pinned so the release is "vault" in namespace "vault":
+    # the Workload Identity binding Terraform created names that exact
+    # namespace/serviceaccount pair, and a different release name renames
+    # the service account, which leaves Vault unable to reach KMS and
+    # therefore sealed.
+    nameOverride: vault
+    namespace: vault
+    chartDir: vault
+    valuesDir: vault
   - name: cert-manager
     nameOverride: cert-manager
     namespace: cert-manager
@@ -47,3 +93,81 @@ appSpec:
     # cert-manager's CRDs embed large OpenAPI schemas; SSA avoids the
     # 256KiB last-applied-configuration limit, same as external-secrets.
     serverSideApply: true
+    # See the contour entry above: server-side apply diffs against Argo CD's
+    # own bundled schema, which is older than this cluster.
+    compareOptions:
+      - ServerSideDiff=true
+  - name: external-secrets
+    # Every credential in the cluster arrives through this. nameOverride is
+    # not cosmetic here: without it the release becomes
+    # "external-secrets-<cluster>-<env>", which renames the controller's
+    # service account — and both secretstores/vault-backend.yaml's
+    # serviceAccountRef and Vault's own Kubernetes auth role are bound to
+    # the plain name.
+    nameOverride: external-secrets
+    namespace: external-secrets
+    chartDir: external-secrets
+    valuesDir: external-secrets
+    # ClusterSecretStore's CRD embeds a large provider schema and exceeds
+    # the 256KiB annotation limit on a client-side apply.
+    serverSideApply: true
+    # See the contour entry above: server-side apply diffs against Argo CD's
+    # own bundled schema, which is older than this cluster.
+    compareOptions:
+      - ServerSideDiff=true
+  - name: harbor
+    # Object names are all prefixed with the release name, and the pipeline
+    # refers to them, so it is pinned.
+    nameOverride: harbor
+    namespace: harbor
+    chartDir: harbor
+    valuesDir: harbor
+  - name: jenkins
+    nameOverride: jenkins
+    namespace: jenkins
+    chartDir: jenkins
+    valuesDir: jenkins
+  - name: postgresql
+    # Backs toolshed's control plane. Own namespace so it is addressed over
+    # cluster DNS like any other platform component and outlives whatever
+    # consumes it: postgresql.postgres.svc.cluster.local:5432
+    nameOverride: postgresql
+    namespace: postgres
+    chartDir: postgresql
+    valuesDir: postgresql
+  - name: redis
+    # Backs toolshed's managed cache add-on, same namespace reasoning as
+    # postgresql: redis.redis.svc.cluster.local:6379
+    nameOverride: redis
+    namespace: redis
+    chartDir: redis
+    valuesDir: redis
+  - name: victoria-metrics-single
+    # Pinned name, because the chart renders its Service as
+    # <release>-server and both vmagent's remote write and Grafana's
+    # datasource address it at
+    # victoria-metrics-single-server.monitoring.svc.cluster.local:8428
+    nameOverride: victoria-metrics-single
+    namespace: monitoring
+    chartDir: victoria-metrics-single
+    valuesDir: victoria-metrics-single
+  - name: vmagent
+    # The scraper. Needs the Service above to exist, so bring it up after.
+    nameOverride: vmagent
+    namespace: monitoring
+    chartDir: vmagent
+    valuesDir: vmagent
+  - name: node-exporter
+    nameOverride: node-exporter
+    namespace: monitoring
+    chartDir: node-exporter
+    valuesDir: node-exporter
+  - name: grafana
+    # Requires secretstores/grafana-admin-credentials.yaml to have synced
+    # first: without that Secret the pod generates a random admin password
+    # nobody has. It does not crash, it just cannot be logged into until
+    # the Secret exists and the pod restarts.
+    nameOverride: grafana
+    namespace: monitoring
+    chartDir: grafana
+    valuesDir: grafana

values/incubator-infra-k8s-admin-prd-ase1-values.yaml

@@ -1,217 +0,0 @@
-clusterSpec:
-  # "k8s-admin-prd-ase1" only resolved in the fleet because that name was
-  # registered as an external cluster in the hub ArgoCD's cluster list.
-  # There's no hub here — one ArgoCD, running on the cluster it manages —
-  # so this has to be the built-in local-cluster alias instead.
-  destination:
-    server: ""
-    name: "in-cluster"
-
-argocdSpec:
-  # Was argocd-admin (a separate hub namespace in the fleet's two-tier
-  # setup). Single ArgoCD instance here, so Application objects live in
-  # the same namespace as ArgoCD itself — see claude.md.
-  namespace: argocd
-
-teamSpec:
-  devops:
-    source:
-      repoURL: http://gitea.192.168.1.7.nip.io/mukul/devops-infra-helm-charts.git
-      targetRevision: main
-      path: helm-templates
-      valueFiles: ../../helm-overrides/k8s-admin-prd-ase1
-    labels:
-      bu: infra
-      team: devops
-      env: prd
-      cluster: k8s-admin-prd-ase1
-
-appSpec:
-  - name: argocd
-    nameOverride: argocd-admin-prd
-    namespace: argocd
-    chartDir: argo-cd
-    valuesDir: argocd-admin-prd
-  - name: gitea
-    # Adopting the already-running standalone install (helm release
-    # "gitea" in namespace "gitea", from deploy_gitea.sh) rather than
-    # deploying a second one — nameOverride pins the rendered
-    # Application's name (and therefore the Helm release name Argo
-    # renders with) to match those existing object names exactly.
-    nameOverride: gitea
-    namespace: gitea
-    chartDir: gitea
-    valuesDir: gitea
-    # NOT using the Application-wide `replace: true` here anymore — it
-    # forces a full PUT of every resource this Application renders, and a
-    # bound PVC's spec is immutable (volumeName/storageClassName get
-    # filled in by the provisioner after binding; a PUT that omits them
-    # looks like clearing them, which the API correctly refuses). The
-    # Deployment-only fix now lives as a per-resource sync-option
-    # annotation in helm-overrides/k8s-admin-prd-ase1/gitea/custom-values.yaml
-    # (deployment.annotations), which only Replaces the Deployment.
-  - name: vault
-    # Adopting the running production-mode Vault (helm release "vault" in
-    # namespace "vault", chart 0.34.1 — see helm-templates/vault/Chart.yaml).
-    # It's already initialized and unsealed; this Application only manages
-    # Vault's Deployment/config, never its data or seal state. Review the
-    # first diff carefully before syncing — this is the highest-consequence
-    # adoption in this repo so far.
-    nameOverride: vault
-    namespace: vault
-    chartDir: vault
-    valuesDir: vault
-  - name: contour
-    # Adopting the running ingress (helm release "contour" in namespace
-    # "projectcontour", chart 0.7.0 — the OFFICIAL projectcontour chart,
-    # not the Bitnami one that used to be wired up as helm-templates/contour
-    # — see the note in that Chart.yaml and claude.md issue #4). This is
-    # the ingress path for every other Application in this repo — review
-    # the diff before syncing, same caution as vault.
-    nameOverride: contour
-    namespace: projectcontour
-    chartDir: contour
-    valuesDir: contour
-  - name: external-secrets
-    # Correction from an earlier version of this file: "no nameOverride
-    # needed" was wrong. Without one, the Application (and therefore the
-    # Helm release name the chart templates with) becomes
-    # "external-secrets-admin-prd" — so the controller's ServiceAccount
-    # actually ends up named external-secrets-admin-prd, not
-    # external-secrets. secretstores/vault-backend.yaml's
-    # serviceAccountRef assumes the plain name, and Vault's role was bound
-    # to bound_service_account_names=external-secrets — both need this
-    # pinned name to match.
-    nameOverride: external-secrets
-    namespace: external-secrets
-    chartDir: external-secrets
-    valuesDir: external-secrets
-    # ClusterSecretStore's CRD (large embedded OpenAPI schema) exceeds the
-    # 256KiB last-applied-configuration annotation limit on a normal
-    # client-side apply. SSA sidesteps it entirely — see the note in
-    # generic-argo-apps-chart's template.
-    serverSideApply: true
-  - name: jenkins
-    # Fresh install, but pinning nameOverride anyway — learned from
-    # external-secrets that skipping it produces
-    # "jenkins-admin-prd"-suffixed resource names, which
-    # jenkins-admin-credentials (the ExternalSecret, namespace "jenkins")
-    # doesn't need to care about, but keeps naming predictable and
-    # consistent with every other app here regardless.
-    nameOverride: jenkins
-    namespace: jenkins
-    chartDir: jenkins
-    valuesDir: jenkins
-  - name: harbor
-    # Fresh install (helm list -n harbor came back empty despite claude.md
-    # saying otherwise). nameOverride pinned for the same predictability
-    # reason as jenkins — rendered object names all end up prefixed with
-    # this (harbor-core, harbor-registry, etc.), which is also what
-    # Jenkins needs to reference for internal image pushes
-    # (harbor-core.harbor.svc.cluster.local).
-    nameOverride: harbor
-    namespace: harbor
-    chartDir: harbor
-    valuesDir: harbor
-  - name: postgresql
-    # Backs toolshed's control plane. Own namespace rather than living
-    # inside toolshed, so it is addressed over cluster DNS like any other
-    # platform component and outlives whatever consumes it:
-    #   postgresql.postgres.svc.cluster.local:5432
-    #
-    # nameOverride pinned for the same reason as everything else here —
-    # without it the rendered Application (and therefore the Helm release
-    # name, and therefore every object name) becomes
-    # "postgresql-admin-prd-prd".
-    #
-    # Hand-written chart, not Bitnami's: that registry has been actively
-    # unstable (infra issue #4) and PostgreSQL ships no official chart.
-    # Requires secretstores/toolshed-postgres-credentials.yaml to have
-    # synced first — the pod cannot start without the Secret.
-    nameOverride: postgresql
-    namespace: postgres
-    chartDir: postgresql
-    valuesDir: postgresql
-  - name: redis
-    # Backs toolshed's managed cache add-on — toolshed provisions a per-app
-    # ACL user, scoped to its own key prefix, on request. Own namespace for
-    # the same reason postgresql has one: addressed over cluster DNS like
-    # any other platform component, outliving whatever consumes it:
-    #   redis.redis.svc.cluster.local:6379
-    #
-    # Hand-written chart, not Bitnami's, for the same reason as postgresql
-    # (infra issue #4) — Redis ships no official chart either.
-    #
-    # Authentication is defined by an ACL file with no requirepass, which
-    # is a security property rather than a preference: see the chart's own
-    # values.yaml, where getting it wrong leaves the server open to
-    # unauthenticated access after its first restart.
-    #
-    # Requires secretstores/toolshed-redis-credentials.yaml to have synced
-    # first — the init container cannot seed the ACL file without it.
-    nameOverride: redis
-    namespace: redis
-    chartDir: redis
-    valuesDir: redis
-  - name: victoria-metrics-single
-    # Replaces the Prometheus server this entry briefly was (see git
-    # history on this file) — same job, lower RAM/disk footprint for the
-    # same metric volume, and it speaks Prometheus's own query API
-    # (/api/v1/query) so nothing downstream (toolshed's metrics
-    # connection, docs/PRODUCT-ARCHITECTURE.md step 5) needed to change,
-    # only the URL it points at.
-    #
-    # Vendored official chart (victoriametrics/helm-charts), same
-    # vendor-the-official-chart pattern as Contour/ArgoCD/Vault/Gitea/
-    # Harbor/Jenkins. This exact directory name already existed in this
-    # repo before — a leftover GKE-targeted vendored copy from the
-    # original Meesho monorepo import — and was removed rather than
-    # adapted; see that chart's own Chart.yaml comment.
-    #
-    # nameOverride pinned to exactly "victoria-metrics-single" for the
-    # same reason as postgresql/gitea/prometheus above: the chart's
-    # server Service renders as "<release-name>-server", so this is what
-    # makes it resolvable at a predictable hostname
-    # (victoria-metrics-single-server.monitoring.svc.cluster.local:8428)
-    # rather than "victoria-metrics-single-admin-prd-prd-server".
-    nameOverride: victoria-metrics-single
-    namespace: monitoring
-    chartDir: victoria-metrics-single
-    valuesDir: victoria-metrics-single
-  - name: vmagent
-    # The scraper — pulls from the same targets the Prometheus server
-    # used to scrape directly (kubelet's cAdvisor endpoint, and anything
-    # carrying a prometheus.io/scrape annotation, e.g. node-exporter
-    # below) and remote_writes into victoria-metrics-single. Needs that
-    # component's Service name, so bring it up after, not before.
-    nameOverride: vmagent
-    namespace: monitoring
-    chartDir: vmagent
-    valuesDir: vmagent
-  - name: node-exporter
-    # Host-level metrics (disk/memory/load) — independent of which TSDB
-    # stores them, so vendored standalone rather than as a subchart of
-    # anything. Was a subchart of the (now removed) Prometheus server
-    # entry; moved out to its own release when that server was replaced,
-    # since victoria-metrics-single has no equivalent bundled subchart.
-    nameOverride: node-exporter
-    namespace: monitoring
-    chartDir: node-exporter
-    valuesDir: node-exporter
-  - name: grafana
-    # Dashboards over VictoriaMetrics — see that chart for why "type:
-    # prometheus" is correct for a VictoriaMetrics URL. This directory
-    # already held a fully-vendored old Grafana chart (v6.58.7) from the
-    # original Meesho monorepo import with generic production config
-    # (fullnameOverride: grafana-infra-prd) — removed and re-vendored
-    # fresh as a thin wrapper, same treatment as victoria-metrics-single.
-    #
-    # Requires secretstores/grafana-admin-credentials.yaml to have synced
-    # first — the pod falls back to a randomly-generated admin password
-    # nobody has if that Secret does not exist yet when it boots (not a
-    # crash, just an inaccessible login until the Secret exists and the
-    # pod restarts).
-    nameOverride: grafana
-    namespace: monitoring
-    chartDir: grafana
-    valuesDir: grafana