From c172756f340979d35d15ee55c4f1e01b2b03751f Mon Sep 17 00:00:00 2001 From: Mukul Sharma Date: Sat, 12 Sep 2026 15:19:35 +0530 Subject: [PATCH] Fix diff failures on server-side-apply apps: ServerSideDiff MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit contour, cert-manager and external-secrets could not be compared or synced: error calculating structured merge diff: error building typed value from live resource: .status.terminatingReplicas: field not declared in schema The ServerSideApply sync option makes Argo CD compute the diff locally against a Kubernetes schema compiled into its own binary. Argo CD v2.13 is older than this cluster: GKE runs 1.35, and Deployments there carry status.terminatingReplicas, which went beta and on-by-default in 1.33. Argo CD's schema has never heard of the field, so the diff aborts before any sync can happen. Nothing is wrong with the manifests, and only the three apps using SSA are affected. ServerSideDiff asks the API server to compute the diff via a dry-run apply, so the schema in use is the cluster's own. Beta since v2.10 and supported on the running version. The generic chart could not express this — Application metadata had no annotations block at all — so it gains an optional per-entry compareOptions list rather than the annotation being hardcoded. This is a workaround for an out-of-date Argo CD, not a fix. The fix is upgrading to a build whose bundled schema matches the cluster; every value key this repo relies on already exists in chart 10.8.4 (Argo CD v3.5.2), so that upgrade is mostly a vendoring exercise plus the 3.0 breaking changes (logs RBAC now enforced, fine-grained RBAC inheritance, resource tracking moving from labels to annotations). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N --- .../templates/genericTemplate.yaml | 15 +++++++++++++ ...or-infra-gke-toolshed-prd-usc1-values.yaml | 21 +++++++++++++++++++ 2 files changed, 36 insertions(+) diff --git a/generic-argo-apps-chart/templates/genericTemplate.yaml b/generic-argo-apps-chart/templates/genericTemplate.yaml index 95ac7c0..1f2ecea 100644 --- a/generic-argo-apps-chart/templates/genericTemplate.yaml +++ b/generic-argo-apps-chart/templates/genericTemplate.yaml @@ -24,11 +24,26 @@ {{ $argoAppNamespace := $top.Values.argocdSpec.namespace }} +{{- $compareOptions := $config.compareOptions | default list -}} + apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: '{{- if $appNameOverride }}{{ $appNameOverride }}{{- else }}{{ printf "%s-%s-%s" $appName $cluster $env }}{{- end }}' namespace: {{ $argoAppNamespace }} +{{- if $compareOptions }} + annotations: + # Opt-in per appSpec entry, as a list of Argo CD compare options. + # + # The one that matters here is ServerSideDiff=true, which is the escape + # hatch for "field not declared in schema" diff failures. Argo CD + # normally computes the diff locally against a Kubernetes schema baked + # into its own binary, so a cluster newer than Argo CD has fields Argo + # CD has never heard of and the diff aborts. This option asks the API + # server to compute the diff instead (a dry-run apply), and the API + # server necessarily knows its own fields. + argocd.argoproj.io/compare-options: {{ join "," $compareOptions | quote }} +{{- end }} labels: {{ toYaml $labels | indent 4 }} finalizers: diff --git a/values/incubator-infra-gke-toolshed-prd-usc1-values.yaml b/values/incubator-infra-gke-toolshed-prd-usc1-values.yaml index 8122840..9eb8c10 100644 --- a/values/incubator-infra-gke-toolshed-prd-usc1-values.yaml +++ b/values/incubator-infra-gke-toolshed-prd-usc1-values.yaml @@ -51,6 +51,19 @@ appSpec: # Contour's HTTPProxy CRD embeds a large schema, same 256KiB # last-applied-configuration problem as cert-manager's. serverSideApply: true + # Required alongside serverSideApply on this cluster. That sync option + # makes Argo CD compute the diff with the Kubernetes schema compiled + # into its own binary, and Argo CD v2.13 predates this cluster: live + # Deployments here carry status.terminatingReplicas, beta and enabled by + # default since Kubernetes 1.33, which that schema has never heard of. + # The diff then aborts with "field not declared in schema" and the app + # can neither be compared nor synced. ServerSideDiff hands the diff to + # the API server, which necessarily knows its own fields. + # + # This is a workaround for an out-of-date Argo CD, not a fix. The fix is + # upgrading Argo CD to a build whose schema matches the cluster. + compareOptions: + - ServerSideDiff=true - name: vault # Fresh install here, unlike the homelab's adoption of a running Vault. # nameOverride pinned so the release is "vault" in namespace "vault": @@ -70,6 +83,10 @@ appSpec: # cert-manager's CRDs embed large OpenAPI schemas; SSA avoids the # 256KiB last-applied-configuration limit, same as external-secrets. serverSideApply: true + # See the contour entry above: server-side apply diffs against Argo CD's + # own bundled schema, which is older than this cluster. + compareOptions: + - ServerSideDiff=true - name: external-secrets # Every credential in the cluster arrives through this. nameOverride is # not cosmetic here: without it the release becomes @@ -84,6 +101,10 @@ appSpec: # ClusterSecretStore's CRD embeds a large provider schema and exceeds # the 256KiB annotation limit on a client-side apply. serverSideApply: true + # See the contour entry above: server-side apply diffs against Argo CD's + # own bundled schema, which is older than this cluster. + compareOptions: + - ServerSideDiff=true - name: harbor # Object names are all prefixed with the release name, and the pipeline # refers to them, so it is pinned.