devops-infra-argo-config-gcp/docs/platform/runbooks/argocd-sync-failure.md

Runbook: ArgoCD Infra App Sync Failure

Decision tree for diagnosing and resolving sync failures on infrastructure tool Applications managed by this repo.

Audience: Platform engineers, on-call SREs, and agents in advisory mode (Layer 2).

Key context: Infra apps auto-sync from main. A sync failure means either a bad merge or an external cluster issue.

---

Triage — is it this repo's fault?

Sync failure observed
│
├─ Is the Application in "Unknown" or "Missing" state?
│  └─ YES → The incubator file may be missing or malformed.
│           Check: incubator/<env>/<cluster>.yaml exists and is valid YAML.
│           Check: The incubator Application itself is healthy in the admin cluster.
│
├─ Is the error "helm template failed" or "render error"?
│  └─ YES → Chart or values problem. Go to Section 1.
│
├─ Is the error "namespace not found" or "destination not found"?
│  └─ YES → Cluster routing problem. Go to Section 2.
│
├─ Is the error "ComparisonError" or "already exists"?
│  └─ YES → Name collision. Go to Section 3.
│
└─ Is the error a Kubernetes API error (forbidden, quota, etc.)?
   └─ YES → Cluster-side issue. Go to Section 4.

---

Section 1: Helm render failure

Symptom: ArgoCD shows "helm template failed" or similar render error.

Diagnosis:

# Reproduce locally
helm template generic-argo-apps-chart/ \
  -f values/<env>/<cluster>-values.yaml

# Check for YAML syntax errors
yamllint values/<env>/<cluster>-values.yaml

Common causes:

Cause	Fix
chartDir doesn't exist in devops-infra-helm-charts/helm-templates/	Create the chart directory in the sister repo, or fix the chartDir value
valuesDir doesn't exist in devops-infra-helm-charts/helm-overrides/<cluster>/	Create the values override, or fix the valuesDir value
YAML syntax error in values file	Fix the YAML (missing colon, bad indentation, etc.)
additionalValueFiles path doesn't exist	Fix the path or remove the entry
Helm chart has a breaking change	Check recent commits in devops-infra-helm-charts

---

Section 2: Cluster routing / namespace issue

Symptom: "destination cluster not found" or "namespace not found."

Diagnosis:

# Check the cluster name in the values file
grep -A2 "clusterSpec:" values/<env>/<cluster>-values.yaml

# Verify the cluster is registered in ArgoCD
argocd cluster list | grep <cluster-name>

Common causes:

Cause	Fix
clusterSpec.destination.name doesn't match GKE cluster name	Fix the name in the values file
Cluster was recently provisioned but not yet registered in ArgoCD	Register the cluster via ArgoCD CLI or Terraform
Cluster was decommissioned	Remove the incubator + values files

---

Section 3: Application name collision

Symptom: "already exists" or "ComparisonError" for an Application.

Diagnosis:

# Check for duplicate names in the values file
grep '^\s*- name:' values/<env>/<cluster>-values.yaml | sort | uniq -d

# Check for nameOverride collisions
grep 'nameOverride:' values/<env>/<cluster>-values.yaml

Common causes:

Cause	Fix
Two appSpec entries have the same name	Rename one or add nameOverride
An Application with the same name exists from a different source	Use nameOverride to disambiguate
Name was changed but old Application wasn't cleaned up	Delete the orphaned Application via ArgoCD CLI

---

Section 4: Cluster-side issue

Symptom: Kubernetes API errors — forbidden, quota exceeded, node selector mismatch, etc.

Diagnosis: This is not a repo-side issue. The chart and values are correct, but the cluster can't fulfill the request.

Common causes:

Cause	Fix
Namespace quota exceeded	Request quota increase or reduce resource requests
Node selector doesn't match any node	Verify nodepool configuration in Terraform
RBAC / service account permissions	Check the AppProject scope and cluster RBAC
CRDs not installed	Install required CRDs before deploying the tool

---

Section 5: Tool deployed to wrong cluster or namespace

Symptom: A tool appears in an unexpected cluster or namespace.

Diagnosis:

# Check where the tool is configured
grep -rl 'name: <tool>' values/

# Verify the values file's cluster destination
grep -A2 "clusterSpec:" values/<env>/<cluster>-values.yaml

# Verify the namespace in the appSpec entry
grep -A4 'name: <tool>' values/<env>/<cluster>-values.yaml

Common causes:

Cause	Fix
appSpec entry added to wrong values file	Move to correct cluster's values file
clusterSpec.destination.name is wrong	Fix the cluster name
Namespace typo	Fix the namespace field in the appSpec entry

---

Emergency: revert a bad merge

If a bad merge causes widespread sync failures:

1. Do NOT force-push to main. This violates R1 and R11.
2. Open a revert PR: git revert <bad-commit> && git push origin revert-branch
3. Get expedited platform-team review and merge the revert.
4. Auto-sync will pick up the revert within minutes.

---

Escalation

If the above doesn't resolve the issue:

• Platform team Slack: Post in #devops-tech with the Application name, cluster, and error message.
• ArgoCD admin UI: Access via the admin cluster to inspect Application state directly.