# Runbook: ArgoCD Infra App Sync Failure

> Decision tree for diagnosing and resolving sync failures on infrastructure tool Applications managed by this repo.
>
> **Audience:** Platform engineers, on-call SREs, and agents in advisory mode (Layer 2).
>
> **Key context:** Infra apps auto-sync from `main`. A sync failure means either a bad merge or an external cluster issue.

---

## Triage — is it this repo's fault?

```
Sync failure observed
│
├─ Is the Application in "Unknown" or "Missing" state?
│  └─ YES → The incubator file may be missing or malformed.
│           Check: incubator/<env>/<cluster>.yaml exists and is valid YAML.
│           Check: The incubator Application itself is healthy in the admin cluster.
│
├─ Is the error "helm template failed" or "render error"?
│  └─ YES → Chart or values problem. Go to Section 1.
│
├─ Is the error "namespace not found" or "destination not found"?
│  └─ YES → Cluster routing problem. Go to Section 2.
│
├─ Is the error "ComparisonError" or "already exists"?
│  └─ YES → Name collision. Go to Section 3.
│
└─ Is the error a Kubernetes API error (forbidden, quota, etc.)?
   └─ YES → Cluster-side issue. Go to Section 4.
```

---

## Section 1: Helm render failure

**Symptom:** ArgoCD shows "helm template failed" or similar render error.

**Diagnosis:**

```bash
# Reproduce locally
helm template generic-argo-apps-chart/ \
  -f values/<env>/<cluster>-values.yaml

# Check for YAML syntax errors
yamllint values/<env>/<cluster>-values.yaml
```

**Common causes:**

| Cause | Fix |
| ----- | --- |
| `chartDir` doesn't exist in `devops-infra-helm-charts/helm-templates/` | Create the chart directory in the sister repo, or fix the `chartDir` value |
| `valuesDir` doesn't exist in `devops-infra-helm-charts/helm-overrides/<cluster>/` | Create the values override, or fix the `valuesDir` value |
| YAML syntax error in values file | Fix the YAML (missing colon, bad indentation, etc.) |
| `additionalValueFiles` path doesn't exist | Fix the path or remove the entry |
| Helm chart has a breaking change | Check recent commits in `devops-infra-helm-charts` |

---

## Section 2: Cluster routing / namespace issue

**Symptom:** "destination cluster not found" or "namespace not found."

**Diagnosis:**

```bash
# Check the cluster name in the values file
grep -A2 "clusterSpec:" values/<env>/<cluster>-values.yaml

# Verify the cluster is registered in ArgoCD
argocd cluster list | grep <cluster-name>
```

**Common causes:**

| Cause | Fix |
| ----- | --- |
| `clusterSpec.destination.name` doesn't match GKE cluster name | Fix the name in the values file |
| Cluster was recently provisioned but not yet registered in ArgoCD | Register the cluster via ArgoCD CLI or Terraform |
| Cluster was decommissioned | Remove the incubator + values files |

---

## Section 3: Application name collision

**Symptom:** "already exists" or "ComparisonError" for an Application.

**Diagnosis:**

```bash
# Check for duplicate names in the values file
grep '^\s*- name:' values/<env>/<cluster>-values.yaml | sort | uniq -d

# Check for nameOverride collisions
grep 'nameOverride:' values/<env>/<cluster>-values.yaml
```

**Common causes:**

| Cause | Fix |
| ----- | --- |
| Two appSpec entries have the same `name` | Rename one or add `nameOverride` |
| An Application with the same name exists from a different source | Use `nameOverride` to disambiguate |
| Name was changed but old Application wasn't cleaned up | Delete the orphaned Application via ArgoCD CLI |

---

## Section 4: Cluster-side issue

**Symptom:** Kubernetes API errors — forbidden, quota exceeded, node selector mismatch, etc.

**Diagnosis:** This is not a repo-side issue. The chart and values are correct, but the cluster can't fulfill the request.

**Common causes:**

| Cause | Fix |
| ----- | --- |
| Namespace quota exceeded | Request quota increase or reduce resource requests |
| Node selector doesn't match any node | Verify nodepool configuration in Terraform |
| RBAC / service account permissions | Check the AppProject scope and cluster RBAC |
| CRDs not installed | Install required CRDs before deploying the tool |

---

## Section 5: Tool deployed to wrong cluster or namespace

**Symptom:** A tool appears in an unexpected cluster or namespace.

**Diagnosis:**

```bash
# Check where the tool is configured
grep -rl 'name: <tool>' values/

# Verify the values file's cluster destination
grep -A2 "clusterSpec:" values/<env>/<cluster>-values.yaml

# Verify the namespace in the appSpec entry
grep -A4 'name: <tool>' values/<env>/<cluster>-values.yaml
```

**Common causes:**

| Cause | Fix |
| ----- | --- |
| appSpec entry added to wrong values file | Move to correct cluster's values file |
| `clusterSpec.destination.name` is wrong | Fix the cluster name |
| Namespace typo | Fix the `namespace` field in the appSpec entry |

---

## Emergency: revert a bad merge

If a bad merge causes widespread sync failures:

1. **Do NOT force-push to `main`.** This violates R1 and R11.
2. Open a revert PR: `git revert <bad-commit> && git push origin revert-branch`
3. Get expedited platform-team review and merge the revert.
4. Auto-sync will pick up the revert within minutes.

---

## Escalation

If the above doesn't resolve the issue:
- **Platform team Slack:** Post in `#devops-tech` with the Application name, cluster, and error message.
- **ArgoCD admin UI:** Access via the admin cluster to inspect Application state directly.