171 lines
5.4 KiB
Markdown
171 lines
5.4 KiB
Markdown
# Runbook: ArgoCD Infra App Sync Failure
|
|
|
|
> Decision tree for diagnosing and resolving sync failures on infrastructure tool Applications managed by this repo.
|
|
>
|
|
> **Audience:** Platform engineers, on-call SREs, and agents in advisory mode (Layer 2).
|
|
>
|
|
> **Key context:** Infra apps auto-sync from `main`. A sync failure means either a bad merge or an external cluster issue.
|
|
|
|
---
|
|
|
|
## Triage — is it this repo's fault?
|
|
|
|
```
|
|
Sync failure observed
|
|
│
|
|
├─ Is the Application in "Unknown" or "Missing" state?
|
|
│ └─ YES → The incubator file may be missing or malformed.
|
|
│ Check: incubator/<env>/<cluster>.yaml exists and is valid YAML.
|
|
│ Check: The incubator Application itself is healthy in the admin cluster.
|
|
│
|
|
├─ Is the error "helm template failed" or "render error"?
|
|
│ └─ YES → Chart or values problem. Go to Section 1.
|
|
│
|
|
├─ Is the error "namespace not found" or "destination not found"?
|
|
│ └─ YES → Cluster routing problem. Go to Section 2.
|
|
│
|
|
├─ Is the error "ComparisonError" or "already exists"?
|
|
│ └─ YES → Name collision. Go to Section 3.
|
|
│
|
|
└─ Is the error a Kubernetes API error (forbidden, quota, etc.)?
|
|
└─ YES → Cluster-side issue. Go to Section 4.
|
|
```
|
|
|
|
---
|
|
|
|
## Section 1: Helm render failure
|
|
|
|
**Symptom:** ArgoCD shows "helm template failed" or similar render error.
|
|
|
|
**Diagnosis:**
|
|
|
|
```bash
|
|
# Reproduce locally
|
|
helm template generic-argo-apps-chart/ \
|
|
-f values/<env>/<cluster>-values.yaml
|
|
|
|
# Check for YAML syntax errors
|
|
yamllint values/<env>/<cluster>-values.yaml
|
|
```
|
|
|
|
**Common causes:**
|
|
|
|
| Cause | Fix |
|
|
| ----- | --- |
|
|
| `chartDir` doesn't exist in `devops-infra-helm-charts/helm-templates/` | Create the chart directory in the sister repo, or fix the `chartDir` value |
|
|
| `valuesDir` doesn't exist in `devops-infra-helm-charts/helm-overrides/<cluster>/` | Create the values override, or fix the `valuesDir` value |
|
|
| YAML syntax error in values file | Fix the YAML (missing colon, bad indentation, etc.) |
|
|
| `additionalValueFiles` path doesn't exist | Fix the path or remove the entry |
|
|
| Helm chart has a breaking change | Check recent commits in `devops-infra-helm-charts` |
|
|
|
|
---
|
|
|
|
## Section 2: Cluster routing / namespace issue
|
|
|
|
**Symptom:** "destination cluster not found" or "namespace not found."
|
|
|
|
**Diagnosis:**
|
|
|
|
```bash
|
|
# Check the cluster name in the values file
|
|
grep -A2 "clusterSpec:" values/<env>/<cluster>-values.yaml
|
|
|
|
# Verify the cluster is registered in ArgoCD
|
|
argocd cluster list | grep <cluster-name>
|
|
```
|
|
|
|
**Common causes:**
|
|
|
|
| Cause | Fix |
|
|
| ----- | --- |
|
|
| `clusterSpec.destination.name` doesn't match GKE cluster name | Fix the name in the values file |
|
|
| Cluster was recently provisioned but not yet registered in ArgoCD | Register the cluster via ArgoCD CLI or Terraform |
|
|
| Cluster was decommissioned | Remove the incubator + values files |
|
|
|
|
---
|
|
|
|
## Section 3: Application name collision
|
|
|
|
**Symptom:** "already exists" or "ComparisonError" for an Application.
|
|
|
|
**Diagnosis:**
|
|
|
|
```bash
|
|
# Check for duplicate names in the values file
|
|
grep '^\s*- name:' values/<env>/<cluster>-values.yaml | sort | uniq -d
|
|
|
|
# Check for nameOverride collisions
|
|
grep 'nameOverride:' values/<env>/<cluster>-values.yaml
|
|
```
|
|
|
|
**Common causes:**
|
|
|
|
| Cause | Fix |
|
|
| ----- | --- |
|
|
| Two appSpec entries have the same `name` | Rename one or add `nameOverride` |
|
|
| An Application with the same name exists from a different source | Use `nameOverride` to disambiguate |
|
|
| Name was changed but old Application wasn't cleaned up | Delete the orphaned Application via ArgoCD CLI |
|
|
|
|
---
|
|
|
|
## Section 4: Cluster-side issue
|
|
|
|
**Symptom:** Kubernetes API errors — forbidden, quota exceeded, node selector mismatch, etc.
|
|
|
|
**Diagnosis:** This is not a repo-side issue. The chart and values are correct, but the cluster can't fulfill the request.
|
|
|
|
**Common causes:**
|
|
|
|
| Cause | Fix |
|
|
| ----- | --- |
|
|
| Namespace quota exceeded | Request quota increase or reduce resource requests |
|
|
| Node selector doesn't match any node | Verify nodepool configuration in Terraform |
|
|
| RBAC / service account permissions | Check the AppProject scope and cluster RBAC |
|
|
| CRDs not installed | Install required CRDs before deploying the tool |
|
|
|
|
---
|
|
|
|
## Section 5: Tool deployed to wrong cluster or namespace
|
|
|
|
**Symptom:** A tool appears in an unexpected cluster or namespace.
|
|
|
|
**Diagnosis:**
|
|
|
|
```bash
|
|
# Check where the tool is configured
|
|
grep -rl 'name: <tool>' values/
|
|
|
|
# Verify the values file's cluster destination
|
|
grep -A2 "clusterSpec:" values/<env>/<cluster>-values.yaml
|
|
|
|
# Verify the namespace in the appSpec entry
|
|
grep -A4 'name: <tool>' values/<env>/<cluster>-values.yaml
|
|
```
|
|
|
|
**Common causes:**
|
|
|
|
| Cause | Fix |
|
|
| ----- | --- |
|
|
| appSpec entry added to wrong values file | Move to correct cluster's values file |
|
|
| `clusterSpec.destination.name` is wrong | Fix the cluster name |
|
|
| Namespace typo | Fix the `namespace` field in the appSpec entry |
|
|
|
|
---
|
|
|
|
## Emergency: revert a bad merge
|
|
|
|
If a bad merge causes widespread sync failures:
|
|
|
|
1. **Do NOT force-push to `main`.** This violates R1 and R11.
|
|
2. Open a revert PR: `git revert <bad-commit> && git push origin revert-branch`
|
|
3. Get expedited platform-team review and merge the revert.
|
|
4. Auto-sync will pick up the revert within minutes.
|
|
|
|
---
|
|
|
|
## Escalation
|
|
|
|
If the above doesn't resolve the issue:
|
|
- **Platform team Slack:** Post in `#devops-tech` with the Application name, cluster, and error message.
|
|
- **ArgoCD admin UI:** Access via the admin cluster to inspect Application state directly.
|