added repo
This commit is contained in:
Your Name
@@ -0,0 +1,170 @@
|
||||
# Runbook: ArgoCD Infra App Sync Failure
|
||||
|
||||
> Decision tree for diagnosing and resolving sync failures on infrastructure tool Applications managed by this repo.
|
||||
>
|
||||
> **Audience:** Platform engineers, on-call SREs, and agents in advisory mode (Layer 2).
|
||||
>
|
||||
> **Key context:** Infra apps auto-sync from `main`. A sync failure means either a bad merge or an external cluster issue.
|
||||
|
||||
---
|
||||
|
||||
## Triage — is it this repo's fault?
|
||||
|
||||
```
|
||||
Sync failure observed
|
||||
│
|
||||
├─ Is the Application in "Unknown" or "Missing" state?
|
||||
│ └─ YES → The incubator file may be missing or malformed.
|
||||
│ Check: incubator/<env>/<cluster>.yaml exists and is valid YAML.
|
||||
│ Check: The incubator Application itself is healthy in the admin cluster.
|
||||
│
|
||||
├─ Is the error "helm template failed" or "render error"?
|
||||
│ └─ YES → Chart or values problem. Go to Section 1.
|
||||
│
|
||||
├─ Is the error "namespace not found" or "destination not found"?
|
||||
│ └─ YES → Cluster routing problem. Go to Section 2.
|
||||
│
|
||||
├─ Is the error "ComparisonError" or "already exists"?
|
||||
│ └─ YES → Name collision. Go to Section 3.
|
||||
│
|
||||
└─ Is the error a Kubernetes API error (forbidden, quota, etc.)?
|
||||
└─ YES → Cluster-side issue. Go to Section 4.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Section 1: Helm render failure
|
||||
|
||||
**Symptom:** ArgoCD shows "helm template failed" or similar render error.
|
||||
|
||||
**Diagnosis:**
|
||||
|
||||
```bash
|
||||
# Reproduce locally
|
||||
helm template generic-argo-apps-chart/ \
|
||||
-f values/<env>/<cluster>-values.yaml
|
||||
|
||||
# Check for YAML syntax errors
|
||||
yamllint values/<env>/<cluster>-values.yaml
|
||||
```
|
||||
|
||||
**Common causes:**
|
||||
|
||||
| Cause | Fix |
|
||||
| ----- | --- |
|
||||
| `chartDir` doesn't exist in `devops-infra-helm-charts/helm-templates/` | Create the chart directory in the sister repo, or fix the `chartDir` value |
|
||||
| `valuesDir` doesn't exist in `devops-infra-helm-charts/helm-overrides/<cluster>/` | Create the values override, or fix the `valuesDir` value |
|
||||
| YAML syntax error in values file | Fix the YAML (missing colon, bad indentation, etc.) |
|
||||
| `additionalValueFiles` path doesn't exist | Fix the path or remove the entry |
|
||||
| Helm chart has a breaking change | Check recent commits in `devops-infra-helm-charts` |
|
||||
|
||||
---
|
||||
|
||||
## Section 2: Cluster routing / namespace issue
|
||||
|
||||
**Symptom:** "destination cluster not found" or "namespace not found."
|
||||
|
||||
**Diagnosis:**
|
||||
|
||||
```bash
|
||||
# Check the cluster name in the values file
|
||||
grep -A2 "clusterSpec:" values/<env>/<cluster>-values.yaml
|
||||
|
||||
# Verify the cluster is registered in ArgoCD
|
||||
argocd cluster list | grep <cluster-name>
|
||||
```
|
||||
|
||||
**Common causes:**
|
||||
|
||||
| Cause | Fix |
|
||||
| ----- | --- |
|
||||
| `clusterSpec.destination.name` doesn't match GKE cluster name | Fix the name in the values file |
|
||||
| Cluster was recently provisioned but not yet registered in ArgoCD | Register the cluster via ArgoCD CLI or Terraform |
|
||||
| Cluster was decommissioned | Remove the incubator + values files |
|
||||
|
||||
---
|
||||
|
||||
## Section 3: Application name collision
|
||||
|
||||
**Symptom:** "already exists" or "ComparisonError" for an Application.
|
||||
|
||||
**Diagnosis:**
|
||||
|
||||
```bash
|
||||
# Check for duplicate names in the values file
|
||||
grep '^\s*- name:' values/<env>/<cluster>-values.yaml | sort | uniq -d
|
||||
|
||||
# Check for nameOverride collisions
|
||||
grep 'nameOverride:' values/<env>/<cluster>-values.yaml
|
||||
```
|
||||
|
||||
**Common causes:**
|
||||
|
||||
| Cause | Fix |
|
||||
| ----- | --- |
|
||||
| Two appSpec entries have the same `name` | Rename one or add `nameOverride` |
|
||||
| An Application with the same name exists from a different source | Use `nameOverride` to disambiguate |
|
||||
| Name was changed but old Application wasn't cleaned up | Delete the orphaned Application via ArgoCD CLI |
|
||||
|
||||
---
|
||||
|
||||
## Section 4: Cluster-side issue
|
||||
|
||||
**Symptom:** Kubernetes API errors — forbidden, quota exceeded, node selector mismatch, etc.
|
||||
|
||||
**Diagnosis:** This is not a repo-side issue. The chart and values are correct, but the cluster can't fulfill the request.
|
||||
|
||||
**Common causes:**
|
||||
|
||||
| Cause | Fix |
|
||||
| ----- | --- |
|
||||
| Namespace quota exceeded | Request quota increase or reduce resource requests |
|
||||
| Node selector doesn't match any node | Verify nodepool configuration in Terraform |
|
||||
| RBAC / service account permissions | Check the AppProject scope and cluster RBAC |
|
||||
| CRDs not installed | Install required CRDs before deploying the tool |
|
||||
|
||||
---
|
||||
|
||||
## Section 5: Tool deployed to wrong cluster or namespace
|
||||
|
||||
**Symptom:** A tool appears in an unexpected cluster or namespace.
|
||||
|
||||
**Diagnosis:**
|
||||
|
||||
```bash
|
||||
# Check where the tool is configured
|
||||
grep -rl 'name: <tool>' values/
|
||||
|
||||
# Verify the values file's cluster destination
|
||||
grep -A2 "clusterSpec:" values/<env>/<cluster>-values.yaml
|
||||
|
||||
# Verify the namespace in the appSpec entry
|
||||
grep -A4 'name: <tool>' values/<env>/<cluster>-values.yaml
|
||||
```
|
||||
|
||||
**Common causes:**
|
||||
|
||||
| Cause | Fix |
|
||||
| ----- | --- |
|
||||
| appSpec entry added to wrong values file | Move to correct cluster's values file |
|
||||
| `clusterSpec.destination.name` is wrong | Fix the cluster name |
|
||||
| Namespace typo | Fix the `namespace` field in the appSpec entry |
|
||||
|
||||
---
|
||||
|
||||
## Emergency: revert a bad merge
|
||||
|
||||
If a bad merge causes widespread sync failures:
|
||||
|
||||
1. **Do NOT force-push to `main`.** This violates R1 and R11.
|
||||
2. Open a revert PR: `git revert <bad-commit> && git push origin revert-branch`
|
||||
3. Get expedited platform-team review and merge the revert.
|
||||
4. Auto-sync will pick up the revert within minutes.
|
||||
|
||||
---
|
||||
|
||||
## Escalation
|
||||
|
||||
If the above doesn't resolve the issue:
|
||||
- **Platform team Slack:** Post in `#devops-tech` with the Application name, cluster, and error message.
|
||||
- **ArgoCD admin UI:** Access via the admin cluster to inspect Application state directly.
|
||||
@@ -0,0 +1,138 @@
|
||||
# Runbook: Deployment Stuck After Sync
|
||||
|
||||
> Symptom → diagnosis → remediation for tools that ArgoCD reports as `Synced` but whose Kubernetes resources are not healthy.
|
||||
>
|
||||
> **Layer:** 2 (Advisory). Agents diagnose and suggest; humans execute kubectl/ArgoCD commands.
|
||||
>
|
||||
> Per AI Blitz Plan §5.2. See also: [argocd-sync-failure.md](argocd-sync-failure.md) (for sync errors), [render-failure.md](render-failure.md) (for Helm render errors).
|
||||
|
||||
---
|
||||
|
||||
## Symptoms
|
||||
|
||||
- ArgoCD Application shows `Synced` but `Degraded` or `Unknown` health
|
||||
- Pods are in `Pending`, `CrashLoopBackOff`, `ImagePullBackOff`, or `OOMKilled`
|
||||
- Tool was working before a chart upgrade or values change
|
||||
- Application is `Synced + Healthy` but the tool isn't functioning as expected
|
||||
|
||||
---
|
||||
|
||||
## Decision tree
|
||||
|
||||
```text
|
||||
Application shows Synced but Degraded?
|
||||
│
|
||||
├── Check pod status in the tool's namespace:
|
||||
│ kubectl get pods -n <namespace> --context=<cluster>
|
||||
│
|
||||
├── Pod status is Pending?
|
||||
│ └── → [A] Resource constraints or node issues
|
||||
│
|
||||
├── Pod status is CrashLoopBackOff?
|
||||
│ └── → [B] Application crash — check logs
|
||||
│
|
||||
├── Pod status is ImagePullBackOff or ErrImagePull?
|
||||
│ └── → [C] Image registry issue
|
||||
│
|
||||
├── Pod status is OOMKilled?
|
||||
│ └── → [D] Memory limit too low
|
||||
│
|
||||
├── Pod runs but tool is unhealthy?
|
||||
│ └── → [E] Config or connectivity issue
|
||||
│
|
||||
└── Pods are fine but ArgoCD shows Degraded?
|
||||
└── → [F] Health check misconfiguration
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## [A] Pod Pending
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
kubectl describe pod <pod-name> -n <namespace> --context=<cluster> | grep -A 20 Events
|
||||
```
|
||||
|
||||
Common causes and fixes:
|
||||
|
||||
| Event message | Cause | Fix |
|
||||
| ------------- | ----- | --- |
|
||||
| `Insufficient cpu/memory` | Node resource exhaustion | Check node capacity; may need to adjust resource requests in `custom-values.yaml` (change in `devops-infra-helm-charts`) |
|
||||
| `did not match node affinity` | Node selector or affinity mismatch | Review `nodeSelector`/`affinity` in `custom-values.yaml` |
|
||||
| `PersistentVolumeClaim not bound` | PVC not provisioned | Check storage class and PVC events |
|
||||
| `Unschedulable` | No nodes available | Check if cluster has sufficient nodes or if Karpenter/cluster-autoscaler is stuck |
|
||||
|
||||
---
|
||||
|
||||
## [B] CrashLoopBackOff
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
kubectl logs <pod-name> -n <namespace> --context=<cluster> --previous
|
||||
```
|
||||
|
||||
Look for startup errors, missing config, failed connections. The fix usually requires changing `custom-values.yaml` in `devops-infra-helm-charts` — that's a separate PR in the sister repo.
|
||||
|
||||
---
|
||||
|
||||
## [C] ImagePullBackOff
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
kubectl describe pod <pod-name> -n <namespace> --context=<cluster> | grep image
|
||||
```
|
||||
|
||||
Common causes:
|
||||
- Image tag doesn't exist (bad chart version pin)
|
||||
- Image registry credentials expired (check `external-secrets` or `imagePullSecrets`)
|
||||
- Private registry unreachable from the cluster
|
||||
|
||||
Fix depends on root cause. Usually requires a chart fix in `devops-infra-helm-charts` or a credential rotation.
|
||||
|
||||
---
|
||||
|
||||
## [D] OOMKilled
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
kubectl top pod -n <namespace> --context=<cluster>
|
||||
kubectl describe pod <pod-name> -n <namespace> --context=<cluster> | grep -A 5 "Last State"
|
||||
```
|
||||
|
||||
Fix: Increase `resources.limits.memory` in `custom-values.yaml` in `devops-infra-helm-charts`. Open PR there.
|
||||
|
||||
---
|
||||
|
||||
## [E] Config or Connectivity Issue
|
||||
|
||||
Tool is running but not working correctly.
|
||||
|
||||
**Diagnosis:** Check tool-specific logs. Most infra tools write structured logs.
|
||||
|
||||
Common causes:
|
||||
- Wrong endpoint URL in `custom-values.yaml`
|
||||
- Service dependency not available (e.g., Vault unreachable for external-secrets)
|
||||
- Wrong namespace for a referenced service
|
||||
|
||||
Fix: Update `custom-values.yaml` in `devops-infra-helm-charts`.
|
||||
|
||||
---
|
||||
|
||||
## [F] ArgoCD Health Check Misconfiguration
|
||||
|
||||
ArgoCD uses health checks to determine if an Application is healthy. Some custom resources have incorrect or missing health checks.
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
argocd app get <appName> --show-operation | grep -A 10 "Health Status"
|
||||
```
|
||||
|
||||
If resources are actually healthy but ArgoCD says Degraded, this may be a health check definition issue in the ArgoCD config itself (not this repo).
|
||||
|
||||
Escalate to platform team via [escalation-matrix.md](../../global/escalation-matrix.md).
|
||||
|
||||
---
|
||||
|
||||
## Escalation
|
||||
|
||||
If diagnosis points to a cluster-level issue (node pressure, networking, GKE problem) rather than a configuration issue, escalate via [escalation-matrix.md](../../global/escalation-matrix.md).
|
||||
@@ -0,0 +1,133 @@
|
||||
# Runbook: Helm Render Failure
|
||||
|
||||
> Symptom → diagnosis → remediation for ArgoCD Applications stuck in `ComparisonError` or `ErrParsingAppProject` state due to Helm template rendering failures.
|
||||
>
|
||||
> **Layer:** 2 (Advisory). Agents diagnose and suggest fixes; humans apply.
|
||||
>
|
||||
> Per AI Blitz Plan §5.2. See also: [argocd-sync-failure.md](argocd-sync-failure.md).
|
||||
|
||||
---
|
||||
|
||||
## Symptoms
|
||||
|
||||
- ArgoCD Application shows `ComparisonError: failed to generate manifest`
|
||||
- `helm template` run locally produces YAML errors
|
||||
- Application stuck in `Unknown` or `Error` health state with no sync progress
|
||||
- Pre-commit hook `cac validate` fails on a PR with YAML parsing errors
|
||||
|
||||
---
|
||||
|
||||
## Decision tree
|
||||
|
||||
```text
|
||||
Application stuck in ComparisonError?
|
||||
│
|
||||
├── Run: argocd app get <appName> --show-operation
|
||||
│ (or: argocd app logs <appName>)
|
||||
│
|
||||
├── Error contains "values file not found"?
|
||||
│ └── → [A] Missing values file
|
||||
│
|
||||
├── Error contains "chart not found" or "no chart found"?
|
||||
│ └── → [B] Missing chart directory
|
||||
│
|
||||
├── Error contains "unmarshal" or "cannot unmarshal" or "yaml: line"?
|
||||
│ └── → [C] YAML syntax error in values or chart
|
||||
│
|
||||
├── Error contains "template: ... nil pointer" or "function not defined"?
|
||||
│ └── → [D] Chart template bug
|
||||
│
|
||||
└── Error contains "AppProject not found" or "unauthorized"?
|
||||
└── → See argocd-sync-failure.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## [A] Missing values file
|
||||
|
||||
**Cause:** The `valueFiles` path in the incubator Application YAML points to a values file that doesn't exist.
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
# Check what valueFiles the incubator references:
|
||||
grep 'valueFiles' incubator/<env>/<cluster>.yaml
|
||||
|
||||
# Verify the values file exists:
|
||||
ls values/<env>/incubator-infra-<cluster>-values.yaml
|
||||
```
|
||||
|
||||
**Fix:** Create the missing values file (use [add-new-cluster.md](../procedures/add-new-cluster.md) for a new cluster) or correct the path in the incubator YAML. Open a PR.
|
||||
|
||||
---
|
||||
|
||||
## [B] Missing chart directory
|
||||
|
||||
**Cause:** An `appSpec[].chartDir` value points to a directory that doesn't exist in `devops-infra-helm-charts/helm-templates/`.
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
# Identify which appSpec entry has the bad chartDir:
|
||||
helm template generic-argo-apps-chart/ -f values/<env>/<cluster>-values.yaml 2>&1 | head -30
|
||||
|
||||
# Verify in helm-charts repo:
|
||||
ls /path/to/devops-infra-helm-charts/helm-templates/<chartDir>/
|
||||
```
|
||||
|
||||
**Fix:** Either:
|
||||
1. Add the chart to `devops-infra-helm-charts` (work in the sister repo).
|
||||
2. Correct the `chartDir` in the values file to an existing directory.
|
||||
|
||||
Open a PR with the fix.
|
||||
|
||||
---
|
||||
|
||||
## [C] YAML syntax error
|
||||
|
||||
**Cause:** Invalid YAML in a values file (e.g., bad indentation, missing quotes, stray character).
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
yamllint values/<env>/<cluster>-values.yaml
|
||||
|
||||
# Or for more context:
|
||||
python3 -c "import yaml; yaml.safe_load(open('values/<env>/<cluster>-values.yaml'))"
|
||||
```
|
||||
|
||||
**Common causes:**
|
||||
- Stray tab character (YAML requires spaces)
|
||||
- Missing `- ` prefix on an appSpec entry
|
||||
- Unquoted special characters in namespace or name fields
|
||||
- Trailing whitespace after a value
|
||||
|
||||
**Fix:** Correct the YAML syntax. Run `yamllint` and `helm template` locally before opening PR. Pre-commit `cac validate` hook should also catch this.
|
||||
|
||||
---
|
||||
|
||||
## [D] Chart template bug
|
||||
|
||||
**Cause:** The `generic-argo-apps-chart/templates/genericTemplate.yaml` has a nil pointer dereference, undefined function, or logic error.
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
helm template generic-argo-apps-chart/ -f values/<env>/<cluster>-values.yaml 2>&1
|
||||
```
|
||||
|
||||
**Impact:** This breaks ALL Applications rendered by this chart, across ALL clusters. This is a fleet-wide outage risk.
|
||||
|
||||
**Fix:** This requires a fix to `generic-argo-apps-chart/`. This is a **HIGH RISK** operation — see [AGENT_BOUNDARIES.md](../../global/AGENT_BOUNDARIES.md). Escalate to platform team immediately via [escalation-matrix.md](../../global/escalation-matrix.md).
|
||||
|
||||
Do not attempt to fix chart templates without explicit platform-team sign-off and test with `helm template` against multiple values files.
|
||||
|
||||
---
|
||||
|
||||
## Preventive checks
|
||||
|
||||
Run before any PR that modifies values files or `generic-argo-apps-chart/`:
|
||||
|
||||
```bash
|
||||
# Test all values files in one env:
|
||||
for f in values/prd/*.yaml; do
|
||||
helm template generic-argo-apps-chart/ -f "$f" > /dev/null \
|
||||
&& echo "OK: $f" || echo "FAIL: $f"
|
||||
done
|
||||
```
|
||||
@@ -0,0 +1,116 @@
|
||||
# Runbook: Values Drift
|
||||
|
||||
> Symptom → diagnosis → remediation for inconsistent `appSpec` entries across cluster values files — same tool, different `chartDir`, `namespace`, or field values across clusters.
|
||||
>
|
||||
> **Layer:** 2 (Advisory). Agents diagnose and report; humans decide whether to remediate.
|
||||
>
|
||||
> Per AI Blitz Plan §5.2.
|
||||
|
||||
---
|
||||
|
||||
## Symptoms
|
||||
|
||||
- A tool behaves differently on two clusters that should be equivalent
|
||||
- A chart version upgrade was applied to some clusters but not others
|
||||
- `grep` across values files shows the same tool with different `chartDir` or `namespace`
|
||||
- After a fleet-wide rollout, one cluster was accidentally skipped
|
||||
|
||||
---
|
||||
|
||||
## Detection
|
||||
|
||||
### Find clusters where a tool exists
|
||||
|
||||
```bash
|
||||
grep -rl 'name: <tool>' values/prd/
|
||||
```
|
||||
|
||||
### Find the chartDir each cluster uses for a tool
|
||||
|
||||
```bash
|
||||
grep -A 4 'name: <tool>' values/prd/*.yaml | grep 'chartDir'
|
||||
```
|
||||
|
||||
### Compare a specific tool across all prd clusters
|
||||
|
||||
```bash
|
||||
# Show full appSpec entry for the tool on every cluster that has it:
|
||||
for f in values/prd/*.yaml; do
|
||||
entry=$(grep -A 5 "^\s*- name: <tool>$" "$f" 2>/dev/null)
|
||||
if [ -n "$entry" ]; then
|
||||
echo "=== $f ==="
|
||||
echo "$entry"
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
### Find clusters missing a tool that should be present
|
||||
|
||||
```bash
|
||||
# List all prd values files:
|
||||
all=$(ls values/prd/)
|
||||
# List files that have the tool:
|
||||
have=$(grep -rl 'name: <tool>' values/prd/ | xargs -I{} basename {})
|
||||
# Diff:
|
||||
diff <(echo "$all") <(echo "$have")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Interpreting drift
|
||||
|
||||
Not all drift is wrong. Some clusters legitimately have different configurations:
|
||||
|
||||
| Type of drift | Usually OK? | Action |
|
||||
| ------------- | ----------- | ------ |
|
||||
| Different `chartDir` (different version) | Only if intentional | Align versions unless the divergence is intentional (e.g., staged rollout in progress) |
|
||||
| Different `namespace` | Rarely OK | Investigate — namespace change is destructive |
|
||||
| Tool absent from a cluster | Depends | Confirm whether the cluster should have the tool |
|
||||
| Different `valuesDir` | Sometimes OK | Clusters can have cluster-specific overrides; confirm the override exists |
|
||||
| Different `nameOverride` | Rarely OK | Should only exist for collision/length reasons |
|
||||
|
||||
---
|
||||
|
||||
## Remediation
|
||||
|
||||
### Align chart versions across clusters
|
||||
|
||||
If a tool is on different versions and should be uniform, follow [upgrade-chart-version.md](../procedures/upgrade-chart-version.md) for the lagging clusters.
|
||||
|
||||
### Add a missing tool to a cluster
|
||||
|
||||
Follow [add-tool-to-cluster.md](../procedures/add-tool-to-cluster.md).
|
||||
|
||||
### Document intentional drift
|
||||
|
||||
If the divergence is intentional (e.g., Cluster A has a special `valuesDir` for a regional override), add a comment to the values file entry:
|
||||
|
||||
```yaml
|
||||
- name: coredns
|
||||
namespace: kube-system
|
||||
chartDir: coredns
|
||||
valuesDir: coredns-special-region # Intentional: this cluster uses a non-standard DNS config
|
||||
additionalValueFiles:
|
||||
- ../../helm-templates/coredns/gcp-ase1a-values.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Periodic drift check
|
||||
|
||||
Run this to generate a drift report across all prd clusters for a tool:
|
||||
|
||||
```bash
|
||||
echo "Tool: <tool>" && echo "---"
|
||||
for f in values/prd/*.yaml; do
|
||||
cluster=$(basename "$f" -values.yaml)
|
||||
entry=$(grep -A 5 "^\s*- name: <tool>$" "$f" 2>/dev/null)
|
||||
if [ -n "$entry" ]; then
|
||||
chartDir=$(echo "$entry" | grep 'chartDir' | awk '{print $2}')
|
||||
namespace=$(echo "$entry" | grep 'namespace' | awk '{print $2}')
|
||||
echo "$cluster: chartDir=$chartDir namespace=$namespace"
|
||||
else
|
||||
echo "$cluster: ABSENT"
|
||||
fi
|
||||
done
|
||||
```
|
||||
Reference in New Issue
Block a user