134 lines
4.3 KiB
Markdown
134 lines
4.3 KiB
Markdown
# Runbook: Helm Render Failure
|
|
|
|
> Symptom → diagnosis → remediation for ArgoCD Applications stuck in `ComparisonError` or `ErrParsingAppProject` state due to Helm template rendering failures.
|
|
>
|
|
> **Layer:** 2 (Advisory). Agents diagnose and suggest fixes; humans apply.
|
|
>
|
|
> Per AI Blitz Plan §5.2. See also: [argocd-sync-failure.md](argocd-sync-failure.md).
|
|
|
|
---
|
|
|
|
## Symptoms
|
|
|
|
- ArgoCD Application shows `ComparisonError: failed to generate manifest`
|
|
- `helm template` run locally produces YAML errors
|
|
- Application stuck in `Unknown` or `Error` health state with no sync progress
|
|
- Pre-commit hook `cac validate` fails on a PR with YAML parsing errors
|
|
|
|
---
|
|
|
|
## Decision tree
|
|
|
|
```text
|
|
Application stuck in ComparisonError?
|
|
│
|
|
├── Run: argocd app get <appName> --show-operation
|
|
│ (or: argocd app logs <appName>)
|
|
│
|
|
├── Error contains "values file not found"?
|
|
│ └── → [A] Missing values file
|
|
│
|
|
├── Error contains "chart not found" or "no chart found"?
|
|
│ └── → [B] Missing chart directory
|
|
│
|
|
├── Error contains "unmarshal" or "cannot unmarshal" or "yaml: line"?
|
|
│ └── → [C] YAML syntax error in values or chart
|
|
│
|
|
├── Error contains "template: ... nil pointer" or "function not defined"?
|
|
│ └── → [D] Chart template bug
|
|
│
|
|
└── Error contains "AppProject not found" or "unauthorized"?
|
|
└── → See argocd-sync-failure.md
|
|
```
|
|
|
|
---
|
|
|
|
## [A] Missing values file
|
|
|
|
**Cause:** The `valueFiles` path in the incubator Application YAML points to a values file that doesn't exist.
|
|
|
|
**Diagnosis:**
|
|
```bash
|
|
# Check what valueFiles the incubator references:
|
|
grep 'valueFiles' incubator/<env>/<cluster>.yaml
|
|
|
|
# Verify the values file exists:
|
|
ls values/<env>/incubator-infra-<cluster>-values.yaml
|
|
```
|
|
|
|
**Fix:** Create the missing values file (use [add-new-cluster.md](../procedures/add-new-cluster.md) for a new cluster) or correct the path in the incubator YAML. Open a PR.
|
|
|
|
---
|
|
|
|
## [B] Missing chart directory
|
|
|
|
**Cause:** An `appSpec[].chartDir` value points to a directory that doesn't exist in `devops-infra-helm-charts/helm-templates/`.
|
|
|
|
**Diagnosis:**
|
|
```bash
|
|
# Identify which appSpec entry has the bad chartDir:
|
|
helm template generic-argo-apps-chart/ -f values/<env>/<cluster>-values.yaml 2>&1 | head -30
|
|
|
|
# Verify in helm-charts repo:
|
|
ls /path/to/devops-infra-helm-charts/helm-templates/<chartDir>/
|
|
```
|
|
|
|
**Fix:** Either:
|
|
1. Add the chart to `devops-infra-helm-charts` (work in the sister repo).
|
|
2. Correct the `chartDir` in the values file to an existing directory.
|
|
|
|
Open a PR with the fix.
|
|
|
|
---
|
|
|
|
## [C] YAML syntax error
|
|
|
|
**Cause:** Invalid YAML in a values file (e.g., bad indentation, missing quotes, stray character).
|
|
|
|
**Diagnosis:**
|
|
```bash
|
|
yamllint values/<env>/<cluster>-values.yaml
|
|
|
|
# Or for more context:
|
|
python3 -c "import yaml; yaml.safe_load(open('values/<env>/<cluster>-values.yaml'))"
|
|
```
|
|
|
|
**Common causes:**
|
|
- Stray tab character (YAML requires spaces)
|
|
- Missing `- ` prefix on an appSpec entry
|
|
- Unquoted special characters in namespace or name fields
|
|
- Trailing whitespace after a value
|
|
|
|
**Fix:** Correct the YAML syntax. Run `yamllint` and `helm template` locally before opening PR. Pre-commit `cac validate` hook should also catch this.
|
|
|
|
---
|
|
|
|
## [D] Chart template bug
|
|
|
|
**Cause:** The `generic-argo-apps-chart/templates/genericTemplate.yaml` has a nil pointer dereference, undefined function, or logic error.
|
|
|
|
**Diagnosis:**
|
|
```bash
|
|
helm template generic-argo-apps-chart/ -f values/<env>/<cluster>-values.yaml 2>&1
|
|
```
|
|
|
|
**Impact:** This breaks ALL Applications rendered by this chart, across ALL clusters. This is a fleet-wide outage risk.
|
|
|
|
**Fix:** This requires a fix to `generic-argo-apps-chart/`. This is a **HIGH RISK** operation — see [AGENT_BOUNDARIES.md](../../global/AGENT_BOUNDARIES.md). Escalate to platform team immediately via [escalation-matrix.md](../../global/escalation-matrix.md).
|
|
|
|
Do not attempt to fix chart templates without explicit platform-team sign-off and test with `helm template` against multiple values files.
|
|
|
|
---
|
|
|
|
## Preventive checks
|
|
|
|
Run before any PR that modifies values files or `generic-argo-apps-chart/`:
|
|
|
|
```bash
|
|
# Test all values files in one env:
|
|
for f in values/prd/*.yaml; do
|
|
helm template generic-argo-apps-chart/ -f "$f" > /dev/null \
|
|
&& echo "OK: $f" || echo "FAIL: $f"
|
|
done
|
|
```
|