devops-infra-argo-config-gcp/docs/platform/runbooks/render-failure.md

Runbook: Helm Render Failure

Symptom → diagnosis → remediation for ArgoCD Applications stuck in ComparisonError or ErrParsingAppProject state due to Helm template rendering failures.

Layer: 2 (Advisory). Agents diagnose and suggest fixes; humans apply.

Per AI Blitz Plan §5.2. See also: argocd-sync-failure.md.

---

Symptoms

• ArgoCD Application shows ComparisonError: failed to generate manifest
• helm template run locally produces YAML errors
• Application stuck in Unknown or Error health state with no sync progress
• Pre-commit hook cac validate fails on a PR with YAML parsing errors

---

Decision tree

Application stuck in ComparisonError?
    │
    ├── Run: argocd app get <appName> --show-operation
    │         (or: argocd app logs <appName>)
    │
    ├── Error contains "values file not found"?
    │   └── → [A] Missing values file
    │
    ├── Error contains "chart not found" or "no chart found"?
    │   └── → [B] Missing chart directory
    │
    ├── Error contains "unmarshal" or "cannot unmarshal" or "yaml: line"?
    │   └── → [C] YAML syntax error in values or chart
    │
    ├── Error contains "template: ... nil pointer" or "function not defined"?
    │   └── → [D] Chart template bug
    │
    └── Error contains "AppProject not found" or "unauthorized"?
        └── → See argocd-sync-failure.md

---

[A] Missing values file

Cause: The valueFiles path in the incubator Application YAML points to a values file that doesn't exist.

Diagnosis:

# Check what valueFiles the incubator references:
grep 'valueFiles' incubator/<env>/<cluster>.yaml

# Verify the values file exists:
ls values/<env>/incubator-infra-<cluster>-values.yaml

Fix: Create the missing values file (use add-new-cluster.md for a new cluster) or correct the path in the incubator YAML. Open a PR.

---

[B] Missing chart directory

Cause: An appSpec[].chartDir value points to a directory that doesn't exist in devops-infra-helm-charts/helm-templates/.

Diagnosis:

# Identify which appSpec entry has the bad chartDir:
helm template generic-argo-apps-chart/ -f values/<env>/<cluster>-values.yaml 2>&1 | head -30

# Verify in helm-charts repo:
ls /path/to/devops-infra-helm-charts/helm-templates/<chartDir>/

Fix: Either:

1. Add the chart to devops-infra-helm-charts (work in the sister repo).
2. Correct the chartDir in the values file to an existing directory.

Open a PR with the fix.

---

[C] YAML syntax error

Cause: Invalid YAML in a values file (e.g., bad indentation, missing quotes, stray character).

Diagnosis:

yamllint values/<env>/<cluster>-values.yaml

# Or for more context:
python3 -c "import yaml; yaml.safe_load(open('values/<env>/<cluster>-values.yaml'))"

Common causes:

• Stray tab character (YAML requires spaces)
• Missing - prefix on an appSpec entry
• Unquoted special characters in namespace or name fields
• Trailing whitespace after a value

Fix: Correct the YAML syntax. Run yamllint and helm template locally before opening PR. Pre-commit cac validate hook should also catch this.

---

[D] Chart template bug

Cause: The generic-argo-apps-chart/templates/genericTemplate.yaml has a nil pointer dereference, undefined function, or logic error.

Diagnosis:

helm template generic-argo-apps-chart/ -f values/<env>/<cluster>-values.yaml 2>&1

Impact: This breaks ALL Applications rendered by this chart, across ALL clusters. This is a fleet-wide outage risk.

Fix: This requires a fix to generic-argo-apps-chart/. This is a HIGH RISK operation — see AGENT_BOUNDARIES.md. Escalate to platform team immediately via escalation-matrix.md.

Do not attempt to fix chart templates without explicit platform-team sign-off and test with helm template against multiple values files.

---

Preventive checks

Run before any PR that modifies values files or generic-argo-apps-chart/:

# Test all values files in one env:
for f in values/prd/*.yaml; do
  helm template generic-argo-apps-chart/ -f "$f" > /dev/null \
    && echo "OK: $f" || echo "FAIL: $f"
done