Runbook: Values Drift

Symptom → diagnosis → remediation for inconsistent appSpec entries across cluster values files — same tool, different chartDir, namespace, or field values across clusters.

Layer: 2 (Advisory). Agents diagnose and report; humans decide whether to remediate.

Per AI Blitz Plan §5.2.

Symptoms

A tool behaves differently on two clusters that should be equivalent
A chart version upgrade was applied to some clusters but not others
grep across values files shows the same tool with different chartDir or namespace
After a fleet-wide rollout, one cluster was accidentally skipped

Detection

Find clusters where a tool exists

grep -rl 'name: <tool>' values/prd/

Find the chartDir each cluster uses for a tool

grep -A 4 'name: <tool>' values/prd/*.yaml | grep 'chartDir'

Compare a specific tool across all prd clusters

# Show full appSpec entry for the tool on every cluster that has it:
for f in values/prd/*.yaml; do
  entry=$(grep -A 5 "^\s*- name: <tool>$" "$f" 2>/dev/null)
  if [ -n "$entry" ]; then
    echo "=== $f ==="
    echo "$entry"
  fi
done

Find clusters missing a tool that should be present

# List all prd values files:
all=$(ls values/prd/)
# List files that have the tool:
have=$(grep -rl 'name: <tool>' values/prd/ | xargs -I{} basename {})
# Diff:
diff <(echo "$all") <(echo "$have")

Interpreting drift

Not all drift is wrong. Some clusters legitimately have different configurations:

Type of drift	Usually OK?	Action
Different `chartDir` (different version)	Only if intentional	Align versions unless the divergence is intentional (e.g., staged rollout in progress)
Different `namespace`	Rarely OK	Investigate — namespace change is destructive
Tool absent from a cluster	Depends	Confirm whether the cluster should have the tool
Different `valuesDir`	Sometimes OK	Clusters can have cluster-specific overrides; confirm the override exists
Different `nameOverride`	Rarely OK	Should only exist for collision/length reasons

Remediation

Align chart versions across clusters

If a tool is on different versions and should be uniform, follow upgrade-chart-version.md for the lagging clusters.

Add a missing tool to a cluster

Follow add-tool-to-cluster.md.

Document intentional drift

If the divergence is intentional (e.g., Cluster A has a special valuesDir for a regional override), add a comment to the values file entry:

  - name: coredns
    namespace: kube-system
    chartDir: coredns
    valuesDir: coredns-special-region   # Intentional: this cluster uses a non-standard DNS config
    additionalValueFiles:
      - ../../helm-templates/coredns/gcp-ase1a-values.yaml

Periodic drift check

Run this to generate a drift report across all prd clusters for a tool:

echo "Tool: <tool>" && echo "---"
for f in values/prd/*.yaml; do
  cluster=$(basename "$f" -values.yaml)
  entry=$(grep -A 5 "^\s*- name: <tool>$" "$f" 2>/dev/null)
  if [ -n "$entry" ]; then
    chartDir=$(echo "$entry" | grep 'chartDir' | awk '{print $2}')
    namespace=$(echo "$entry" | grep 'namespace' | awk '{print $2}')
    echo "$cluster: chartDir=$chartDir namespace=$namespace"
  else
    echo "$cluster: ABSENT"
  fi
done

3.4 KiB Raw Blame History