# Runbook: Values Drift

> Symptom → diagnosis → remediation for inconsistent `appSpec` entries across cluster values files — same tool, different `chartDir`, `namespace`, or field values across clusters.
>
> **Layer:** 2 (Advisory). Agents diagnose and report; humans decide whether to remediate.
>
> Per AI Blitz Plan §5.2.

---

## Symptoms

- A tool behaves differently on two clusters that should be equivalent
- A chart version upgrade was applied to some clusters but not others
- `grep` across values files shows the same tool with different `chartDir` or `namespace`
- After a fleet-wide rollout, one cluster was accidentally skipped

---

## Detection

### Find clusters where a tool exists

```bash
grep -rl 'name: <tool>' values/prd/
```

### Find the chartDir each cluster uses for a tool

```bash
grep -A 4 'name: <tool>' values/prd/*.yaml | grep 'chartDir'
```

### Compare a specific tool across all prd clusters

```bash
# Show full appSpec entry for the tool on every cluster that has it:
for f in values/prd/*.yaml; do
  entry=$(grep -A 5 "^\s*- name: <tool>$" "$f" 2>/dev/null)
  if [ -n "$entry" ]; then
    echo "=== $f ==="
    echo "$entry"
  fi
done
```

### Find clusters missing a tool that should be present

```bash
# List all prd values files:
all=$(ls values/prd/)
# List files that have the tool:
have=$(grep -rl 'name: <tool>' values/prd/ | xargs -I{} basename {})
# Diff:
diff <(echo "$all") <(echo "$have")
```

---

## Interpreting drift

Not all drift is wrong. Some clusters legitimately have different configurations:

| Type of drift | Usually OK? | Action |
| ------------- | ----------- | ------ |
| Different `chartDir` (different version) | Only if intentional | Align versions unless the divergence is intentional (e.g., staged rollout in progress) |
| Different `namespace` | Rarely OK | Investigate — namespace change is destructive |
| Tool absent from a cluster | Depends | Confirm whether the cluster should have the tool |
| Different `valuesDir` | Sometimes OK | Clusters can have cluster-specific overrides; confirm the override exists |
| Different `nameOverride` | Rarely OK | Should only exist for collision/length reasons |

---

## Remediation

### Align chart versions across clusters

If a tool is on different versions and should be uniform, follow [upgrade-chart-version.md](../procedures/upgrade-chart-version.md) for the lagging clusters.

### Add a missing tool to a cluster

Follow [add-tool-to-cluster.md](../procedures/add-tool-to-cluster.md).

### Document intentional drift

If the divergence is intentional (e.g., Cluster A has a special `valuesDir` for a regional override), add a comment to the values file entry:

```yaml
  - name: coredns
    namespace: kube-system
    chartDir: coredns
    valuesDir: coredns-special-region   # Intentional: this cluster uses a non-standard DNS config
    additionalValueFiles:
      - ../../helm-templates/coredns/gcp-ase1a-values.yaml
```

---

## Periodic drift check

Run this to generate a drift report across all prd clusters for a tool:

```bash
echo "Tool: <tool>" && echo "---"
for f in values/prd/*.yaml; do
  cluster=$(basename "$f" -values.yaml)
  entry=$(grep -A 5 "^\s*- name: <tool>$" "$f" 2>/dev/null)
  if [ -n "$entry" ]; then
    chartDir=$(echo "$entry" | grep 'chartDir' | awk '{print $2}')
    namespace=$(echo "$entry" | grep 'namespace' | awk '{print $2}')
    echo "$cluster: chartDir=$chartDir namespace=$namespace"
  else
    echo "$cluster: ABSENT"
  fi
done
```