117 lines
3.4 KiB
Markdown
117 lines
3.4 KiB
Markdown
# Runbook: Values Drift
|
|
|
|
> Symptom → diagnosis → remediation for inconsistent `appSpec` entries across cluster values files — same tool, different `chartDir`, `namespace`, or field values across clusters.
|
|
>
|
|
> **Layer:** 2 (Advisory). Agents diagnose and report; humans decide whether to remediate.
|
|
>
|
|
> Per AI Blitz Plan §5.2.
|
|
|
|
---
|
|
|
|
## Symptoms
|
|
|
|
- A tool behaves differently on two clusters that should be equivalent
|
|
- A chart version upgrade was applied to some clusters but not others
|
|
- `grep` across values files shows the same tool with different `chartDir` or `namespace`
|
|
- After a fleet-wide rollout, one cluster was accidentally skipped
|
|
|
|
---
|
|
|
|
## Detection
|
|
|
|
### Find clusters where a tool exists
|
|
|
|
```bash
|
|
grep -rl 'name: <tool>' values/prd/
|
|
```
|
|
|
|
### Find the chartDir each cluster uses for a tool
|
|
|
|
```bash
|
|
grep -A 4 'name: <tool>' values/prd/*.yaml | grep 'chartDir'
|
|
```
|
|
|
|
### Compare a specific tool across all prd clusters
|
|
|
|
```bash
|
|
# Show full appSpec entry for the tool on every cluster that has it:
|
|
for f in values/prd/*.yaml; do
|
|
entry=$(grep -A 5 "^\s*- name: <tool>$" "$f" 2>/dev/null)
|
|
if [ -n "$entry" ]; then
|
|
echo "=== $f ==="
|
|
echo "$entry"
|
|
fi
|
|
done
|
|
```
|
|
|
|
### Find clusters missing a tool that should be present
|
|
|
|
```bash
|
|
# List all prd values files:
|
|
all=$(ls values/prd/)
|
|
# List files that have the tool:
|
|
have=$(grep -rl 'name: <tool>' values/prd/ | xargs -I{} basename {})
|
|
# Diff:
|
|
diff <(echo "$all") <(echo "$have")
|
|
```
|
|
|
|
---
|
|
|
|
## Interpreting drift
|
|
|
|
Not all drift is wrong. Some clusters legitimately have different configurations:
|
|
|
|
| Type of drift | Usually OK? | Action |
|
|
| ------------- | ----------- | ------ |
|
|
| Different `chartDir` (different version) | Only if intentional | Align versions unless the divergence is intentional (e.g., staged rollout in progress) |
|
|
| Different `namespace` | Rarely OK | Investigate — namespace change is destructive |
|
|
| Tool absent from a cluster | Depends | Confirm whether the cluster should have the tool |
|
|
| Different `valuesDir` | Sometimes OK | Clusters can have cluster-specific overrides; confirm the override exists |
|
|
| Different `nameOverride` | Rarely OK | Should only exist for collision/length reasons |
|
|
|
|
---
|
|
|
|
## Remediation
|
|
|
|
### Align chart versions across clusters
|
|
|
|
If a tool is on different versions and should be uniform, follow [upgrade-chart-version.md](../procedures/upgrade-chart-version.md) for the lagging clusters.
|
|
|
|
### Add a missing tool to a cluster
|
|
|
|
Follow [add-tool-to-cluster.md](../procedures/add-tool-to-cluster.md).
|
|
|
|
### Document intentional drift
|
|
|
|
If the divergence is intentional (e.g., Cluster A has a special `valuesDir` for a regional override), add a comment to the values file entry:
|
|
|
|
```yaml
|
|
- name: coredns
|
|
namespace: kube-system
|
|
chartDir: coredns
|
|
valuesDir: coredns-special-region # Intentional: this cluster uses a non-standard DNS config
|
|
additionalValueFiles:
|
|
- ../../helm-templates/coredns/gcp-ase1a-values.yaml
|
|
```
|
|
|
|
---
|
|
|
|
## Periodic drift check
|
|
|
|
Run this to generate a drift report across all prd clusters for a tool:
|
|
|
|
```bash
|
|
echo "Tool: <tool>" && echo "---"
|
|
for f in values/prd/*.yaml; do
|
|
cluster=$(basename "$f" -values.yaml)
|
|
entry=$(grep -A 5 "^\s*- name: <tool>$" "$f" 2>/dev/null)
|
|
if [ -n "$entry" ]; then
|
|
chartDir=$(echo "$entry" | grep 'chartDir' | awk '{print $2}')
|
|
namespace=$(echo "$entry" | grep 'namespace' | awk '{print $2}')
|
|
echo "$cluster: chartDir=$chartDir namespace=$namespace"
|
|
else
|
|
echo "$cluster: ABSENT"
|
|
fi
|
|
done
|
|
```
|