added repo

2026-08-26 04:03:34 +05:30
parent 2389ec1fd6
commit 1055e1394f
150 changed files with 12395 additions and 0 deletions
@@ -0,0 +1,116 @@
+# Runbook: Values Drift
+
+> Symptom → diagnosis → remediation for inconsistent `appSpec` entries across cluster values files — same tool, different `chartDir`, `namespace`, or field values across clusters.
+>
+> **Layer:** 2 (Advisory). Agents diagnose and report; humans decide whether to remediate.
+>
+> Per AI Blitz Plan §5.2.
+
+---
+
+## Symptoms
+
+- A tool behaves differently on two clusters that should be equivalent
+- A chart version upgrade was applied to some clusters but not others
+- `grep` across values files shows the same tool with different `chartDir` or `namespace`
+- After a fleet-wide rollout, one cluster was accidentally skipped
+
+---
+
+## Detection
+
+### Find clusters where a tool exists
+
+```bash
+grep -rl 'name: <tool>' values/prd/
+```
+
+### Find the chartDir each cluster uses for a tool
+
+```bash
+grep -A 4 'name: <tool>' values/prd/*.yaml | grep 'chartDir'
+```
+
+### Compare a specific tool across all prd clusters
+
+```bash
+# Show full appSpec entry for the tool on every cluster that has it:
+for f in values/prd/*.yaml; do
+  entry=$(grep -A 5 "^\s*- name: <tool>$" "$f" 2>/dev/null)
+  if [ -n "$entry" ]; then
+    echo "=== $f ==="
+    echo "$entry"
+  fi
+done
+```
+
+### Find clusters missing a tool that should be present
+
+```bash
+# List all prd values files:
+all=$(ls values/prd/)
+# List files that have the tool:
+have=$(grep -rl 'name: <tool>' values/prd/ | xargs -I{} basename {})
+# Diff:
+diff <(echo "$all") <(echo "$have")
+```
+
+---
+
+## Interpreting drift
+
+Not all drift is wrong. Some clusters legitimately have different configurations:
+
+| Type of drift | Usually OK? | Action |
+| ------------- | ----------- | ------ |
+| Different `chartDir` (different version) | Only if intentional | Align versions unless the divergence is intentional (e.g., staged rollout in progress) |
+| Different `namespace` | Rarely OK | Investigate — namespace change is destructive |
+| Tool absent from a cluster | Depends | Confirm whether the cluster should have the tool |
+| Different `valuesDir` | Sometimes OK | Clusters can have cluster-specific overrides; confirm the override exists |
+| Different `nameOverride` | Rarely OK | Should only exist for collision/length reasons |
+
+---
+
+## Remediation
+
+### Align chart versions across clusters
+
+If a tool is on different versions and should be uniform, follow [upgrade-chart-version.md](../procedures/upgrade-chart-version.md) for the lagging clusters.
+
+### Add a missing tool to a cluster
+
+Follow [add-tool-to-cluster.md](../procedures/add-tool-to-cluster.md).
+
+### Document intentional drift
+
+If the divergence is intentional (e.g., Cluster A has a special `valuesDir` for a regional override), add a comment to the values file entry:
+
+```yaml
+  - name: coredns
+    namespace: kube-system
+    chartDir: coredns
+    valuesDir: coredns-special-region   # Intentional: this cluster uses a non-standard DNS config
+    additionalValueFiles:
+      - ../../helm-templates/coredns/gcp-ase1a-values.yaml
+```
+
+---
+
+## Periodic drift check
+
+Run this to generate a drift report across all prd clusters for a tool:
+
+```bash
+echo "Tool: <tool>" && echo "---"
+for f in values/prd/*.yaml; do
+  cluster=$(basename "$f" -values.yaml)
+  entry=$(grep -A 5 "^\s*- name: <tool>$" "$f" 2>/dev/null)
+  if [ -n "$entry" ]; then
+    chartDir=$(echo "$entry" | grep 'chartDir' | awk '{print $2}')
+    namespace=$(echo "$entry" | grep 'namespace' | awk '{print $2}')
+    echo "$cluster: chartDir=$chartDir namespace=$namespace"
+  else
+    echo "$cluster: ABSENT"
+  fi
+done
+```