gitadmin/devops-infra-helm-charts-gcp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added repo

Browse Source
This commit is contained in:
Your Name
2026-08-26 03:39:42 +05:30
parent 45c25a95af
commit b8575bb8b9
6889 changed files with 1217125 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/platform/runbooks/vault-unavailable.md
+251
View File
@@ -0,0 +1,251 @@
> Per AI Blitz Plan §platform.runbooks. Layer: 1. Repo: devops-infra-helm-charts.
# Runbook — Vault unavailable / External Secrets failing to render
> **Type:** Decision tree.
> **Entry symptom:** Pods are CrashLooping referencing a missing `Secret`, or `kubectl describe externalsecret` shows `SecretSyncedError`, or `kubectl get secrets <name>` returns NotFound for a name an `ExternalSecret` should be creating.
> **Layer:** mostly Layer 2 (advisory). Layer 1 only when the fix is a values diff in this repo.
>
> **Important Layer 3 boundary:** **Vault HA itself is Layer 3.** Vault runs in production-only with no lower environment, so the agent must NOT attempt server-side fixes (unsealing, leader-election toggling, raft config changes, restoring from snapshot). Those are platform-team / security-team operations. The agent's role here is *diagnose, narrow root cause, and escalate*.
---
## Architecture refresher
```
workload Pod (mounts Secret <name>)
↑ created by
External Secrets Operator (ESO) controller
│
└─ reads SecretStore / ClusterSecretStore
│
├─ kind: gcpsm → GCP Secret Manager
│ └─ auth via Workload Identity (KSA → GSA binding)
└─ kind: vault → Vault HA cluster
└─ auth via Kubernetes auth (KSA token review)
```
Most clusters in the fleet use **GCP Secret Manager** as the primary backend (via Workload Identity), with Vault HA as the secondary for legacy services. Some clusters use Vault as the primary. Confirm before troubleshooting:
```bash
ls helm-overrides/<cluster>/external-secrets/
yq e '.spec.provider' helm-overrides/<cluster>/external-secrets/*.yaml 2>/dev/null
```
---
## Entry — gather context
```bash
CLUSTER=<cluster>
CTX=<kubectl-context>
NS=<namespace-of-the-failing-workload>
ES=<external-secret-name>
# ESO controller status
kubectl --context=$CTX -n external-secrets get pods
kubectl --context=$CTX -n external-secrets logs deploy/external-secrets | tail -50
# The failing ExternalSecret
kubectl --context=$CTX -n $NS describe externalsecret $ES
kubectl --context=$CTX -n $NS get externalsecret $ES -o yaml | yq e '.status'
# The SecretStore / ClusterSecretStore it references
STORE=$(kubectl --context=$CTX -n $NS get externalsecret $ES -o jsonpath='{.spec.secretStoreRef.name}')
KIND=$(kubectl --context=$CTX -n $NS get externalsecret $ES -o jsonpath='{.spec.secretStoreRef.kind}')
kubectl --context=$CTX get $KIND $STORE -o yaml | yq e '.status, .spec.provider'
```
---
## Decision tree
```text
START
│
├── §1 — Is ESO controller running and healthy?
│ ├── No → §1a — ESO outage
│ └── Yes →
│
├── §2 — Is the SecretStore / ClusterSecretStore Ready?
│ ├── No → §2a — Store config / auth broken
│ └── Yes →
│
├── §3 — Does the ExternalSecret reference a real remote key?
│ ├── No → §3a — Bad spec.data[].remoteRef.key (Layer 1 likely)
│ └── Yes →
│
├── §4 — Is the auth path working? (WI binding or Vault K8s auth)
│ ├── No → §4a — Identity binding (Layer 2/3)
│ └── Yes →
│
└── §5 — Is the upstream backend healthy?
├── GCP Secret Manager 5xx → §5a — escalate to platform
└── Vault unsealed / leader OK? → §5b — Vault outage (Layer 3)
```
---
## §1 — ESO controller health
```bash
kubectl --context=$CTX -n external-secrets get pods
kubectl --context=$CTX -n external-secrets logs deploy/external-secrets --tail=100 | grep -iE 'error|failed|panic'
```
| Symptom | Cause | Layer | Action |
|---------|-------|-------|--------|
| Pods CrashLooping | Bad chart upgrade or RBAC misconfig | 1 | Check `helm-overrides/<cluster>/external-secrets/custom-values.yaml`. Last bump? Revert. |
| Pods Pending | Scheduling — wrong nodeSelector | 1 | See [pod-pending-scheduling.md](pod-pending-scheduling.md). |
| Pods Running, no logs about reconcile | Cluster-watch RBAC missing | 2 | Escalate. |
| `Forbidden` errors on CRD list | RBAC on the CRDs | 1 / 2 | Verify chart values' `rbac.create: true`. |
---
## §2 — SecretStore / ClusterSecretStore status
```bash
kubectl --context=$CTX get $KIND $STORE -o yaml | yq e '.status'
```
Look for `conditions[].status` and `conditions[].message`.
| Status / message | Cause | Layer | Action |
|------------------|-------|-------|--------|
| `Ready: False, ValidationFailed` | Store spec invalid | 1 | Fix the `SecretStore` YAML in `helm-overrides/<cluster>/external-secrets/`. |
| `Ready: False, InvalidProviderConfig` | Provider block malformed | 1 | Validate `spec.provider.gcpsm.projectID` / `spec.provider.vault.server`. |
| `Ready: False, AuthFailed` (gcpsm) | Workload Identity binding broken | 2/3 | §4 below. |
| `Ready: False, AuthFailed` (vault) | KSA token review fails | 2/3 | §4 below. |
| `Ready: True` | Store OK; problem is elsewhere | — | Continue to §3. |
---
## §3 — ExternalSecret spec validity
```bash
kubectl --context=$CTX -n $NS get externalsecret $ES -o yaml | yq e '.spec.data, .spec.dataFrom'
```
For each `remoteRef.key`:
- **GCP SM:** the key is the secret name in the project. Verify it exists:
```bash
gcloud secrets list --project=<gcp-project> --filter="name:<key>"
```
(Read-only — no write to GCP SM from the agent.)
- **Vault:** the key is the path under the engine. Cannot directly verify without Vault access; rely on the ESO controller's reconcile error message.
If the controller logs say `secret not found in backend` → the `remoteRef.key` is wrong. **Layer 1 fix:** correct the key in the `ExternalSecret` YAML.
---
## §4 — Auth path
### §4a — GCP Secret Manager (Workload Identity)
```bash
# The ServiceAccount the ESO controller (or this ExternalSecret's pod) runs as
kubectl --context=$CTX -n external-secrets get sa external-secrets -o yaml | yq e '.metadata.annotations'
# Should have:
# iam.gke.io/gcp-service-account: <gsa>@<project>.iam.gserviceaccount.com
```
| Sub-check | Layer | Action |
|-----------|-------|--------|
| KSA missing the `iam.gke.io/gcp-service-account` annotation | 1 | Add via `helm-overrides/<cluster>/external-secrets/custom-values.yaml` `serviceAccount.annotations`. |
| GSA exists but no IAM binding to KSA | 2 | Escalate to platform — IAM is out of repo. |
| GSA lacks `roles/secretmanager.secretAccessor` on the project | 2 | Escalate to platform. |
### §4b — Vault Kubernetes auth
| Sub-check | Layer | Action |
|-----------|-------|--------|
| `SecretStore.spec.provider.vault.auth.kubernetes.role` references a Vault role | — | Read-only; verify against existing working stores on the same cluster. |
| ESO logs say `permission denied` from Vault | 3 | The role/policy on the Vault server is wrong. **Cannot fix from this repo.** Escalate to security / Vault platform team. |
| ESO logs say `Vault is sealed` | 3 | **Vault HA outage. Do not attempt to unseal.** Escalate immediately. |
---
## §5 — Backend health
### §5a — GCP Secret Manager
GCP SM is a managed service. 5xx from it is rare and is a GCP-side incident. Action: escalate to platform; check GCP status dashboard. **No fix in this repo.**
### §5b — Vault HA
Vault HA on Meesho's fleet runs in production-only, no lower env. It is **Layer 3** — agent does not write to or operate Vault. Diagnostic *read* of pod state is OK; mutation is not.
```bash
# Diagnostic only
kubectl --context=$CTX -n vault get pods
kubectl --context=$CTX -n vault logs <vault-pod> | tail -50
# Look for: "core: Vault is sealed", "leader election", "raft", panic stacks
```
| Observed | Action |
|----------|--------|
| Some Vault pods sealed (HA quorum still up) | Escalate to security team. **Do not unseal.** |
| All Vault pods sealed (full outage) | Page security team. ESO will surface stale data only as long as the in-memory cache holds. |
| Leader-election thrashing | Escalate; could be a network/raft issue. |
| Vault pod Pending | Scheduling — see [pod-pending-scheduling.md](pod-pending-scheduling.md). Even here, restart of a Vault pod requires a security-team-led unseal afterwards. |
---
## Mitigations while Vault is down
If a workload's `ExternalSecret` is failing because Vault is unavailable, options are limited:
1. **Wait** — ESO caches the last-rendered Secret value. Pods that already mounted continue. New pod scheduling fails until Vault returns.
2. **Switch the `ExternalSecret` to GCP SM** if the same secret exists there (most do, with Vault as legacy). Layer 1 PR — change `secretStoreRef.name` to the GCP SM store. **Coordinate with security** before doing this in an outage.
3. **Hand-create the Secret as a temporary `kubectl apply`** — out of scope for the agent. This is incident response by a human operator.
The agent should **not** auto-cut option 2 without explicit human approval — switching the source of truth for a secret has security implications.
---
## Escalation matrix
| Symptom | First responder | Escalate to |
|---------|-----------------|-------------|
| §1 (ESO down) | DevOps on-call | Platform team |
| §2 (Store config) | DevOps on-call | Layer 1 PR + reviewer |
| §3 (Bad remoteRef) | DevOps on-call | Layer 1 PR + reviewer |
| §4a (WI binding) | Platform team | — |
| §4b (Vault role/policy) | Security team | — |
| §5a (GCP SM 5xx) | Platform team | GCP support |
| §5b (Vault outage) | **Security team — page** | — |
---
## Done conditions
- `ExternalSecret` `status.conditions[type=Ready].status == True`.
- The downstream `Secret` exists with expected keys.
- Workload pods consuming the Secret are Running.
- ESO logs are quiet for at least 5 minutes after the fix.
---
## What this repo can and cannot fix
| Fix kind | Layer | This repo? |
|----------|-------|-----------|
| `ExternalSecret` / `SecretStore` YAML edits | 1 | Yes — values PR. |
| ESO chart values (sizing, RBAC, metrics) | 1 | Yes. |
| Workload-Identity KSA annotation | 1 | Yes (in chart values' `serviceAccount.annotations`). |
| GSA IAM bindings on GCP | 2 | No — platform team. |
| Vault server-side role/policy | 3 | **Never** — security team. |
| Vault unseal / raft / leader | 3 | **Never** — security team. |
---
## Related
- Schema: [../schemas/raw-manifest-sidecar-schema.md §ExternalSecret](../schemas/raw-manifest-sidecar-schema.md).
- Runbook: [pod-pending-scheduling.md](pod-pending-scheduling.md) — if any of the above pods are Pending.
- Runbook: [argocd-sync-failure.md](argocd-sync-failure.md) — if the `ExternalSecret` itself didn't sync.
- Boundaries: [../../global/AGENT_BOUNDARIES.md](../../global/AGENT_BOUNDARIES.md), [../../global/SANCTITY_RULES.md](../../global/SANCTITY_RULES.md).
- Escalation: [../../global/escalation-matrix.md](../../global/escalation-matrix.md).