gitadmin/devops-infra-helm-charts-gcp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b8575bb8b9a28e3c2f6deef59947edc4a74ff357
devops-infra-helm-charts-gcp/docs/platform/runbooks/vault-unavailable.md
T
Your Name b8575bb8b9 added repo
2026-08-26 03:39:42 +05:30

11 KiB
Raw Blame History

Per AI Blitz Plan §platform.runbooks. Layer: 1. Repo: devops-infra-helm-charts.

Runbook — Vault unavailable / External Secrets failing to render

Type: Decision tree. Entry symptom: Pods are CrashLooping referencing a missing Secret, or kubectl describe externalsecret shows SecretSyncedError, or kubectl get secrets <name> returns NotFound for a name an ExternalSecret should be creating. Layer: mostly Layer 2 (advisory). Layer 1 only when the fix is a values diff in this repo.

Important Layer 3 boundary: Vault HA itself is Layer 3. Vault runs in production-only with no lower environment, so the agent must NOT attempt server-side fixes (unsealing, leader-election toggling, raft config changes, restoring from snapshot). Those are platform-team / security-team operations. The agent's role here is diagnose, narrow root cause, and escalate.

Architecture refresher

workload Pod (mounts Secret <name>)
   ↑ created by
External Secrets Operator (ESO) controller
   │
   └─ reads SecretStore / ClusterSecretStore
        │
        ├─ kind: gcpsm  → GCP Secret Manager
        │     └─ auth via Workload Identity (KSA → GSA binding)
        └─ kind: vault  → Vault HA cluster
              └─ auth via Kubernetes auth (KSA token review)

Most clusters in the fleet use GCP Secret Manager as the primary backend (via Workload Identity), with Vault HA as the secondary for legacy services. Some clusters use Vault as the primary. Confirm before troubleshooting:

ls helm-overrides/<cluster>/external-secrets/
yq e '.spec.provider' helm-overrides/<cluster>/external-secrets/*.yaml 2>/dev/null

Entry — gather context

CLUSTER=<cluster>
CTX=<kubectl-context>
NS=<namespace-of-the-failing-workload>
ES=<external-secret-name>

# ESO controller status
kubectl --context=$CTX -n external-secrets get pods
kubectl --context=$CTX -n external-secrets logs deploy/external-secrets | tail -50

# The failing ExternalSecret
kubectl --context=$CTX -n $NS describe externalsecret $ES
kubectl --context=$CTX -n $NS get externalsecret $ES -o yaml | yq e '.status'

# The SecretStore / ClusterSecretStore it references
STORE=$(kubectl --context=$CTX -n $NS get externalsecret $ES -o jsonpath='{.spec.secretStoreRef.name}')
KIND=$(kubectl --context=$CTX -n $NS get externalsecret $ES -o jsonpath='{.spec.secretStoreRef.kind}')
kubectl --context=$CTX get $KIND $STORE -o yaml | yq e '.status, .spec.provider'

Decision tree

START
  │
  ├── §1 — Is ESO controller running and healthy?
  │     ├── No → §1a — ESO outage
  │     └── Yes →
  │
  ├── §2 — Is the SecretStore / ClusterSecretStore Ready?
  │     ├── No → §2a — Store config / auth broken
  │     └── Yes →
  │
  ├── §3 — Does the ExternalSecret reference a real remote key?
  │     ├── No → §3a — Bad spec.data[].remoteRef.key (Layer 1 likely)
  │     └── Yes →
  │
  ├── §4 — Is the auth path working? (WI binding or Vault K8s auth)
  │     ├── No → §4a — Identity binding (Layer 2/3)
  │     └── Yes →
  │
  └── §5 — Is the upstream backend healthy?
        ├── GCP Secret Manager 5xx → §5a — escalate to platform
        └── Vault unsealed / leader OK? → §5b — Vault outage (Layer 3)

§1 — ESO controller health

kubectl --context=$CTX -n external-secrets get pods
kubectl --context=$CTX -n external-secrets logs deploy/external-secrets --tail=100 | grep -iE 'error|failed|panic'
Symptom Cause Layer Action
Pods CrashLooping Bad chart upgrade or RBAC misconfig 1 Check helm-overrides/<cluster>/external-secrets/custom-values.yaml. Last bump? Revert.
Pods Pending Scheduling — wrong nodeSelector 1 See pod-pending-scheduling.md.
Pods Running, no logs about reconcile Cluster-watch RBAC missing 2 Escalate.
Forbidden errors on CRD list RBAC on the CRDs 1 / 2 Verify chart values' rbac.create: true.

§2 — SecretStore / ClusterSecretStore status

kubectl --context=$CTX get $KIND $STORE -o yaml | yq e '.status'

Look for conditions[].status and conditions[].message.

Status / message Cause Layer Action
Ready: False, ValidationFailed Store spec invalid 1 Fix the SecretStore YAML in helm-overrides/<cluster>/external-secrets/.
Ready: False, InvalidProviderConfig Provider block malformed 1 Validate spec.provider.gcpsm.projectID / spec.provider.vault.server.
Ready: False, AuthFailed (gcpsm) Workload Identity binding broken 2/3 §4 below.
Ready: False, AuthFailed (vault) KSA token review fails 2/3 §4 below.
Ready: True Store OK; problem is elsewhere — Continue to §3.

§3 — ExternalSecret spec validity

kubectl --context=$CTX -n $NS get externalsecret $ES -o yaml | yq e '.spec.data, .spec.dataFrom'

For each remoteRef.key:

  • GCP SM: the key is the secret name in the project. Verify it exists:

    gcloud secrets list --project=<gcp-project> --filter="name:<key>"

    (Read-only — no write to GCP SM from the agent.)

  • Vault: the key is the path under the engine. Cannot directly verify without Vault access; rely on the ESO controller's reconcile error message.

If the controller logs say secret not found in backend → the remoteRef.key is wrong. Layer 1 fix: correct the key in the ExternalSecret YAML.

§4 — Auth path

§4a — GCP Secret Manager (Workload Identity)

# The ServiceAccount the ESO controller (or this ExternalSecret's pod) runs as
kubectl --context=$CTX -n external-secrets get sa external-secrets -o yaml | yq e '.metadata.annotations'

# Should have:
#   iam.gke.io/gcp-service-account: <gsa>@<project>.iam.gserviceaccount.com
Sub-check Layer Action
KSA missing the iam.gke.io/gcp-service-account annotation 1 Add via helm-overrides/<cluster>/external-secrets/custom-values.yaml serviceAccount.annotations.
GSA exists but no IAM binding to KSA 2 Escalate to platform — IAM is out of repo.
GSA lacks roles/secretmanager.secretAccessor on the project 2 Escalate to platform.

§4b — Vault Kubernetes auth

Sub-check Layer Action
SecretStore.spec.provider.vault.auth.kubernetes.role references a Vault role — Read-only; verify against existing working stores on the same cluster.
ESO logs say permission denied from Vault 3 The role/policy on the Vault server is wrong. Cannot fix from this repo. Escalate to security / Vault platform team.
ESO logs say Vault is sealed 3 Vault HA outage. Do not attempt to unseal. Escalate immediately.

§5 — Backend health

§5a — GCP Secret Manager

GCP SM is a managed service. 5xx from it is rare and is a GCP-side incident. Action: escalate to platform; check GCP status dashboard. No fix in this repo.

§5b — Vault HA

Vault HA on Meesho's fleet runs in production-only, no lower env. It is Layer 3 — agent does not write to or operate Vault. Diagnostic read of pod state is OK; mutation is not.

# Diagnostic only
kubectl --context=$CTX -n vault get pods
kubectl --context=$CTX -n vault logs <vault-pod> | tail -50
# Look for: "core: Vault is sealed", "leader election", "raft", panic stacks
Observed Action
Some Vault pods sealed (HA quorum still up) Escalate to security team. Do not unseal.
All Vault pods sealed (full outage) Page security team. ESO will surface stale data only as long as the in-memory cache holds.
Leader-election thrashing Escalate; could be a network/raft issue.
Vault pod Pending Scheduling — see pod-pending-scheduling.md. Even here, restart of a Vault pod requires a security-team-led unseal afterwards.

Mitigations while Vault is down

If a workload's ExternalSecret is failing because Vault is unavailable, options are limited:

  1. Wait — ESO caches the last-rendered Secret value. Pods that already mounted continue. New pod scheduling fails until Vault returns.
  2. Switch the ExternalSecret to GCP SM if the same secret exists there (most do, with Vault as legacy). Layer 1 PR — change secretStoreRef.name to the GCP SM store. Coordinate with security before doing this in an outage.
  3. Hand-create the Secret as a temporary kubectl apply — out of scope for the agent. This is incident response by a human operator.

The agent should not auto-cut option 2 without explicit human approval — switching the source of truth for a secret has security implications.

Escalation matrix

Symptom First responder Escalate to
§1 (ESO down) DevOps on-call Platform team
§2 (Store config) DevOps on-call Layer 1 PR + reviewer
§3 (Bad remoteRef) DevOps on-call Layer 1 PR + reviewer
§4a (WI binding) Platform team —
§4b (Vault role/policy) Security team —
§5a (GCP SM 5xx) Platform team GCP support
§5b (Vault outage) Security team — page —

Done conditions

  • ExternalSecret status.conditions[type=Ready].status == True.
  • The downstream Secret exists with expected keys.
  • Workload pods consuming the Secret are Running.
  • ESO logs are quiet for at least 5 minutes after the fix.

What this repo can and cannot fix

Fix kind Layer This repo?
ExternalSecret / SecretStore YAML edits 1 Yes — values PR.
ESO chart values (sizing, RBAC, metrics) 1 Yes.
Workload-Identity KSA annotation 1 Yes (in chart values' serviceAccount.annotations).
GSA IAM bindings on GCP 2 No — platform team.
Vault server-side role/policy 3 Never — security team.
Vault unseal / raft / leader 3 Never — security team.

Related

Reference in New Issue View Git Blame Copy Permalink