devops-infra-helm-charts-gcp/docs/platform/runbooks/ingress-down.md

Runbook — Ingress (Contour) is down on a cluster

Type: Decision tree. Entry symptom: services on a cluster are returning 5xx, not reachable, or DNS doesn't resolve to working endpoints. Layer: mostly Layer 2 (advisory — recommend kubectl actions). Layer 1 only when the fix is a values diff in this repo.

Contour is the ingress for almost every BU cluster, and most clusters run multiple Contour releases — contour-external, contour-external-1, contour-internal-0, contour-internal-1, contour-internal-intra-{0,1}. Each maps to a different node pool / dedicated taint or compute class. The matrix is in contour-nodeselector-tolerations-summary.md. Read it before editing any Contour values.

---

Entry — gather context

CLUSTER=<cluster>
CTX=<kubectl-context>

# Which Contour instances does this cluster run?
ls helm-overrides/$CLUSTER | grep '^contour'

# Pod state for each Contour
for c in $(ls helm-overrides/$CLUSTER | grep '^contour'); do
  echo "=== $c ==="
  kubectl --context=$CTX get pods -n projectcontour -l app.kubernetes.io/instance=$c -o wide 2>/dev/null \
    || kubectl --context=$CTX get pods --all-namespaces -l app.kubernetes.io/instance=$c -o wide
done

---

Decision tree

START
  │
  └── Which Contour instance is affected?
        │
        ├── External (`contour-external*`)        → §A — North-south traffic
        ├── Internal (`contour-internal-*`)       → §B — East-west traffic
        └── Both / unsure                         → §C — Cluster-wide (worst case)

For each branch:

  └── What's the failure shape?
        │
        ├── Pods Pending                        → §1 — Scheduling failure
        ├── Pods CrashLooping                   → §2 — Contour boot failure
        ├── Pods Running but no endpoints       → §3 — Service / load-balancer unhealthy
        ├── HTTPS responses are 5xx             → §4 — Backend / cert / config error
        └── DNS doesn't resolve                 → §5 — external-dns / DNS plumbing

---

§1 — Contour pods Pending

This is almost always a scheduling-key mismatch. The single most common Contour incident.

kubectl --context=$CTX describe pod <contour-pod>

Read the Events: section. Common causes:

Reason	Fix in
0/N nodes available: 1 node(s) had untolerated taint <key>=<value>	Tolerations in helm-overrides/<cluster>/<contour-instance>/custom-values.yaml. Cross-reference contour-nodeselector-tolerations-summary.md. Layer 1.
0/N nodes available: 1 node(s) didn't match Pod's node affinity/selector	nodeSelector in the values. Same fix. Layer 1.
0/N nodes available: 1 node(s) had no available compute class (GKE Autopilot)	Either the ComputeClass resource is missing, or nodeSelector: cloud.google.com/compute-class: <X> references a class that doesn't exist. Check helm-overrides/<cluster>/<contour-instance>/computeclass/*-cc.yaml. Layer 1.
0/N nodes available: insufficient cpu/memory	Node-pool autoscaler is at max. Layer 2 — escalate to cluster owner / platform team.

The fix is always to bring the values' nodeSelector / tolerations into agreement with the cluster's actual node-pool topology. Never guess — read the matrix and the cluster's other apps.

---

§2 — Contour pods CrashLooping

kubectl --context=$CTX logs <contour-pod> -c contour --previous
kubectl --context=$CTX logs <contour-pod> -c envoy --previous

Pattern	Cause	Fix in
failed to load TLS certificates	Missing Secret, expired cert, wrong key	cert-manager / external-secrets. Layer 2.
error parsing config: invalid HTTPProxy	A user HTTPProxy is malformed and Contour refuses to load	The user's namespace. Layer 2 — recommend kubectl get httpproxy -A to find the offender, then fix in the consuming team's repo.
bind: address already in use	Two Contour pods on the same node fighting for the host port	Pod anti-affinity in values. Layer 1.
error: ratelimit service ... unavailable	contour-rate-limit sidecar / external service is down	Operator action. Layer 2.

---

§3 — Contour Running but no endpoints / LB unhealthy

The Contour pods are healthy, but downstream LB / DNS / Service routing is broken.

kubectl --context=$CTX get svc -n projectcontour
kubectl --context=$CTX get endpoints -n projectcontour
kubectl --context=$CTX describe svc <contour-svc> -n projectcontour

Sub-check	Action
Service of type LoadBalancer has no EXTERNAL-IP	GCP LB provisioning failed. Check the Service annotations match the cluster's LB pattern. Layer 2 — escalate.
Endpoints empty	The Contour pods aren't matching the Service selector. Often a label drift between values and the chart's defaults. Layer 1 — fix selectors.
Service annotations mention cloud.google.com/load-balancer-type: Internal but external traffic is expected	Wrong Contour instance / Service annotation. Layer 1.
Health checks failing on the GCP LB	Backend pods aren't ready. See §4.

---

§4 — HTTPS responses are 5xx

Don't curl the production endpoint yourself — that violates SANCTITY_RULES R3. Instead, recommend the user check from a controlled vantage:

# From inside the cluster
kubectl --context=$CTX run -it --rm curl-test --image=curlimages/curl --restart=Never -- \
  curl -v https://<service>.<ns>.svc.cluster.local

# Contour access logs
kubectl --context=$CTX logs <envoy-pod> -c envoy | tail -50

Pattern	Cause	Fix in
503 no_healthy_upstream	Backend pods all unhealthy	App's pod readiness — see pod-pending-scheduling.md. Layer 2.
502 upstream connect error	Backend connection refused / TLS mismatch	App config / mTLS. Layer 2.
404 route not found	No matching HTTPProxy/Ingress	The user's HTTPProxy is missing or has the wrong host/path. Layer 2.
500 from the app	Application error	App-team playbook. Out of scope for this runbook.

---

§5 — DNS doesn't resolve

Sub-check	Action
Is external-dns running on this cluster?	kubectl --context=$CTX get pods -n external-dns.
Are there Service resources with external-dns.alpha.kubernetes.io/hostname annotations?	kubectl --context=$CTX get svc -A -o json | jq '.items[] | select(.metadata.annotations."external-dns.alpha.kubernetes.io/hostname")'.
Did external-dns reconcile recently?	kubectl --context=$CTX logs deploy/external-dns -n external-dns | tail -30.
Is the Cloud DNS zone wired up?	Layer 2 — out of scope; escalate to platform team.

---

§C — Cluster-wide ingress outage

If both internal and external Contour are degraded simultaneously:

1. Stop. Don't iterate values fixes. This is incident-grade.
2. Layer 2 — escalate to platform team immediately.
3. Check whether a recent merge in this repo or the sister repo correlates: git log --since='2 hours ago' -- helm-overrides/$CLUSTER/contour* and the same in devops-infra-argo-config.
4. If a recent merge is implicated: revert it, click Sync to roll back to the previous state.
5. If no recent merge: cluster-level issue (node pool, network policy, GCP LB) — out of scope for this repo.

---

When this repo is the right place to fix

A Contour outage traces back to devops-infra-helm-charts only when:

1. nodeSelector / tolerations / computeClass were copied from the wrong cluster.
2. A chart bump introduced an immutable-selector change or removed a values key.
3. A blue-green migration was cutover prematurely (Application points at <contour>-vX.Y.Z but values are still on the old shape).
4. fullnameOverride was changed (very rare, but catastrophic).

For 80%+ of ingress incidents, the fix is outside this repo (cluster issues, app-side issues, GCP LB, DNS).

---

Escalation matrix

Symptom	First responder	Escalate to
§1 (pending — scheduling)	Yourself with the values fix	Cluster owner if node pool full
§2 (CrashLoop — TLS)	cert-manager team	Security if cert source unknown
§2 (CrashLoop — invalid HTTPProxy)	Consuming team	Platform if Contour itself is broken
§3 (no endpoints / LB)	Cluster owner	Platform team
§4 (5xx)	App team	Platform team if Contour-side
§5 (DNS)	Platform team	—
§C (cluster-wide outage)	Platform team — pager	—

---

Done conditions

• kubectl get pods -n projectcontour shows N/N Ready for the affected instances.
• Synthetic probes from the app team return expected status codes.
• If the root cause was in this repo, the fix is on main and synced.

---

Related

• Reference: contour-nodeselector-tolerations-summary.md.
• Runbook: pod-pending-scheduling.md.
• Runbook: argocd-sync-failure.md.
• ADR: ADR-A3-per-cluster-scheduling.md.