added repo
This commit is contained in:
Your Name
@@ -0,0 +1,197 @@
|
||||
# Runbook — Ingress (Contour) is down on a cluster
|
||||
|
||||
> **Type:** Decision tree.
|
||||
> **Entry symptom:** services on a cluster are returning 5xx, not reachable, or DNS doesn't resolve to working endpoints.
|
||||
> **Layer:** mostly Layer 2 (advisory — recommend kubectl actions). Layer 1 only when the fix is a values diff in this repo.
|
||||
|
||||
Contour is the ingress for almost every BU cluster, and most clusters run **multiple Contour releases** — `contour-external`, `contour-external-1`, `contour-internal-0`, `contour-internal-1`, `contour-internal-intra-{0,1}`. Each maps to a different node pool / dedicated taint or compute class. The matrix is in [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md). **Read it before editing any Contour values.**
|
||||
|
||||
---
|
||||
|
||||
## Entry — gather context
|
||||
|
||||
```bash
|
||||
CLUSTER=<cluster>
|
||||
CTX=<kubectl-context>
|
||||
|
||||
# Which Contour instances does this cluster run?
|
||||
ls helm-overrides/$CLUSTER | grep '^contour'
|
||||
|
||||
# Pod state for each Contour
|
||||
for c in $(ls helm-overrides/$CLUSTER | grep '^contour'); do
|
||||
echo "=== $c ==="
|
||||
kubectl --context=$CTX get pods -n projectcontour -l app.kubernetes.io/instance=$c -o wide 2>/dev/null \
|
||||
|| kubectl --context=$CTX get pods --all-namespaces -l app.kubernetes.io/instance=$c -o wide
|
||||
done
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision tree
|
||||
|
||||
```text
|
||||
START
|
||||
│
|
||||
└── Which Contour instance is affected?
|
||||
│
|
||||
├── External (`contour-external*`) → §A — North-south traffic
|
||||
├── Internal (`contour-internal-*`) → §B — East-west traffic
|
||||
└── Both / unsure → §C — Cluster-wide (worst case)
|
||||
```
|
||||
|
||||
For each branch:
|
||||
|
||||
```text
|
||||
└── What's the failure shape?
|
||||
│
|
||||
├── Pods Pending → §1 — Scheduling failure
|
||||
├── Pods CrashLooping → §2 — Contour boot failure
|
||||
├── Pods Running but no endpoints → §3 — Service / load-balancer unhealthy
|
||||
├── HTTPS responses are 5xx → §4 — Backend / cert / config error
|
||||
└── DNS doesn't resolve → §5 — external-dns / DNS plumbing
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## §1 — Contour pods Pending
|
||||
|
||||
This is almost always a scheduling-key mismatch. **The single most common Contour incident.**
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX describe pod <contour-pod>
|
||||
```
|
||||
|
||||
Read the `Events:` section. Common causes:
|
||||
|
||||
| Reason | Fix in |
|
||||
|--------|--------|
|
||||
| `0/N nodes available: 1 node(s) had untolerated taint <key>=<value>` | Tolerations in `helm-overrides/<cluster>/<contour-instance>/custom-values.yaml`. Cross-reference [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md). **Layer 1.** |
|
||||
| `0/N nodes available: 1 node(s) didn't match Pod's node affinity/selector` | `nodeSelector` in the values. Same fix. **Layer 1.** |
|
||||
| `0/N nodes available: 1 node(s) had no available compute class` (GKE Autopilot) | Either the `ComputeClass` resource is missing, or `nodeSelector: cloud.google.com/compute-class: <X>` references a class that doesn't exist. Check `helm-overrides/<cluster>/<contour-instance>/computeclass/*-cc.yaml`. **Layer 1.** |
|
||||
| `0/N nodes available: insufficient cpu/memory` | Node-pool autoscaler is at max. **Layer 2** — escalate to cluster owner / platform team. |
|
||||
|
||||
The fix is always to bring the values' `nodeSelector` / `tolerations` into agreement with the cluster's actual node-pool topology. Never guess — read the matrix and the cluster's other apps.
|
||||
|
||||
---
|
||||
|
||||
## §2 — Contour pods CrashLooping
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX logs <contour-pod> -c contour --previous
|
||||
kubectl --context=$CTX logs <contour-pod> -c envoy --previous
|
||||
```
|
||||
|
||||
| Pattern | Cause | Fix in |
|
||||
|---------|-------|--------|
|
||||
| `failed to load TLS certificates` | Missing `Secret`, expired cert, wrong key | `cert-manager` / `external-secrets`. **Layer 2.** |
|
||||
| `error parsing config: invalid HTTPProxy` | A user `HTTPProxy` is malformed and Contour refuses to load | The user's namespace. **Layer 2** — recommend `kubectl get httpproxy -A` to find the offender, then fix in the consuming team's repo. |
|
||||
| `bind: address already in use` | Two Contour pods on the same node fighting for the host port | Pod anti-affinity in values. **Layer 1.** |
|
||||
| `error: ratelimit service ... unavailable` | `contour-rate-limit` sidecar / external service is down | Operator action. **Layer 2.** |
|
||||
|
||||
---
|
||||
|
||||
## §3 — Contour Running but no endpoints / LB unhealthy
|
||||
|
||||
The Contour pods are healthy, but downstream LB / DNS / Service routing is broken.
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX get svc -n projectcontour
|
||||
kubectl --context=$CTX get endpoints -n projectcontour
|
||||
kubectl --context=$CTX describe svc <contour-svc> -n projectcontour
|
||||
```
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| `Service` of type `LoadBalancer` has no `EXTERNAL-IP` | GCP LB provisioning failed. Check the `Service` annotations match the cluster's LB pattern. **Layer 2** — escalate. |
|
||||
| `Endpoints` empty | The Contour pods aren't matching the `Service` selector. Often a label drift between values and the chart's defaults. **Layer 1** — fix selectors. |
|
||||
| `Service` annotations mention `cloud.google.com/load-balancer-type: Internal` but external traffic is expected | Wrong Contour instance / Service annotation. **Layer 1.** |
|
||||
| `Health checks failing` on the GCP LB | Backend pods aren't ready. See §4. |
|
||||
|
||||
---
|
||||
|
||||
## §4 — HTTPS responses are 5xx
|
||||
|
||||
Don't curl the production endpoint yourself — that violates [SANCTITY_RULES R3](../../global/SANCTITY_RULES.md). Instead, recommend the user check from a controlled vantage:
|
||||
|
||||
```bash
|
||||
# From inside the cluster
|
||||
kubectl --context=$CTX run -it --rm curl-test --image=curlimages/curl --restart=Never -- \
|
||||
curl -v https://<service>.<ns>.svc.cluster.local
|
||||
|
||||
# Contour access logs
|
||||
kubectl --context=$CTX logs <envoy-pod> -c envoy | tail -50
|
||||
```
|
||||
|
||||
| Pattern | Cause | Fix in |
|
||||
|---------|-------|--------|
|
||||
| `503 no_healthy_upstream` | Backend pods all unhealthy | App's pod readiness — see [pod-pending-scheduling.md](pod-pending-scheduling.md). **Layer 2.** |
|
||||
| `502 upstream connect error` | Backend connection refused / TLS mismatch | App config / mTLS. **Layer 2.** |
|
||||
| `404 route not found` | No matching `HTTPProxy`/`Ingress` | The user's `HTTPProxy` is missing or has the wrong host/path. **Layer 2.** |
|
||||
| `500` from the app | Application error | App-team playbook. **Out of scope for this runbook.** |
|
||||
|
||||
---
|
||||
|
||||
## §5 — DNS doesn't resolve
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| Is `external-dns` running on this cluster? | `kubectl --context=$CTX get pods -n external-dns`. |
|
||||
| Are there `Service` resources with `external-dns.alpha.kubernetes.io/hostname` annotations? | `kubectl --context=$CTX get svc -A -o json \| jq '.items[] \| select(.metadata.annotations."external-dns.alpha.kubernetes.io/hostname")'`. |
|
||||
| Did `external-dns` reconcile recently? | `kubectl --context=$CTX logs deploy/external-dns -n external-dns \| tail -30`. |
|
||||
| Is the Cloud DNS zone wired up? | **Layer 2** — out of scope; escalate to platform team. |
|
||||
|
||||
---
|
||||
|
||||
## §C — Cluster-wide ingress outage
|
||||
|
||||
If both internal and external Contour are degraded simultaneously:
|
||||
|
||||
1. **Stop. Don't iterate values fixes.** This is incident-grade.
|
||||
2. **Layer 2 — escalate to platform team immediately.**
|
||||
3. Check whether a recent merge in this repo or the sister repo correlates: `git log --since='2 hours ago' -- helm-overrides/$CLUSTER/contour*` and the same in `devops-infra-argo-config`.
|
||||
4. If a recent merge is implicated: revert it, click Sync to roll back to the previous state.
|
||||
5. If no recent merge: cluster-level issue (node pool, network policy, GCP LB) — out of scope for this repo.
|
||||
|
||||
---
|
||||
|
||||
## When this repo *is* the right place to fix
|
||||
|
||||
A Contour outage traces back to `devops-infra-helm-charts` only when:
|
||||
|
||||
1. **`nodeSelector` / `tolerations` / `computeClass`** were copied from the wrong cluster.
|
||||
2. **A chart bump** introduced an immutable-selector change or removed a values key.
|
||||
3. **A blue-green migration** was cutover prematurely (Application points at `<contour>-vX.Y.Z` but values are still on the old shape).
|
||||
4. **`fullnameOverride`** was changed (very rare, but catastrophic).
|
||||
|
||||
For 80%+ of ingress incidents, the fix is **outside** this repo (cluster issues, app-side issues, GCP LB, DNS).
|
||||
|
||||
---
|
||||
|
||||
## Escalation matrix
|
||||
|
||||
| Symptom | First responder | Escalate to |
|
||||
|---------|-----------------|-------------|
|
||||
| §1 (pending — scheduling) | Yourself with the values fix | Cluster owner if node pool full |
|
||||
| §2 (CrashLoop — TLS) | cert-manager team | Security if cert source unknown |
|
||||
| §2 (CrashLoop — invalid HTTPProxy) | Consuming team | Platform if Contour itself is broken |
|
||||
| §3 (no endpoints / LB) | Cluster owner | Platform team |
|
||||
| §4 (5xx) | App team | Platform team if Contour-side |
|
||||
| §5 (DNS) | Platform team | — |
|
||||
| §C (cluster-wide outage) | Platform team — pager | — |
|
||||
|
||||
---
|
||||
|
||||
## Done conditions
|
||||
|
||||
- `kubectl get pods -n projectcontour` shows N/N Ready for the affected instances.
|
||||
- Synthetic probes from the app team return expected status codes.
|
||||
- If the root cause was in this repo, the fix is on `main` and synced.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Reference: [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md).
|
||||
- Runbook: [pod-pending-scheduling.md](pod-pending-scheduling.md).
|
||||
- Runbook: [argocd-sync-failure.md](argocd-sync-failure.md).
|
||||
- ADR: [ADR-A3-per-cluster-scheduling.md](../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md).
|
||||
Reference in New Issue
Block a user