devops-infra-argo-config-gcp/docs/platform/runbooks/deployment-stuck.md

Runbook: Deployment Stuck After Sync

Symptom → diagnosis → remediation for tools that ArgoCD reports as Synced but whose Kubernetes resources are not healthy.

Layer: 2 (Advisory). Agents diagnose and suggest; humans execute kubectl/ArgoCD commands.

Per AI Blitz Plan §5.2. See also: argocd-sync-failure.md (for sync errors), render-failure.md (for Helm render errors).

---

Symptoms

• ArgoCD Application shows Synced but Degraded or Unknown health
• Pods are in Pending, CrashLoopBackOff, ImagePullBackOff, or OOMKilled
• Tool was working before a chart upgrade or values change
• Application is Synced + Healthy but the tool isn't functioning as expected

---

Decision tree

Application shows Synced but Degraded?
    │
    ├── Check pod status in the tool's namespace:
    │   kubectl get pods -n <namespace> --context=<cluster>
    │
    ├── Pod status is Pending?
    │   └── → [A] Resource constraints or node issues
    │
    ├── Pod status is CrashLoopBackOff?
    │   └── → [B] Application crash — check logs
    │
    ├── Pod status is ImagePullBackOff or ErrImagePull?
    │   └── → [C] Image registry issue
    │
    ├── Pod status is OOMKilled?
    │   └── → [D] Memory limit too low
    │
    ├── Pod runs but tool is unhealthy?
    │   └── → [E] Config or connectivity issue
    │
    └── Pods are fine but ArgoCD shows Degraded?
        └── → [F] Health check misconfiguration

---

[A] Pod Pending

Diagnosis:

kubectl describe pod <pod-name> -n <namespace> --context=<cluster> | grep -A 20 Events

Common causes and fixes:

Event message	Cause	Fix
Insufficient cpu/memory	Node resource exhaustion	Check node capacity; may need to adjust resource requests in custom-values.yaml (change in devops-infra-helm-charts)
did not match node affinity	Node selector or affinity mismatch	Review nodeSelector/affinity in custom-values.yaml
PersistentVolumeClaim not bound	PVC not provisioned	Check storage class and PVC events
Unschedulable	No nodes available	Check if cluster has sufficient nodes or if Karpenter/cluster-autoscaler is stuck

---

[B] CrashLoopBackOff

Diagnosis:

kubectl logs <pod-name> -n <namespace> --context=<cluster> --previous

Look for startup errors, missing config, failed connections. The fix usually requires changing custom-values.yaml in devops-infra-helm-charts — that's a separate PR in the sister repo.

---

[C] ImagePullBackOff

Diagnosis:

kubectl describe pod <pod-name> -n <namespace> --context=<cluster> | grep image

Common causes:

• Image tag doesn't exist (bad chart version pin)
• Image registry credentials expired (check external-secrets or imagePullSecrets)
• Private registry unreachable from the cluster

Fix depends on root cause. Usually requires a chart fix in devops-infra-helm-charts or a credential rotation.

---

[D] OOMKilled

Diagnosis:

kubectl top pod -n <namespace> --context=<cluster>
kubectl describe pod <pod-name> -n <namespace> --context=<cluster> | grep -A 5 "Last State"

Fix: Increase resources.limits.memory in custom-values.yaml in devops-infra-helm-charts. Open PR there.

---

[E] Config or Connectivity Issue

Tool is running but not working correctly.

Diagnosis: Check tool-specific logs. Most infra tools write structured logs.

Common causes:

• Wrong endpoint URL in custom-values.yaml
• Service dependency not available (e.g., Vault unreachable for external-secrets)
• Wrong namespace for a referenced service

Fix: Update custom-values.yaml in devops-infra-helm-charts.

---

[F] ArgoCD Health Check Misconfiguration

ArgoCD uses health checks to determine if an Application is healthy. Some custom resources have incorrect or missing health checks.

Diagnosis:

argocd app get <appName> --show-operation | grep -A 10 "Health Status"

If resources are actually healthy but ArgoCD says Degraded, this may be a health check definition issue in the ArgoCD config itself (not this repo).

Escalate to platform team via escalation-matrix.md.

---

Escalation

If diagnosis points to a cluster-level issue (node pressure, networking, GKE problem) rather than a configuration issue, escalate via escalation-matrix.md.