3.6 KiB
3.6 KiB
Escalation Matrix — devops-infra-argo-config
When an agent or human hits a boundary, blocker, or incident in this repo, use this matrix to determine who to contact and how fast.
Audience: Agents (to know when to halt and hand off), on-call humans (to know who owns what).
Per AI Blitz Plan §5.2. Layer: 1-T.
Escalation triggers
| Situation | Urgency | Who | How |
|---|---|---|---|
Agent would need to edit repository.yaml |
Non-urgent | Platform team | Slack #devops-tech — this is automation-owned |
Agent would need to push to main/develop/pre-prod directly |
Non-urgent | Platform team | Open PR instead; if truly urgent, page on-call |
generic-argo-apps-chart/ change needed |
Non-urgent | Platform team senior review | PR + explicit sign-off from two platform team members |
| Helm render failure after PR merge, tools broken | Urgent | On-call platform engineer | PagerDuty Devops + Slack #devops-tech |
| ArgoCD sync stuck on a cluster > 10 minutes | Urgent | On-call platform engineer | PagerDuty Devops + Slack #devops-tech |
| Accidental secret committed to git | Critical | Platform team lead + security | Slack #devops-tech immediately; do NOT merge; requires history purge |
| Incubator file deleted, cluster tooling orphaned | Critical | On-call platform engineer | PagerDuty Devops + Slack #devops-tech — restore file immediately |
clusterSpec.destination.name changed to wrong cluster |
Critical | On-call platform engineer | Revert PR immediately; tools may be deploying to wrong cluster |
teamSpec.source.repoURL changed |
Critical | On-call platform engineer | Revert PR immediately; all cluster tools affected |
AppProject (sre/sec) modified |
Urgent | Platform team + security | Security review required before merge |
| Pre-commit hooks consistently failing on valid YAML | Non-urgent | Platform team | Slack #devops-tech — may be hook version issue |
| Cluster decommission requested | Non-urgent | Platform team + cluster owner | Coordinated removal: incubator + values files + helm-overrides cleanup |
Urgency definitions
| Level | Meaning | Response time |
|---|---|---|
| Critical | Production tooling broken or security incident | Page immediately; respond in < 15 min |
| Urgent | Cluster sync stuck, tools not deploying | Page if outside business hours; Slack if in-hours; respond in < 1 hour |
| Non-urgent | Question, blocked agent, process clarification | Slack message; respond in same business day |
Contact channels
| Channel | Purpose |
|---|---|
Slack #devops-tech |
Day-to-day questions, production incidents, and security-sensitive issues (secret exposure, AppProject changes) |
PagerDuty Devops |
On-call page for critical/urgent issues outside business hours |
Agent halt protocol
When an agent reaches a Layer 3 boundary or an unresolvable blocker:
- Stop — do not attempt a workaround.
- State what operation was attempted and why it's blocked (cite rule from SANCTITY_RULES.md or AGENT_BOUNDARIES.md).
- Recommend the correct human path (e.g., "Open a PR targeting platform team review" or "Page on-call via PagerDuty").
- Do not retry the blocked operation with slightly different parameters.
Example halt message:
Blocked by R5 (SANCTITY_RULES.md):
repository.yamlis owned byregistry-bootstrapautomation and must not be hand-edited. To change ownership metadata, the request must go to the upstream automation team via Slack#devops-tech.