devops-lib-gcp/docs/adr/0008-canary-mandatory-for-tier1-services-in-prd.md

ADR-0008: Canary Deploy Mandatory for Tier-1 (sp0/up0) Services in Production

Status: Accepted Category: RELIABILITY Date decided: Mid-project Date documented: 2026-05-12

Context

Meesho's services are classified by priority tier (sp0, up0, sp1, up1, etc.). Tier-1 services (sp0/up0) handle the highest traffic volumes and are critical to core business flows. A bad deploy that hits 100% of production traffic on a Tier-1 service has a catastrophic blast radius — full outage, revenue impact, and customer-facing failure. Teams were inconsistently configuring canary rollouts: some enabled them, some skipped them, and some configured them with skipAnalysis: true which bypasses the automated rollout analysis.

This inconsistency was the contributing factor in at least one production incident where a bad deploy on a Tier-1 service reached full traffic before the issue was detected.

Decision

deployArgoCD.groovy enforces canary deployment as a hard requirement for all services with priority_v2: sp0 or priority_v2: up0 deploying to the prd environment. The enforcement checks:

1. canary.enabled: true must be set
2. canary.skipAnalysis: false — analysis cannot be bypassed
3. canary.enableManualPromotion: true — a human must promote the canary to full traffic

If any of these conditions are not met, the deployment is blocked with an explicit error: "Enable canary and retry". Enforcement is applied at the library level — service teams cannot override it.

Alternatives Considered

• Documentation and guidelines only: Rejected — teams were already aware of canary best practices but inconsistently applied them; a documented recommendation had failed to produce uniform behaviour.
• Enforcement in Ringmaster only: Considered but rejected — enforcement at the library level means it applies to all deploy paths, including any future tooling that calls deployArgoCD.groovy.
• Enforce for all services, not just sp0/up0: Considered but rejected as too disruptive — lower-priority services have smaller blast radii and the overhead of canary analysis was not justified for all tiers.

Consequences

Positive:

• Tier-1 bad deploys cannot reach 100% of production traffic without a human promotion step.
• Canary analysis (metrics, error rate) runs automatically before promotion, catching regressions before they impact all users.
• Enforcement is consistent across all Tier-1 services — no team can skip it.

Negative:

• Canary rollouts add time to Tier-1 deployments — promotion requires human action, which can delay hotfixes.
• The hotfix path (hotfix/* branches) sets skipAnalysis: true to allow bypassing canary analysis in emergencies, which re-introduces the risk for the hotfix scenario.
• Services that newly cross the sp0/up0 threshold must configure canary before their next prd deploy or they will be blocked.

Neutral:

• The enforcement only applies to non-cron, non-worker, non-scheduler, non-consumer deployments — background jobs are excluded.

Constraints

A production incident on a Tier-1 service drove this decision. Post-incident, the risk of leaving canary configuration to team discretion was deemed unacceptable. The library-level enforcement was the fastest way to guarantee coverage across all affected services without requiring each team to update their configuration proactively.

Current Assessment

Still appropriate — no changes needed.

Notes

• Key file: src/com/meesho/stages/deployArgoCD.groovy:407-429 — canary enforcement block
• Hotfix bypass: value_binding1['canary']['skipAnalysis'] = (env.hot_fix) ? true : ... at line 355 — hotfixes can bypass canary analysis
• The addHeadless flag and Node services are also excluded from enforcement