57 lines
3.7 KiB
Markdown
57 lines
3.7 KiB
Markdown
# ADR-0008: Canary Deploy Mandatory for Tier-1 (sp0/up0) Services in Production
|
|
|
|
**Status:** Accepted
|
|
**Category:** RELIABILITY
|
|
**Date decided:** Mid-project
|
|
**Date documented:** 2026-05-12
|
|
|
|
## Context
|
|
|
|
Meesho's services are classified by priority tier (`sp0`, `up0`, `sp1`, `up1`, etc.). Tier-1 services (`sp0`/`up0`) handle the highest traffic volumes and are critical to core business flows. A bad deploy that hits 100% of production traffic on a Tier-1 service has a catastrophic blast radius — full outage, revenue impact, and customer-facing failure. Teams were inconsistently configuring canary rollouts: some enabled them, some skipped them, and some configured them with `skipAnalysis: true` which bypasses the automated rollout analysis.
|
|
|
|
This inconsistency was the contributing factor in at least one production incident where a bad deploy on a Tier-1 service reached full traffic before the issue was detected.
|
|
|
|
## Decision
|
|
|
|
`deployArgoCD.groovy` enforces canary deployment as a hard requirement for all services with `priority_v2: sp0` or `priority_v2: up0` deploying to the `prd` environment. The enforcement checks:
|
|
1. `canary.enabled: true` must be set
|
|
2. `canary.skipAnalysis: false` — analysis cannot be bypassed
|
|
3. `canary.enableManualPromotion: true` — a human must promote the canary to full traffic
|
|
|
|
If any of these conditions are not met, the deployment is blocked with an explicit error: `"Enable canary and retry"`. Enforcement is applied at the library level — service teams cannot override it.
|
|
|
|
## Alternatives Considered
|
|
|
|
- **Documentation and guidelines only**: Rejected — teams were already aware of canary best practices but inconsistently applied them; a documented recommendation had failed to produce uniform behaviour.
|
|
- **Enforcement in Ringmaster only**: Considered but rejected — enforcement at the library level means it applies to all deploy paths, including any future tooling that calls `deployArgoCD.groovy`.
|
|
- **Enforce for all services, not just sp0/up0**: Considered but rejected as too disruptive — lower-priority services have smaller blast radii and the overhead of canary analysis was not justified for all tiers.
|
|
|
|
## Consequences
|
|
|
|
**Positive:**
|
|
- Tier-1 bad deploys cannot reach 100% of production traffic without a human promotion step.
|
|
- Canary analysis (metrics, error rate) runs automatically before promotion, catching regressions before they impact all users.
|
|
- Enforcement is consistent across all Tier-1 services — no team can skip it.
|
|
|
|
**Negative:**
|
|
- Canary rollouts add time to Tier-1 deployments — promotion requires human action, which can delay hotfixes.
|
|
- The hotfix path (`hotfix/*` branches) sets `skipAnalysis: true` to allow bypassing canary analysis in emergencies, which re-introduces the risk for the hotfix scenario.
|
|
- Services that newly cross the sp0/up0 threshold must configure canary before their next prd deploy or they will be blocked.
|
|
|
|
**Neutral:**
|
|
- The enforcement only applies to non-cron, non-worker, non-scheduler, non-consumer deployments — background jobs are excluded.
|
|
|
|
## Constraints
|
|
|
|
A production incident on a Tier-1 service drove this decision. Post-incident, the risk of leaving canary configuration to team discretion was deemed unacceptable. The library-level enforcement was the fastest way to guarantee coverage across all affected services without requiring each team to update their configuration proactively.
|
|
|
|
## Current Assessment
|
|
|
|
**Still appropriate** — no changes needed.
|
|
|
|
## Notes
|
|
|
|
- Key file: `src/com/meesho/stages/deployArgoCD.groovy:407-429` — canary enforcement block
|
|
- Hotfix bypass: `value_binding1['canary']['skipAnalysis'] = (env.hot_fix) ? true : ...` at line 355 — hotfixes can bypass canary analysis
|
|
- The `addHeadless` flag and Node services are also excluded from enforcement
|