# ADR-0008: Canary Deploy Mandatory for Tier-1 (sp0/up0) Services in Production

**Status:** Accepted
**Category:** RELIABILITY
**Date decided:** Mid-project
**Date documented:** 2026-05-12

## Context

Meesho's services are classified by priority tier (`sp0`, `up0`, `sp1`, `up1`, etc.). Tier-1 services (`sp0`/`up0`) handle the highest traffic volumes and are critical to core business flows. A bad deploy that hits 100% of production traffic on a Tier-1 service has a catastrophic blast radius — full outage, revenue impact, and customer-facing failure. Teams were inconsistently configuring canary rollouts: some enabled them, some skipped them, and some configured them with `skipAnalysis: true` which bypasses the automated rollout analysis.

This inconsistency was the contributing factor in at least one production incident where a bad deploy on a Tier-1 service reached full traffic before the issue was detected.

## Decision

`deployArgoCD.groovy` enforces canary deployment as a hard requirement for all services with `priority_v2: sp0` or `priority_v2: up0` deploying to the `prd` environment. The enforcement checks:
1. `canary.enabled: true` must be set
2. `canary.skipAnalysis: false` — analysis cannot be bypassed
3. `canary.enableManualPromotion: true` — a human must promote the canary to full traffic

If any of these conditions are not met, the deployment is blocked with an explicit error: `"Enable canary and retry"`. Enforcement is applied at the library level — service teams cannot override it.

## Alternatives Considered

- **Documentation and guidelines only**: Rejected — teams were already aware of canary best practices but inconsistently applied them; a documented recommendation had failed to produce uniform behaviour.
- **Enforcement in Ringmaster only**: Considered but rejected — enforcement at the library level means it applies to all deploy paths, including any future tooling that calls `deployArgoCD.groovy`.
- **Enforce for all services, not just sp0/up0**: Considered but rejected as too disruptive — lower-priority services have smaller blast radii and the overhead of canary analysis was not justified for all tiers.

## Consequences

**Positive:**
- Tier-1 bad deploys cannot reach 100% of production traffic without a human promotion step.
- Canary analysis (metrics, error rate) runs automatically before promotion, catching regressions before they impact all users.
- Enforcement is consistent across all Tier-1 services — no team can skip it.

**Negative:**
- Canary rollouts add time to Tier-1 deployments — promotion requires human action, which can delay hotfixes.
- The hotfix path (`hotfix/*` branches) sets `skipAnalysis: true` to allow bypassing canary analysis in emergencies, which re-introduces the risk for the hotfix scenario.
- Services that newly cross the sp0/up0 threshold must configure canary before their next prd deploy or they will be blocked.

**Neutral:**
- The enforcement only applies to non-cron, non-worker, non-scheduler, non-consumer deployments — background jobs are excluded.

## Constraints

A production incident on a Tier-1 service drove this decision. Post-incident, the risk of leaving canary configuration to team discretion was deemed unacceptable. The library-level enforcement was the fastest way to guarantee coverage across all affected services without requiring each team to update their configuration proactively.

## Current Assessment

**Still appropriate** — no changes needed.

## Notes

- Key file: `src/com/meesho/stages/deployArgoCD.groovy:407-429` — canary enforcement block
- Hotfix bypass: `value_binding1['canary']['skipAnalysis'] = (env.hot_fix) ? true : ...` at line 355 — hotfixes can bypass canary analysis
- The `addHeadless` flag and Node services are also excluded from enforcement