gitadmin/devops-infra-argo-config-gcp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added repo

Browse Source
This commit is contained in:
Your Name
2026-08-26 04:03:34 +05:30
parent 2389ec1fd6
commit 1055e1394f
150 changed files with 12395 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/global/AGENT_BOUNDARIES.md
+94
View File
@@ -0,0 +1,94 @@
# Agent Boundaries — `devops-infra-argo-config`
> Authoritative document defining which operations agents can perform in this repository.
>
> **Audience:** Claude Code agents, human reviewers of agent-generated PRs.
>
> **Model:** 3-Layer Operating Model per AI Blitz Plan §4.2.
---
## The 3-Layer Model
| Layer | What agents do | Safety gate |
| ----- | -------------- | ----------- |
| **Layer 1-T** (Tool-Mediated) | Generate YAML diff via tools, open PR. No direct YAML edits. | PR review by platform team. |
| **Layer 2** (Agent-Readable Advisory) | Research, analyze, suggest. Human executes. | Human judgment. |
| **Layer 3** (Agent-Blocked) | Refuse the write. Direct user to correct owner. | Hard stop. |
**This repo is Layer 1-T.** All YAML changes should go through `incubator-tool` or `argo-app-tool` where available. Direct YAML edits bypass tool validation but are acceptable when tools are not available, provided the PR checklist is followed.
---
## Per-operation classification
### Layer 1-T — Standard operations
| Operation | Files affected | Blast radius | Tool | Approval |
| --------- | ------------- | ------------ | ---- | -------- |
| Add appSpec entry to values file | `values/<env>/<cluster>-values.yaml` | Single cluster, single tool | `argo-app-tool` | Platform team PR review |
| Remove appSpec entry from values file | `values/<env>/<cluster>-values.yaml` | Single cluster, single tool | `argo-app-tool` | Platform team PR review |
| Update appSpec entry (chartDir, valuesDir, namespace) | `values/<env>/<cluster>-values.yaml` | Single cluster, single tool | `argo-app-tool` | Platform team PR review |
| Add `nameOverride` to appSpec entry | `values/<env>/<cluster>-values.yaml` | Single cluster, single tool | `argo-app-tool` | Requires justification in PR |
| Create incubator file for new cluster | `incubator/<env>/<cluster>.yaml` | New cluster bootstrap | `incubator-tool` | Platform team PR review |
| Create values file for new cluster | `values/<env>/<cluster>-values.yaml` | New cluster bootstrap | `argo-app-tool` | Platform team PR review |
| Add external-name-service files | `external-name-service-*/**` | Cross-cluster DNS routing | Manual | Platform team PR review |
| Modify AppProject | `projects/*.yaml` | All Applications referencing the project | Manual | Platform team + security review |
### Layer 1-T — HIGH RISK operations
These are Layer 1-T (agent may generate the diff + open PR) but carry elevated risk:
| Operation | Why high risk | Extra requirement |
| --------- | ------------- | ----------------- |
| Edit `generic-argo-apps-chart/` | Auto-sync means a bad template change breaks ALL clusters immediately | Explicit platform-team sign-off; test with `helm template` against multiple values files |
| Modify `clusterSpec.destination` in values file | Redirects all tools to a different cluster | Verify cluster name exists in GKE |
| Change `teamSpec.source.targetRevision` | Switches chart source branch — affects all tools on the cluster | Must match env branch: `main` (prd), `develop` (stg/dev), `pre-prod` (int) |
| Change `teamSpec.source.repoURL` | Switches chart source repo entirely | Requires platform-team approval |
| Bulk-add appSpec entries to multiple clusters | Fleet-wide tool rollout | Each cluster's entry must be individually verified |
### Layer 2 — Advisory only (no agent write)
| Operation | Agent action |
| --------- | ------------ |
| Run `argocd app sync <appName>` | Suggest the command; do not execute. Infra apps auto-sync from `main`. |
| Run `argocd app diff <appName>` | Suggest the command if cluster access is available. |
| Investigate sync failure | Read logs, analyze values, suggest fix. See [runbooks/argocd-sync-failure.md](../platform/runbooks/argocd-sync-failure.md). |
| Check which clusters have a tool | Run `grep -rl 'name: <tool>' values/` — read-only. |
| Audit AppProject scope | Read `projects/` and list dependent Applications — advisory only. |
### Layer 3 — Hard stops (agent must refuse)
| Operation | Why blocked | Redirect |
| --------- | ----------- | -------- |
| Edit `repository.yaml` | Owned by `registry-bootstrap` automation | Direct user to upstream automation |
| Run `kubectl apply` against a cluster | Out of scope — this repo is GitOps, not in-cluster mutation | Direct to incident response procedures |
| Push directly to `main` | Branch protection enforced at org level | Open PR instead |
| Delete an incubator file | Orphans the cluster's tooling | Require confirmation that cluster is decommissioned |
| Force-push to any branch | Destructive, irreversible | Never do this |
| Skip pre-commit hooks (`--no-verify`) | Bypasses TruffleHog, CAC, Yaak | Fix the hook failure instead |
| Add secrets to YAML | TruffleHog will block; if bypassed, requires history purge | Use external-secrets operator |
---
## Cross-cut verification checklist (every Layer 1-T PR)
Before merging any agent-generated PR:
1. Tool was used (or manual edit follows tool conventions).
2. `chartDir` exists in `devops-infra-helm-charts/helm-templates/`.
3. `valuesDir` exists in `devops-infra-helm-charts/helm-overrides/<cluster>/`.
4. Auto-generated Application name is <= 253 characters.
5. No name collision with existing appSpec entries in the same values file.
6. `nameOverride` is used only when justified (R7).
7. `spec.project` references an existing AppProject (`sre` or `sec`).
---
## Escalation procedure
When an operation falls outside Layer 1-T boundaries:
1. Refuse the write.
2. Cite this document and the specific layer/operation.
3. Suggest the human ask the platform team or file a CMR (Change Management Request).
docs/global/SANCTITY_RULES.md
+118
View File
@@ -0,0 +1,118 @@
# Sanctity Rules — `devops-infra-argo-config`
> Non-negotiable rules for this repository. Violations are hard stops — not warnings, not suggestions.
>
> **Audience:** Every agent and human who opens a PR against this repo.
>
> **Enforcement:** Pre-commit hooks (TruffleHog, CAC validate, Yaak) + branch protection on `main`. A merge to `main` **immediately deploys** — infra apps are auto-sync.
---
## R1: `main` is production
A merge to `main` triggers immediate ArgoCD auto-sync across all managed clusters. There is no staging gate between merge and deploy.
- No experiments on `main`.
- No force-pushes to `main`.
- Every PR must pass pre-commit hooks and receive platform-team review before merge.
---
## R2: The incubator ↔ values ↔ generic-chart contract is sacred
Each cluster is bootstrapped by exactly three things working together:
1. **Incubator file** (`incubator/<env>/<cluster>.yaml`) — parent ArgoCD Application that points at the generic chart + the cluster's values file.
2. **Values file** (`values/<env>/<cluster>-values.yaml`) — defines `clusterSpec`, `teamSpec`, `argocdSpec`, and the `appSpec[]` list.
3. **Generic chart** (`generic-argo-apps-chart/`) — renders one child ArgoCD Application per `appSpec` entry.
Breaking any leg of this triangle breaks the cluster's tooling. The naming conventions in CLAUDE.md are load-bearing — they are identifiers, not cosmetic.
---
## R3: `appSpec` entries must reference existing chart and values in `devops-infra-helm-charts`
Every `appSpec[].chartDir` must correspond to a directory under `devops-infra-helm-charts/helm-templates/`. Every `appSpec[].valuesDir` must correspond to a directory under `devops-infra-helm-charts/helm-overrides/<cluster>/`. If either doesn't exist, the ArgoCD Application will fail to render.
**Verify before adding:**
```bash
# chartDir exists?
ls devops-infra-helm-charts/helm-templates/<chartDir>/
# valuesDir exists?
ls devops-infra-helm-charts/helm-overrides/<cluster>/<valuesDir>/custom-values.yaml
```
---
## R4: Child Application names are auto-generated — do not invent names
The generic chart template generates Application names as:
```
<appSpec.name>-<mungedCluster>-<env>
```
Where cluster munging strips: `k8s-`, `prd-`, `int-`, `dev-`, `-ase1`; maps `-ase1c` to `-c`.
Example: `keda` on `k8s-central-prd-ase1` becomes `keda-central-prd`.
Only use `nameOverride` when:
- The auto-generated name would exceed 253 characters (Kubernetes limit).
- There is a genuine name collision between two appSpec entries.
---
## R5: `repository.yaml` is automation-owned
Owned by `registry-bootstrap`. Direct edits will be overwritten. To change ownership metadata, go to the upstream automation.
---
## R6: Secrets stay out of git
No credentials, tokens, API keys, or passwords in any YAML file. The TruffleHog pre-commit hook will block the commit. If a secret is accidentally committed, it must be force-purged from history — this requires platform-team involvement.
---
## R7: `nameOverride` is exceptional, not routine
Most PRs should not include `nameOverride`. When reviewing, flag any `nameOverride` that lacks a comment explaining why the auto-generated name was insufficient.
---
## R8: One incubator file per cluster, one values file per cluster
The 1:1 mapping between incubator files and values files is structural. Do not create multiple incubator files for the same cluster or share a values file across clusters.
---
## R9: Cluster folder/file names are routing keys
The cluster name in the values file (`clusterSpec.destination.name`) must match the incubator file name and the `helm-overrides/` folder name in the sister repo. These names are used by ArgoCD for cluster routing — a mismatch means tools deploy to the wrong cluster or fail silently.
---
## R10: Pre-commit hooks must pass
Run `pre-commit run --all-files` before pushing. Never use `--no-verify` to skip hooks. If a hook fails, fix the root cause.
The hooks are: TruffleHog (secrets scan), CAC validate (config validation), Yaak (additional linting).
---
## R11: Branch protection trumps everything
All changes go via PR with required review. Direct pushes to `main` are blocked at the GitHub org level. No exceptions for "quick fixes" — the blast radius of a bad merge is immediate cluster-wide impact.
---
## R12: AppProject wildcards are dangerous
The `sre` and `sec` projects currently allow `*` for source repos and destinations. Widening or modifying AppProject scope requires auditing every Application that references the project. A misconfigured project could allow unauthorized deployments.
---
## How to cite these rules
In PR reviews or agent halt messages, cite as: *"Blocked by R3 (SANCTITY_RULES.md) — chartDir must exist in devops-infra-helm-charts before adding appSpec."*
docs/global/coding-guidelines/argocd.md
+135
View File
@@ -0,0 +1,135 @@
> Per [AI Blitz Plan §6](../AGENT_BOUNDARIES.md). Layer: 1-T.
# Coding Guidelines — ArgoCD Application Manifests
Conventions for ArgoCD `Application` YAML files in `incubator/<env>/`.
---
## Required fields
Every incubator Application must have all of these:
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: incubator-infra-<cluster> # Must match filename without .yaml
namespace: <argocd-namespace> # argocd-prd | argocd-dev | argocd-shared-int
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: 'https://github.com/Meesho/devops-infra-argo-config'
targetRevision: <branch> # main | develop | pre-prod
path: generic-argo-apps-chart
helm:
valueFiles:
- ../values/<env>/incubator-infra-<cluster>-values.yaml
destination:
name: in-cluster
namespace: <argocd-namespace>
```
---
## `metadata.name` rules
- Must equal the filename without `.yaml` — this is ArgoCD's tracking key (R9).
- Convention: `incubator-infra-<gke-cluster-name>` (e.g. `incubator-infra-k8s-central-prd-ase1`).
- Changing this after the Application is registered breaks ArgoCD tracking; treat as immutable.
---
## `metadata.namespace` (ArgoCD namespace)
| Environment | Namespace |
| ----------- | --------- |
| prd | `argocd-prd` |
| stg / dev | `argocd-dev` |
| int | `argocd-shared-int` |
The `metadata.namespace` and `spec.destination.namespace` must be the same value.
---
## `spec.source.repoURL` allow-list
Only one value is permitted:
```text
https://github.com/Meesho/devops-infra-argo-config
```
Changing this is a **Layer 3** operation. See [AGENT_BOUNDARIES.md](../AGENT_BOUNDARIES.md).
---
## `spec.source.targetRevision`
Must match the environment branch:
| Environment | `targetRevision` |
| ----------- | ---------------- |
| prd | `main` |
| stg / dev | `develop` |
| int | `pre-prod` |
Never pin to a feature branch. Feature branches are not monitored by auto-sync.
---
## `spec.source.path`
Always `generic-argo-apps-chart`. This is the Helm chart that renders child Applications from the values file.
---
## `spec.destination.name`
Always `in-cluster`. Incubator Applications run on the ArgoCD admin cluster itself, not on the target GKE cluster.
---
## `spec.destination.namespace`
Must match `metadata.namespace` — same ArgoCD namespace.
---
## Finalizer
The finalizer `resources-finalizer.argocd.argoproj.io` must be present. It causes ArgoCD to cascade-delete child Applications when the parent incubator is deleted. Removing it creates orphaned child Applications.
---
## `helm.valueFiles` path convention
Paths are relative to the chart root (`generic-argo-apps-chart/`), so `..` is required to reach the repo root:
```text
generic-argo-apps-chart/ (chart root = ArgoCD's working dir for this source)
../values/<env>/incubator-infra-<cluster>-values.yaml
└── resolves to: values/<env>/incubator-infra-<cluster>-values.yaml at repo root
```
---
## Naming invariant
Directory and values path vary by env — but the invariant itself is universal:
```text
prd (main):
Filename: incubator/prd/incubator-infra-k8s-central-prd-ase1.yaml
metadata.name: incubator-infra-k8s-central-prd-ase1
helm.valueFiles: ../values/prd/incubator-infra-k8s-central-prd-ase1-values.yaml
stg (develop):
Filename: incubator/infra/incubator-infra-k8s-central-stg-ase1.yaml
metadata.name: incubator-infra-k8s-central-stg-ase1
helm.valueFiles: ../values/dev/ incubator-infra-k8s-central-stg-ase1-values.yaml
```
All three must be consistent. Any mismatch causes a routing failure.
docs/global/coding-guidelines/helm.md
+141
View File
@@ -0,0 +1,141 @@
> Per [AI Blitz Plan §6](../AGENT_BOUNDARIES.md). Layer: 1-T.
# Coding Guidelines — Helm Values Files
Conventions for cluster values files (`values/<env>/incubator-infra-<cluster>-values.yaml`).
---
## File naming
```text
values/<values-dir>/incubator-infra-<gke-cluster-name>-values.yaml
```
The values directory name does **not** always match the environment name:
| Environment | Branch | Values dir | Example |
| ----------- | ------ | ---------- | ------- |
| prd | `main` | `values/prd/` | `values/prd/incubator-infra-k8s-central-prd-ase1-values.yaml` |
| stg | `develop` | `values/dev/` | `values/dev/incubator-infra-k8s-central-stg-ase1-values.yaml` |
| dev | `develop` | `values/dev/` | `values/dev/incubator-infra-k8s-central-dev-ase1-values.yaml` |
| int | `pre-prod` | `values/int/` | `values/int/incubator-infra-k8s-shared-int-ase1-values.yaml` |
| admin | `main` / `pre-prod` | `values/admin/` | `values/admin/incubator-infra-k8s-devops-admin-ase1-values.yaml` |
Note: stg and dev clusters both live under `values/dev/` on the `develop` branch.
---
## Top-level structure
Four keys in this exact order — no additions, no reordering:
```yaml
clusterSpec:
argocdSpec:
teamSpec:
appSpec:
```
---
## `clusterSpec`
```yaml
clusterSpec:
destination:
server: "" # Always empty string — name-based routing only
name: "k8s-central-prd-ase1" # Must match GKE cluster name and helm-overrides/ folder
```
- `server` is always `""`. Never set a URL here.
- `name` is immutable after cluster registration. Changing it redirects all tooling.
---
## `argocdSpec`
```yaml
argocdSpec:
namespace: argocd-prd # argocd-prd | argocd-dev | argocd-shared-int
```
| Environment | Namespace |
| ----------- | --------- |
| prd | `argocd-prd` |
| stg / dev | `argocd-dev` |
| int | `argocd-shared-int` |
---
## `teamSpec`
```yaml
teamSpec:
devops:
source:
repoURL: https://github.com/Meesho/devops-infra-helm-charts
targetRevision: main # main | develop | pre-prod
path: helm-templates
valueFiles: ../../helm-overrides/<cluster>
labels:
bu: infra
team: devops
env: prd # prd | stg | dev | int | admin
cluster: k8s-central-prd-ase1 # Must match clusterSpec.destination.name
```
**`repoURL` allow-list:** Only `https://github.com/Meesho/devops-infra-helm-charts` is permitted.
**`targetRevision`** must match the environment:
| Environment | `targetRevision` |
| ----------- | ---------------- |
| prd | `main` |
| stg / dev | `develop` |
| int | `pre-prod` |
**`valueFiles` path:** Relative from `helm-templates/` root (two levels up) to the cluster's override directory. Always `../../helm-overrides/<cluster>`.
**`labels.cluster`** must equal `clusterSpec.destination.name` exactly — it is used in Application name generation.
---
## `appSpec`
```yaml
appSpec:
- name: keda # Short, lowercase, hyphen-separated tool name
namespace: keda-central-prd # Target K8s namespace
chartDir: keda # Directory under helm-templates/ in helm-charts repo
valuesDir: keda # Directory under helm-overrides/<cluster>/ in helm-charts repo
```
- One entry = one ArgoCD child Application.
- Generated name: `<name>-<mungedCluster>-<env>` (e.g. `keda-central-prd`).
- `chartDir` and `valuesDir` **must exist** in `devops-infra-helm-charts` before adding the entry (R3).
### Optional fields
```yaml
- name: coredns
namespace: kube-system
chartDir: coredns
valuesDir: coredns
nameOverride: coredns-central-prd # Only for name collisions or >253 char names (R7)
additionalValueFiles:
- ../../helm-templates/coredns/gcp-ase1a-values.yaml
```
---
## Common mistakes
| Mistake | Impact | Fix |
| ------- | ------ | --- |
| `server: <url>` instead of `""` | May route to wrong cluster | Always use `""` |
| Wrong `targetRevision` for env | Tools pull from wrong chart branch | Use `main`/`develop`/`pre-prod` per env |
| `chartDir` not in `helm-templates/` | ArgoCD render failure | Verify in `devops-infra-helm-charts` first |
| `valuesDir` missing `custom-values.yaml` | ArgoCD render failure | Add `custom-values.yaml` to the override dir |
| `labels.cluster` ≠ `clusterSpec.destination.name` | Application name munging uses wrong cluster | Keep them identical |
| Duplicate `name` in `appSpec` | Application name collision | Use `nameOverride` for multi-instance tools |
docs/global/coding-guidelines/infra-argo.md
+149
View File
@@ -0,0 +1,149 @@
# Coding Guidelines — Values Files and appSpec Entries
> YAML authoring conventions for `devops-infra-argo-config`.
>
> **Scope:** `values/<env>/<cluster>-values.yaml` files and `appSpec` entries within them.
---
## Values file structure
Every values file has exactly four top-level keys in this order:
```yaml
clusterSpec: # Cluster identity — destination for ArgoCD
argocdSpec: # ArgoCD namespace
teamSpec: # Source repo, labels, team identity
appSpec: # List of tools to deploy to this cluster
```
Do not add other top-level keys. Do not reorder these keys.
---
## `clusterSpec` conventions
```yaml
clusterSpec:
destination:
server: "" # Leave empty — use name-based routing
name: "k8s-central-prd-ase1" # Must match the cluster name in GKE
```
- `server` is always empty string `""` — name-based routing is the standard.
- `name` must exactly match the GKE cluster name and the `helm-overrides/` folder in the sister repo.
---
## `argocdSpec` conventions
```yaml
argocdSpec:
namespace: argocd-prd # prd → argocd-prd | stg/dev → argocd-dev | int → argocd-shared-int
```
---
## `teamSpec` conventions
```yaml
teamSpec:
devops:
source:
repoURL: https://github.com/Meesho/devops-infra-helm-charts
targetRevision: main # Env-specific: prd=main, stg/dev=develop, int=pre-prod
path: helm-templates # Chart root in the helm-charts repo
valueFiles: ../../helm-overrides/k8s-central-prd-ase1 # Relative path to overrides
labels:
bu: infra # Always "infra" for this repo
team: devops # Always "devops" — maps to sre AppProject
env: prd # prd, int, dev, or admin
cluster: k8s-central-prd-ase1 # Must match clusterSpec.destination.name
```
- `targetRevision` must match the environment branch: `main` (prd), `develop` (stg/dev), `pre-prod` (int). Deviating from this requires explicit justification.
- `valueFiles` is a relative path from the chart source to the cluster's override directory in `devops-infra-helm-charts`.
- Labels are used by the generic chart template for Application naming and metadata.
---
## `appSpec` entry conventions
Each entry in `appSpec` defines one ArgoCD child Application:
```yaml
appSpec:
- name: keda # Short tool name (used in Application name generation)
namespace: keda-central-prd # Target namespace (auto-created by ArgoCD)
chartDir: keda # Directory under helm-templates/ in helm-charts repo
valuesDir: keda # Directory under helm-overrides/<cluster>/ in helm-charts repo
```
### Required fields
| Field | Description | Convention |
| ----- | ----------- | ---------- |
| `name` | Short tool identifier | Lowercase, hyphen-separated. Used in rendered Application name. |
| `namespace` | Kubernetes namespace for the tool | Pattern: `<tool>-<mungedCluster>` or shared namespace (e.g., `victoriametrics`, `kube-system`) |
| `chartDir` | Chart directory name in `helm-templates/` | Must exist in `devops-infra-helm-charts` |
| `valuesDir` | Override directory name in `helm-overrides/<cluster>/` | Must exist in `devops-infra-helm-charts` |
### Optional fields
| Field | When to use |
| ----- | ----------- |
| `nameOverride` | Only when the auto-generated name exceeds 253 chars or collides with another entry |
| `additionalValueFiles` | When a tool needs region-shared overlays (e.g., CoreDNS GCP zone values) |
### Ordering
New appSpec entries should be appended at the end of the list. Do not sort alphabetically — the order reflects deployment history and makes diffs cleaner.
---
## Namespace naming patterns
| Pattern | When |
| ------- | ---- |
| `<tool>-<mungedCluster>` | Default. Example: `keda-central-prd`, `contour-external-central-prd` |
| Shared namespace | When multiple tools share a namespace. Example: `victoriametrics` for all VM tools, `kube-system` for system tools |
| Tool-specific with role | Multi-instance tools. Example: `contour-internal-0-central-prd`, `contour-internal-1-central-prd` |
---
## Application name generation
The generic chart template generates names as:
```
<name>-<mungedCluster>-<env>
```
**Cluster munging rules** (applied in order by the Helm template):
1. Replace `dp-` with placeholder, `backup` with placeholder
2. Strip: `p-`, `prd-`, `int-`, `dev-`, `-cluster`
3. Replace: `prod-ops` → `infra`, `-ase1c` → `-c`, `-ase1` → (empty), `k8s-` → (empty)
4. Restore placeholders
**Example:** `k8s-central-prd-ase1` → `central-prd` → Application name: `keda-central-prd`
---
## YAML formatting
- 2-space indentation (no tabs).
- No trailing whitespace.
- Single newline at end of file.
- Quote strings only when YAML requires it (e.g., empty strings `""`).
- Use block style for lists (one `- ` per line), not flow style.
---
## Common mistakes
| Mistake | Why it's wrong | Fix |
| ------- | -------------- | --- |
| Adding `nameOverride` without justification | Breaks naming consistency; see R7 | Remove unless name > 253 chars or collision |
| `chartDir` that doesn't exist in helm-charts | ArgoCD will fail to render the Application | Verify with `ls helm-templates/<chartDir>/` |
| Duplicate `name` in same appSpec list | Two Applications will have the same name → conflict | Use unique tool names or `nameOverride` for multi-instance |
| Changing `targetRevision` away from env branch | All tools on the cluster pull from the wrong branch | Must match env: `main` (prd), `develop` (stg/dev), `pre-prod` (int) |
docs/global/escalation-matrix.md
+60
View File
@@ -0,0 +1,60 @@
# Escalation Matrix — `devops-infra-argo-config`
> When an agent or human hits a boundary, blocker, or incident in this repo, use this matrix to determine who to contact and how fast.
>
> **Audience:** Agents (to know when to halt and hand off), on-call humans (to know who owns what).
>
> Per [AI Blitz Plan §5.2](AGENT_BOUNDARIES.md). Layer: **1-T**.
---
## Escalation triggers
| Situation | Urgency | Who | How |
| --------- | ------- | --- | --- |
| Agent would need to edit `repository.yaml` | Non-urgent | Platform team | Slack `#devops-tech` — this is automation-owned |
| Agent would need to push to `main`/`develop`/`pre-prod` directly | Non-urgent | Platform team | Open PR instead; if truly urgent, page on-call |
| `generic-argo-apps-chart/` change needed | Non-urgent | Platform team senior review | PR + explicit sign-off from two platform team members |
| Helm render failure after PR merge, tools broken | **Urgent** | On-call platform engineer | PagerDuty `Devops` + Slack `#devops-tech` |
| ArgoCD sync stuck on a cluster > 10 minutes | **Urgent** | On-call platform engineer | PagerDuty `Devops` + Slack `#devops-tech` |
| Accidental secret committed to git | **Critical** | Platform team lead + security | Slack `#devops-tech` immediately; do NOT merge; requires history purge |
| Incubator file deleted, cluster tooling orphaned | **Critical** | On-call platform engineer | PagerDuty `Devops` + Slack `#devops-tech` — restore file immediately |
| `clusterSpec.destination.name` changed to wrong cluster | **Critical** | On-call platform engineer | Revert PR immediately; tools may be deploying to wrong cluster |
| `teamSpec.source.repoURL` changed | **Critical** | On-call platform engineer | Revert PR immediately; all cluster tools affected |
| AppProject (`sre`/`sec`) modified | **Urgent** | Platform team + security | Security review required before merge |
| Pre-commit hooks consistently failing on valid YAML | Non-urgent | Platform team | Slack `#devops-tech` — may be hook version issue |
| Cluster decommission requested | Non-urgent | Platform team + cluster owner | Coordinated removal: incubator + values files + helm-overrides cleanup |
---
## Urgency definitions
| Level | Meaning | Response time |
| ----- | ------- | ------------- |
| **Critical** | Production tooling broken or security incident | Page immediately; respond in < 15 min |
| **Urgent** | Cluster sync stuck, tools not deploying | Page if outside business hours; Slack if in-hours; respond in < 1 hour |
| **Non-urgent** | Question, blocked agent, process clarification | Slack message; respond in same business day |
---
## Contact channels
| Channel | Purpose |
| ------- | ------- |
| Slack `#devops-tech` | Day-to-day questions, production incidents, and security-sensitive issues (secret exposure, AppProject changes) |
| PagerDuty `Devops` | On-call page for critical/urgent issues outside business hours |
---
## Agent halt protocol
When an agent reaches a Layer 3 boundary or an unresolvable blocker:
1. **Stop** — do not attempt a workaround.
2. **State** what operation was attempted and why it's blocked (cite rule from [SANCTITY_RULES.md](SANCTITY_RULES.md) or [AGENT_BOUNDARIES.md](AGENT_BOUNDARIES.md)).
3. **Recommend** the correct human path (e.g., "Open a PR targeting platform team review" or "Page on-call via PagerDuty").
4. **Do not retry** the blocked operation with slightly different parameters.
Example halt message:
> Blocked by R5 (SANCTITY_RULES.md): `repository.yaml` is owned by `registry-bootstrap` automation and must not be hand-edited. To change ownership metadata, the request must go to the upstream automation team via Slack `#devops-tech`.
docs/platform/procedures/add-new-cluster.md
+168
View File
@@ -0,0 +1,168 @@
# Procedure: Onboard a New Cluster
> Step-by-step guide for adding a new Kubernetes cluster to the ArgoCD infrastructure GitOps control plane.
>
> **Layer:** 1-T (Tool-Mediated). Two files must be created: one incubator file and one values file.
>
> **Blast radius:** New cluster only. Existing clusters are unaffected.
>
> **Prerequisite:** The GKE cluster must already be provisioned via `terraform-google-modules` and registered as an ArgoCD cluster destination.
---
## Inputs
| Input | Example | Where it comes from |
| ----- | ------- | ------------------- |
| Cluster name | `k8s-dsgpu-prd-ase1` | GKE cluster provisioning (Terraform) |
| Environment | `prd` | Cluster naming convention |
| ArgoCD namespace | See table below | Depends on environment |
| Target branch | See table below | Depends on environment |
| Initial tools | `keda`, `contour`, `external-secrets`, etc. | Platform team decision |
**Environment → branch / ArgoCD namespace mapping:**
| Environment | Branch | ArgoCD namespace | Values dir |
| ----------- | ------ | ---------------- | ---------- |
| prd | `main` | `argocd-prd` | `values/prd/` |
| stg / dev | `develop` | `argocd-dev` | `values/dev/` |
| int | `pre-prod` | `argocd-shared-int` | `values/int/` |
---
## Step 1: Create the incubator file
Create `incubator/<env>/<cluster>.yaml`:
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: incubator-infra-<cluster>
namespace: <argocd-namespace> # argocd-prd | argocd-dev | argocd-shared-int
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: 'https://github.com/Meesho/devops-infra-argo-config'
targetRevision: <branch> # main (prd) | develop (stg/dev) | pre-prod (int)
path: generic-argo-apps-chart
helm:
valueFiles:
- ../values/<env>/incubator-infra-<cluster>-values.yaml
destination:
name: in-cluster
namespace: <argocd-namespace> # argocd-prd | argocd-dev | argocd-shared-int
```
**Key points:**
- `metadata.name`: `incubator-infra-<cluster>` (matches the file name minus `.yaml`).
- `spec.project`: `default` (the incubator itself runs in the ArgoCD admin cluster).
- `spec.source.path`: Always `generic-argo-apps-chart`.
- `helm.valueFiles`: Relative path to the values file from the chart directory.
- `spec.destination.name`: `in-cluster` (the incubator runs on the ArgoCD admin cluster, not the target cluster).
---
## Step 2: Create the values file
Create `values/<env>/incubator-infra-<cluster>-values.yaml`:
```yaml
clusterSpec:
destination:
server: ""
name: "<cluster>"
argocdSpec:
namespace: <argocd-namespace> # argocd-prd | argocd-dev | argocd-shared-int
teamSpec:
devops:
source:
repoURL: https://github.com/Meesho/devops-infra-helm-charts
targetRevision: <branch> # main (prd) | develop (stg/dev) | pre-prod (int)
path: helm-templates
valueFiles: ../../helm-overrides/<cluster>
labels:
bu: infra
team: devops
env: <env>
cluster: <cluster>
appSpec: []
```
Start with an empty `appSpec` list. Add tools in a follow-up PR after the cluster is bootstrapped.
---
## Step 3: Register the cluster in the admin cluster's values file
The admin cluster (`k8s-admin-prd-ase1`) may need an entry to reference the new cluster's ArgoCD Application. Check if the admin values file at `values/prd/incubator-infra-k8s-admin-prd-ase1-values.yaml` needs an `appSpec` entry for the new cluster's ArgoCD instance.
---
## Step 4: Create external-name-service files (if zone-c)
For multi-zone clusters (e.g., `-ase1c`), create:
1. `external-name-service-incubator/<env>/<zone>/external-name-service-incubator-<cluster>.yaml`
2. Update external-name-service values for the zone if needed.
---
## Step 5: Validate locally
```bash
# Render the incubator Application
helm template generic-argo-apps-chart/ \
-f values/<env>/incubator-infra-<cluster>-values.yaml
# Verify YAML syntax
yamllint values/<env>/incubator-infra-<cluster>-values.yaml
yamllint incubator/<env>/incubator-infra-<cluster>.yaml
```
---
## Step 6: Open PR
- Both files (incubator + values) in the same PR.
- Target branch: matches environment — `main` (prd), `develop` (stg/dev), `pre-prod` (int).
- Required: Platform team review.
---
## Step 7: Post-merge bootstrap (manual, platform team)
After the PR merges:
1. The incubator Application auto-syncs to the ArgoCD admin cluster.
2. ArgoCD reads the values file and renders child Applications (initially none if `appSpec` is empty).
3. Verify in ArgoCD UI that the incubator Application is healthy.
4. Add initial tools via [add-tool-to-cluster.md](add-tool-to-cluster.md).
---
## Naming convention reference
| Component | Convention | Example |
| --------- | ---------- | ------- |
| Incubator file | `incubator/<env>/incubator-infra-<cluster>.yaml` | `incubator/prd/incubator-infra-k8s-dsgpu-prd-ase1.yaml` |
| Values file | `values/<env>/incubator-infra-<cluster>-values.yaml` | `values/prd/incubator-infra-k8s-dsgpu-prd-ase1-values.yaml` |
| Incubator Application name | `incubator-infra-<cluster>` | `incubator-infra-k8s-dsgpu-prd-ase1` |
---
## Checklist
- [ ] GKE cluster is provisioned and registered in ArgoCD
- [ ] Incubator file created with correct name and paths
- [ ] Values file created with correct `clusterSpec.destination.name`
- [ ] `valueFiles` path in incubator points to correct values file
- [ ] `teamSpec.source.valueFiles` points to correct `helm-overrides/<cluster>` directory
- [ ] Labels match the cluster name and environment
- [ ] Both files pass `yamllint`
- [ ] Pre-commit hooks pass
docs/platform/procedures/add-tool-to-cluster.md
+182
View File
@@ -0,0 +1,182 @@
# Procedure: Add, Update, or Remove a Tool from a Cluster
> Step-by-step guide for modifying `appSpec` entries in cluster values files.
>
> **Layer:** 1-T (Tool-Mediated). Use `argo-app-tool` where available; manual YAML edits acceptable with checklist.
>
> **Blast radius:** Single cluster, single tool. Auto-sync means changes deploy immediately after merge to the env branch (`main` for prd, `develop` for stg, `pre-prod` for int).
---
## Prerequisites
Before starting, confirm:
1. The tool's **chart** exists in `devops-infra-helm-charts/helm-templates/<chartDir>/` on the correct branch.
2. The tool's **values override** exists in `devops-infra-helm-charts/helm-overrides/<cluster>/<valuesDir>/custom-values.yaml` on the correct branch.
3. You know which **cluster** and **environment** the tool should deploy to.
Both this repo and `devops-infra-helm-charts` use the same branch convention:
| Environment | Branch |
| ----------- | ------ |
| Production (prd) | `main` |
| Staging (stg) | `develop` |
| Integration (int) | `pre-prod` |
If either the chart or override doesn't exist on the target branch, the work starts in `devops-infra-helm-charts` first. See [devops-infra-helm-charts procedures](https://github.com/Meesho/devops-infra-helm-charts).
---
## Add a new tool
### Step 1: Identify the target values file
```text
values/<env>/incubator-infra-<cluster>-values.yaml
```
Example: `values/prd/incubator-infra-k8s-central-prd-ase1-values.yaml`
### Step 2: Verify chart and values exist in the sister repo
```bash
# Chart directory
ls /path/to/devops-infra-helm-charts/helm-templates/<chartDir>/
# Values override
ls /path/to/devops-infra-helm-charts/helm-overrides/<cluster>/<valuesDir>/custom-values.yaml
```
If either is missing, stop. Create them in `devops-infra-helm-charts` first.
### Step 3: Compute the auto-generated Application name
Apply the munging rules from [coding-guidelines/infra-argo.md](../global/coding-guidelines/infra-argo.md):
```text
<name>-<mungedCluster>-<env>
```
Example: `kyverno` on `k8s-farmiso-prd-ase1` → `kyverno-farmiso-prd`
Verify:
- Name is <= 253 characters.
- Name doesn't collide with an existing Application (grep the values file).
### Step 4: Add the appSpec entry
Append to the `appSpec` list in the values file:
```yaml
- name: kyverno
namespace: kyverno-farmiso-prd
chartDir: kyverno
valuesDir: kyverno
```
Only add `nameOverride` if Step 3 revealed a collision or length issue.
Only add `additionalValueFiles` if the tool requires region-shared overlays.
### Step 5: Validate locally
```bash
# Render the generic chart with the updated values
helm template generic-argo-apps-chart/ \
-f values/prd/incubator-infra-<cluster>-values.yaml | grep -A 20 "name: kyverno"
# Verify no YAML syntax errors
yamllint values/prd/incubator-infra-<cluster>-values.yaml
```
### Step 6: Open PR
- Target branch: `main` (prd) / `develop` (stg) / `pre-prod` (int)
- Required: Platform team review
- Pre-commit hooks must pass (TruffleHog, CAC, Yaak)
### Step 7: After merge
ArgoCD auto-syncs from the env branch. The new Application will appear in ArgoCD within minutes. Verify in the ArgoCD UI that:
- The Application is created with the expected name.
- It syncs successfully.
- The target namespace is created.
---
## Update an existing tool
### Change chart version
Update `chartDir` to point to the new chart directory:
```yaml
- name: contour-internal-0
chartDir: contour-v1.33.3 # was: contour
```
Verify the new `chartDir` exists in `helm-templates/`.
### Change values directory
Update `valuesDir`:
```yaml
- name: vmagent
valuesDir: victoria-metrics-agent-new # was: victoria-metrics-agent
```
Verify the new `valuesDir` exists in `helm-overrides/<cluster>/`.
### Change namespace
Update `namespace`. This is a destructive operation — ArgoCD will create the new namespace and deploy there, but the old namespace's resources are **not automatically cleaned up**.
```yaml
- name: ai-gateway
namespace: ai-gateway-prd # was: ai-gateway
```
After merge, manually clean up the old namespace if no other tools use it.
---
## Remove a tool from a cluster
### Step 1: Delete the appSpec entry
Remove the entire `- name: ...` block from the values file.
### Step 2: Verify no other entries depend on it
Check if any other appSpec entries reference the same namespace or have dependencies on this tool.
### Step 3: Open PR
After merge, the ArgoCD Application will be deleted by the finalizer (`resources-finalizer.argocd.argoproj.io`), which will also clean up the deployed resources.
**Warning:** If the Application has `CreateNamespace=true` in syncOptions and the namespace is shared with other tools, removing the Application will NOT delete the namespace. Namespace cleanup is manual.
---
## Multi-cluster rollout
When adding a tool to multiple clusters:
1. Add the appSpec entry to each cluster's values file in the **same PR**.
2. Verify chart/values exist for **each** cluster — valuesDir names may differ per cluster.
3. Each cluster should have its own override directory in `devops-infra-helm-charts/helm-overrides/<cluster>/<valuesDir>/`.
---
## Checklist
- [ ] `chartDir` exists in `devops-infra-helm-charts/helm-templates/`
- [ ] `valuesDir` exists in `devops-infra-helm-charts/helm-overrides/<cluster>/`
- [ ] Auto-generated Application name <= 253 characters
- [ ] No name collision with existing appSpec entries
- [ ] `nameOverride` used only if justified
- [ ] `helm template` renders without errors
- [ ] `yamllint` passes
- [ ] Pre-commit hooks pass
docs/platform/procedures/deboard-tool-from-cluster.md
+118
View File
@@ -0,0 +1,118 @@
# Procedure: Deboard a Tool from a Cluster
> Step-by-step guide for safely removing an `appSpec` entry from a cluster values file.
>
> **Layer:** 1-T (Tool-Mediated). Agent generates the diff and opens a PR.
>
> **Blast radius:** Single cluster. After merge, ArgoCD deletes the Application and its resources via finalizer.
>
> Per AI Blitz Plan §5.2.
---
## Warning: deletion is destructive
Removing an `appSpec` entry causes ArgoCD to delete the child Application, which triggers the `resources-finalizer.argocd.argoproj.io` to **delete all Kubernetes resources** the Application manages. This includes Deployments, Services, ConfigMaps, PVCs, and the namespace (if `CreateNamespace=true` and the namespace is not shared).
**Confirm with the tool owner before proceeding.**
---
## Prerequisites
1. Confirm the tool is no longer needed on this cluster.
2. Confirm no other system depends on the tool's namespace (e.g., shared `victoriametrics` namespace — removing one app doesn't remove the namespace, but verify).
3. If the tool has persistent storage (PVC), confirm data can be discarded or has been backed up.
---
## Steps
### Step 1: Find the appSpec entry
```bash
grep -n -A 5 'name: <tool>' values/<env>/incubator-infra-<cluster>-values.yaml
```
Record the line numbers of the entire entry block.
### Step 2: Check for namespace sharing
```bash
# Does any other appSpec entry use the same namespace?
grep 'namespace: <namespace>' values/<env>/incubator-infra-<cluster>-values.yaml
```
If another entry shares the namespace, removing this entry will NOT delete the namespace — only the Application's resources. This is usually safe, but confirm.
### Step 3: Compute the Application name that will be deleted
```text
<name>-<mungedCluster>-<env>
```
Use this to verify the Application in ArgoCD before the PR merges.
### Step 4: Remove the appSpec entry
Delete the entire entry block from the values file:
```yaml
# Remove this block entirely:
- name: <tool>
namespace: <namespace>
chartDir: <chartDir>
valuesDir: <valuesDir>
```
Do not leave empty lines or dangling list markers.
### Step 5: Validate
```bash
helm template generic-argo-apps-chart/ \
-f values/<env>/incubator-infra-<cluster>-values.yaml
yamllint values/<env>/incubator-infra-<cluster>-values.yaml
```
Confirm the removed Application no longer appears in the rendered output.
### Step 6: Open PR
- Title: `deboard <tool> from <cluster>`
- PR body must include: what the tool was doing, why it's being removed, who confirmed the removal, and namespace cleanup plan.
- Required: Platform team review + tool owner acknowledgment
### Step 7: Post-merge cleanup
After merge and ArgoCD sync completes:
1. Verify in ArgoCD that the Application has been deleted.
2. Verify the namespace is gone (or still exists if shared — expected).
3. If `custom-values.yaml` in `devops-infra-helm-charts` is no longer needed, clean it up in a separate PR.
---
## Special case: decommissioning a cluster
If you are removing **all** tools from a cluster as part of cluster decommission:
1. Do **not** delete the incubator file in this PR — that is a separate step requiring confirmation.
2. First empty the `appSpec` list: `appSpec: []`
3. After all tools are confirmed deleted, open a second PR to delete the incubator and values files.
See escalation matrix: [docs/global/escalation-matrix.md](../../global/escalation-matrix.md) — cluster decommission requires coordinated sign-off.
---
## Checklist
- [ ] Tool owner confirmed removal
- [ ] Namespace sharing checked — no unintended namespace deletion
- [ ] Persistent storage impact assessed
- [ ] Application name identified for post-merge verification
- [ ] `helm template` renders without errors
- [ ] `yamllint` passes
- [ ] Pre-commit hooks pass
- [ ] PR body includes removal rationale and namespace cleanup plan
docs/platform/procedures/fleet-wide-tool-rollout.md
+126
View File
@@ -0,0 +1,126 @@
# Procedure: Fleet-Wide Tool Rollout
> Step-by-step guide for adding a new tool to multiple clusters in a single PR.
>
> **Layer:** 1-T (Tool-Mediated). Agent generates the diff and opens a PR.
>
> **Blast radius:** ALL targeted clusters simultaneously. Auto-sync means all clusters deploy on merge.
>
> Per AI Blitz Plan §5.2. Skill: [skills/infra/fleet-wide-rollout.md](../../../skills/infra/fleet-wide-rollout.md).
---
## When to use this procedure
- Adding an observability tool (e.g., Coroot, Pyroscope) to a set of clusters
- Rolling out a security policy tool (e.g., Kyverno) fleet-wide
- Deploying a new mandatory platform component across all production clusters
---
## Prerequisites
Before touching any values file:
1. **Chart exists** in `devops-infra-helm-charts/helm-templates/<chartDir>/`.
2. **Override values exist** for EACH target cluster: `devops-infra-helm-charts/helm-overrides/<cluster>/<valuesDir>/custom-values.yaml`. ValuesDir names may differ per cluster — verify each one.
3. Decide whether this is a **full fleet** rollout or **subset** (specific envs, BUs, or roles).
If any chart or override is missing, stop. Create them in `devops-infra-helm-charts` first.
---
## Steps
### Step 1: Identify target clusters
List all values files for the target env:
```bash
ls values/prd/
ls values/int/
```
Filter to the clusters you need. Common subsets:
| Subset | Description |
| ------ | ----------- |
| All prd | All files in `values/prd/` |
| Data clusters | `datascience`, `dataengg`, `dengspark`, `dscispark`, `dsgpu` |
| App clusters | `demand`, `supply`, `farmiso`, `central` |
| Admin cluster | `admin-prd`, `devops-admin`, `sec-admin` |
### Step 2: Verify override exists for each cluster
```bash
for cluster in <cluster-list>; do
echo -n "$cluster: "
ls /path/to/devops-infra-helm-charts/helm-overrides/$cluster/<valuesDir>/custom-values.yaml \
&& echo "ok" || echo "MISSING"
done
```
**Halt** if any cluster is missing its override. The override must be added to `devops-infra-helm-charts` first. Only proceed when all clusters are covered.
### Step 3: Compute Application names for all clusters
For each target cluster, compute `<name>-<mungedCluster>-<env>`. Verify no name collisions with existing entries.
### Step 4: Add appSpec entries to all target values files
For each cluster values file, append:
```yaml
- name: <tool>
namespace: <namespace>
chartDir: <chartDir>
valuesDir: <valuesDir>
```
The `valuesDir` value may differ per cluster if override directories are cluster-named.
### Step 5: Validate each cluster
```bash
for f in values/prd/incubator-infra-k8s-<cluster>-prd-ase1-values.yaml; do
echo "=== $f ==="
helm template generic-argo-apps-chart/ -f "$f" | grep -c "kind: Application"
yamllint "$f"
done
```
All files must render cleanly.
### Step 6: Open PR
- Title: `onboard <tool> to [all prd clusters | <subset description>]`
- PR body must include:
- Complete list of modified clusters
- The `chartDir` and `valuesDir` used
- Link to the `devops-infra-helm-charts` PR that added the chart/overrides (if applicable)
- Confirmation that all override directories exist
- Required: Platform team review
---
## Staged rollout alternative
If you want cluster-by-cluster rollout (to catch issues early):
1. Start with one low-risk cluster (e.g., `k8s-datascience-prd-ase1`).
2. Open PR 1, merge, verify in ArgoCD.
3. Open PR 2 with the remaining clusters.
Staged rollout requires multiple PRs and more human attention but reduces blast radius per deploy.
---
## Checklist
- [ ] Chart exists in `devops-infra-helm-charts/helm-templates/`
- [ ] Override exists for EVERY target cluster in `devops-infra-helm-charts/helm-overrides/<cluster>/`
- [ ] All Application names computed and verified no collisions
- [ ] `helm template` passes for every modified values file
- [ ] `yamllint` passes for every modified values file
- [ ] PR body lists all modified clusters
- [ ] Pre-commit hooks pass
docs/platform/procedures/upgrade-chart-version.md
+123
View File
@@ -0,0 +1,123 @@
# Procedure: Upgrade a Chart Version
> Step-by-step guide for changing the `chartDir` of an `appSpec` entry to reference a new chart version.
>
> **Layer:** 1-T (Tool-Mediated). Agent generates the diff and opens a PR.
>
> **Blast radius:** Single cluster (if one values file) or fleet-wide (if multiple clusters). Auto-sync means changes deploy immediately after merge to the env branch (`main` for prd, `develop` for stg, `pre-prod` for int).
>
> Per AI Blitz Plan §5.2. Skill: [skills/infra/upgrade-chart-version.md](../../../skills/infra/upgrade-chart-version.md).
---
## When to use this procedure
- Upgrading a tool to a new version by switching `chartDir` (e.g., `contour` → `contour-v1.33.3`)
- Pinning a tool back to a previous chart directory
- Upgrading a tool across multiple clusters as part of a coordinated rollout
**Do not** use this procedure to change `teamSpec.source.targetRevision` — that's a HIGH RISK operation requiring explicit platform-team sign-off.
---
## Prerequisites
1. The **new chart directory** exists in `devops-infra-helm-charts/helm-templates/<newChartDir>/`.
2. The **override values** for the new chart version are compatible — check `devops-infra-helm-charts/helm-overrides/<cluster>/<valuesDir>/custom-values.yaml` for any new required fields.
3. You know which clusters need upgrading.
---
## Single-cluster upgrade
### Step 1: Identify the target appSpec entry
Find the tool in the cluster's values file:
```bash
grep -A 4 'name: <tool>' values/<env>/incubator-infra-<cluster>-values.yaml
```
Note the current `chartDir`.
### Step 2: Verify the new chart directory exists
```bash
ls /path/to/devops-infra-helm-charts/helm-templates/<newChartDir>/
```
**Halt** if missing — the chart must be added to `devops-infra-helm-charts` first.
### Step 3: Check override compatibility
```bash
# Review the custom-values.yaml for any changed keys in the new chart
cat /path/to/devops-infra-helm-charts/helm-overrides/<cluster>/<valuesDir>/custom-values.yaml
```
If the new chart has breaking changes (renamed keys, removed defaults), the `custom-values.yaml` in `devops-infra-helm-charts` must be updated first — that is a separate PR in the sister repo.
### Step 4: Update the `chartDir`
Change the `chartDir` field in the values file:
```yaml
# Before:
- name: contour-internal-0
chartDir: contour
valuesDir: contour-internal
# After:
- name: contour-internal-0
chartDir: contour-v1.33.3
valuesDir: contour-internal
```
Do not change `name`, `namespace`, or `valuesDir` unless the upgrade specifically requires it.
### Step 5: Validate
```bash
helm template generic-argo-apps-chart/ \
-f values/<env>/incubator-infra-<cluster>-values.yaml \
| grep -B 2 -A 30 "name: <tool>"
yamllint values/<env>/incubator-infra-<cluster>-values.yaml
```
Verify the rendered Application points to the new chart path.
### Step 6: Open PR
- Title: `upgrade <tool> to <newChartDir> on <cluster>`
- Required: Platform team review
- Pre-commit hooks must pass
---
## Multi-cluster upgrade
When upgrading the same tool across multiple clusters:
1. Update `chartDir` in **all affected values files in the same PR**.
2. Validate each cluster's values file independently with `helm template`.
3. Title: `upgrade <tool> to <newChartDir> across [list of clusters or "all prd clusters"]`
**Ordering note:** ArgoCD auto-syncs all clusters on merge. There is no cluster-by-cluster rollout order. If you need a staged rollout (one cluster at a time), open separate PRs.
---
## Rollback
If the upgrade causes issues, create a revert PR changing `chartDir` back to the previous value. Do not force-push.
---
## Checklist
- [ ] New `chartDir` exists in `devops-infra-helm-charts/helm-templates/`
- [ ] `custom-values.yaml` is compatible with the new chart version
- [ ] `helm template` renders without errors for each updated cluster
- [ ] `yamllint` passes
- [ ] Pre-commit hooks pass
- [ ] For multi-cluster: all clusters validated before PR opens
docs/platform/runbooks/argocd-sync-failure.md
+170
View File
@@ -0,0 +1,170 @@
# Runbook: ArgoCD Infra App Sync Failure
> Decision tree for diagnosing and resolving sync failures on infrastructure tool Applications managed by this repo.
>
> **Audience:** Platform engineers, on-call SREs, and agents in advisory mode (Layer 2).
>
> **Key context:** Infra apps auto-sync from `main`. A sync failure means either a bad merge or an external cluster issue.
---
## Triage — is it this repo's fault?
```
Sync failure observed
│
├─ Is the Application in "Unknown" or "Missing" state?
│ └─ YES → The incubator file may be missing or malformed.
│ Check: incubator/<env>/<cluster>.yaml exists and is valid YAML.
│ Check: The incubator Application itself is healthy in the admin cluster.
│
├─ Is the error "helm template failed" or "render error"?
│ └─ YES → Chart or values problem. Go to Section 1.
│
├─ Is the error "namespace not found" or "destination not found"?
│ └─ YES → Cluster routing problem. Go to Section 2.
│
├─ Is the error "ComparisonError" or "already exists"?
│ └─ YES → Name collision. Go to Section 3.
│
└─ Is the error a Kubernetes API error (forbidden, quota, etc.)?
└─ YES → Cluster-side issue. Go to Section 4.
```
---
## Section 1: Helm render failure
**Symptom:** ArgoCD shows "helm template failed" or similar render error.
**Diagnosis:**
```bash
# Reproduce locally
helm template generic-argo-apps-chart/ \
-f values/<env>/<cluster>-values.yaml
# Check for YAML syntax errors
yamllint values/<env>/<cluster>-values.yaml
```
**Common causes:**
| Cause | Fix |
| ----- | --- |
| `chartDir` doesn't exist in `devops-infra-helm-charts/helm-templates/` | Create the chart directory in the sister repo, or fix the `chartDir` value |
| `valuesDir` doesn't exist in `devops-infra-helm-charts/helm-overrides/<cluster>/` | Create the values override, or fix the `valuesDir` value |
| YAML syntax error in values file | Fix the YAML (missing colon, bad indentation, etc.) |
| `additionalValueFiles` path doesn't exist | Fix the path or remove the entry |
| Helm chart has a breaking change | Check recent commits in `devops-infra-helm-charts` |
---
## Section 2: Cluster routing / namespace issue
**Symptom:** "destination cluster not found" or "namespace not found."
**Diagnosis:**
```bash
# Check the cluster name in the values file
grep -A2 "clusterSpec:" values/<env>/<cluster>-values.yaml
# Verify the cluster is registered in ArgoCD
argocd cluster list | grep <cluster-name>
```
**Common causes:**
| Cause | Fix |
| ----- | --- |
| `clusterSpec.destination.name` doesn't match GKE cluster name | Fix the name in the values file |
| Cluster was recently provisioned but not yet registered in ArgoCD | Register the cluster via ArgoCD CLI or Terraform |
| Cluster was decommissioned | Remove the incubator + values files |
---
## Section 3: Application name collision
**Symptom:** "already exists" or "ComparisonError" for an Application.
**Diagnosis:**
```bash
# Check for duplicate names in the values file
grep '^\s*- name:' values/<env>/<cluster>-values.yaml | sort | uniq -d
# Check for nameOverride collisions
grep 'nameOverride:' values/<env>/<cluster>-values.yaml
```
**Common causes:**
| Cause | Fix |
| ----- | --- |
| Two appSpec entries have the same `name` | Rename one or add `nameOverride` |
| An Application with the same name exists from a different source | Use `nameOverride` to disambiguate |
| Name was changed but old Application wasn't cleaned up | Delete the orphaned Application via ArgoCD CLI |
---
## Section 4: Cluster-side issue
**Symptom:** Kubernetes API errors — forbidden, quota exceeded, node selector mismatch, etc.
**Diagnosis:** This is not a repo-side issue. The chart and values are correct, but the cluster can't fulfill the request.
**Common causes:**
| Cause | Fix |
| ----- | --- |
| Namespace quota exceeded | Request quota increase or reduce resource requests |
| Node selector doesn't match any node | Verify nodepool configuration in Terraform |
| RBAC / service account permissions | Check the AppProject scope and cluster RBAC |
| CRDs not installed | Install required CRDs before deploying the tool |
---
## Section 5: Tool deployed to wrong cluster or namespace
**Symptom:** A tool appears in an unexpected cluster or namespace.
**Diagnosis:**
```bash
# Check where the tool is configured
grep -rl 'name: <tool>' values/
# Verify the values file's cluster destination
grep -A2 "clusterSpec:" values/<env>/<cluster>-values.yaml
# Verify the namespace in the appSpec entry
grep -A4 'name: <tool>' values/<env>/<cluster>-values.yaml
```
**Common causes:**
| Cause | Fix |
| ----- | --- |
| appSpec entry added to wrong values file | Move to correct cluster's values file |
| `clusterSpec.destination.name` is wrong | Fix the cluster name |
| Namespace typo | Fix the `namespace` field in the appSpec entry |
---
## Emergency: revert a bad merge
If a bad merge causes widespread sync failures:
1. **Do NOT force-push to `main`.** This violates R1 and R11.
2. Open a revert PR: `git revert <bad-commit> && git push origin revert-branch`
3. Get expedited platform-team review and merge the revert.
4. Auto-sync will pick up the revert within minutes.
---
## Escalation
If the above doesn't resolve the issue:
- **Platform team Slack:** Post in `#devops-tech` with the Application name, cluster, and error message.
- **ArgoCD admin UI:** Access via the admin cluster to inspect Application state directly.
docs/platform/runbooks/deployment-stuck.md
+138
View File
@@ -0,0 +1,138 @@
# Runbook: Deployment Stuck After Sync
> Symptom → diagnosis → remediation for tools that ArgoCD reports as `Synced` but whose Kubernetes resources are not healthy.
>
> **Layer:** 2 (Advisory). Agents diagnose and suggest; humans execute kubectl/ArgoCD commands.
>
> Per AI Blitz Plan §5.2. See also: [argocd-sync-failure.md](argocd-sync-failure.md) (for sync errors), [render-failure.md](render-failure.md) (for Helm render errors).
---
## Symptoms
- ArgoCD Application shows `Synced` but `Degraded` or `Unknown` health
- Pods are in `Pending`, `CrashLoopBackOff`, `ImagePullBackOff`, or `OOMKilled`
- Tool was working before a chart upgrade or values change
- Application is `Synced + Healthy` but the tool isn't functioning as expected
---
## Decision tree
```text
Application shows Synced but Degraded?
│
├── Check pod status in the tool's namespace:
│ kubectl get pods -n <namespace> --context=<cluster>
│
├── Pod status is Pending?
│ └── → [A] Resource constraints or node issues
│
├── Pod status is CrashLoopBackOff?
│ └── → [B] Application crash — check logs
│
├── Pod status is ImagePullBackOff or ErrImagePull?
│ └── → [C] Image registry issue
│
├── Pod status is OOMKilled?
│ └── → [D] Memory limit too low
│
├── Pod runs but tool is unhealthy?
│ └── → [E] Config or connectivity issue
│
└── Pods are fine but ArgoCD shows Degraded?
└── → [F] Health check misconfiguration
```
---
## [A] Pod Pending
**Diagnosis:**
```bash
kubectl describe pod <pod-name> -n <namespace> --context=<cluster> | grep -A 20 Events
```
Common causes and fixes:
| Event message | Cause | Fix |
| ------------- | ----- | --- |
| `Insufficient cpu/memory` | Node resource exhaustion | Check node capacity; may need to adjust resource requests in `custom-values.yaml` (change in `devops-infra-helm-charts`) |
| `did not match node affinity` | Node selector or affinity mismatch | Review `nodeSelector`/`affinity` in `custom-values.yaml` |
| `PersistentVolumeClaim not bound` | PVC not provisioned | Check storage class and PVC events |
| `Unschedulable` | No nodes available | Check if cluster has sufficient nodes or if Karpenter/cluster-autoscaler is stuck |
---
## [B] CrashLoopBackOff
**Diagnosis:**
```bash
kubectl logs <pod-name> -n <namespace> --context=<cluster> --previous
```
Look for startup errors, missing config, failed connections. The fix usually requires changing `custom-values.yaml` in `devops-infra-helm-charts` — that's a separate PR in the sister repo.
---
## [C] ImagePullBackOff
**Diagnosis:**
```bash
kubectl describe pod <pod-name> -n <namespace> --context=<cluster> | grep image
```
Common causes:
- Image tag doesn't exist (bad chart version pin)
- Image registry credentials expired (check `external-secrets` or `imagePullSecrets`)
- Private registry unreachable from the cluster
Fix depends on root cause. Usually requires a chart fix in `devops-infra-helm-charts` or a credential rotation.
---
## [D] OOMKilled
**Diagnosis:**
```bash
kubectl top pod -n <namespace> --context=<cluster>
kubectl describe pod <pod-name> -n <namespace> --context=<cluster> | grep -A 5 "Last State"
```
Fix: Increase `resources.limits.memory` in `custom-values.yaml` in `devops-infra-helm-charts`. Open PR there.
---
## [E] Config or Connectivity Issue
Tool is running but not working correctly.
**Diagnosis:** Check tool-specific logs. Most infra tools write structured logs.
Common causes:
- Wrong endpoint URL in `custom-values.yaml`
- Service dependency not available (e.g., Vault unreachable for external-secrets)
- Wrong namespace for a referenced service
Fix: Update `custom-values.yaml` in `devops-infra-helm-charts`.
---
## [F] ArgoCD Health Check Misconfiguration
ArgoCD uses health checks to determine if an Application is healthy. Some custom resources have incorrect or missing health checks.
**Diagnosis:**
```bash
argocd app get <appName> --show-operation | grep -A 10 "Health Status"
```
If resources are actually healthy but ArgoCD says Degraded, this may be a health check definition issue in the ArgoCD config itself (not this repo).
Escalate to platform team via [escalation-matrix.md](../../global/escalation-matrix.md).
---
## Escalation
If diagnosis points to a cluster-level issue (node pressure, networking, GKE problem) rather than a configuration issue, escalate via [escalation-matrix.md](../../global/escalation-matrix.md).
docs/platform/runbooks/render-failure.md
+133
View File
@@ -0,0 +1,133 @@
# Runbook: Helm Render Failure
> Symptom → diagnosis → remediation for ArgoCD Applications stuck in `ComparisonError` or `ErrParsingAppProject` state due to Helm template rendering failures.
>
> **Layer:** 2 (Advisory). Agents diagnose and suggest fixes; humans apply.
>
> Per AI Blitz Plan §5.2. See also: [argocd-sync-failure.md](argocd-sync-failure.md).
---
## Symptoms
- ArgoCD Application shows `ComparisonError: failed to generate manifest`
- `helm template` run locally produces YAML errors
- Application stuck in `Unknown` or `Error` health state with no sync progress
- Pre-commit hook `cac validate` fails on a PR with YAML parsing errors
---
## Decision tree
```text
Application stuck in ComparisonError?
│
├── Run: argocd app get <appName> --show-operation
│ (or: argocd app logs <appName>)
│
├── Error contains "values file not found"?
│ └── → [A] Missing values file
│
├── Error contains "chart not found" or "no chart found"?
│ └── → [B] Missing chart directory
│
├── Error contains "unmarshal" or "cannot unmarshal" or "yaml: line"?
│ └── → [C] YAML syntax error in values or chart
│
├── Error contains "template: ... nil pointer" or "function not defined"?
│ └── → [D] Chart template bug
│
└── Error contains "AppProject not found" or "unauthorized"?
└── → See argocd-sync-failure.md
```
---
## [A] Missing values file
**Cause:** The `valueFiles` path in the incubator Application YAML points to a values file that doesn't exist.
**Diagnosis:**
```bash
# Check what valueFiles the incubator references:
grep 'valueFiles' incubator/<env>/<cluster>.yaml
# Verify the values file exists:
ls values/<env>/incubator-infra-<cluster>-values.yaml
```
**Fix:** Create the missing values file (use [add-new-cluster.md](../procedures/add-new-cluster.md) for a new cluster) or correct the path in the incubator YAML. Open a PR.
---
## [B] Missing chart directory
**Cause:** An `appSpec[].chartDir` value points to a directory that doesn't exist in `devops-infra-helm-charts/helm-templates/`.
**Diagnosis:**
```bash
# Identify which appSpec entry has the bad chartDir:
helm template generic-argo-apps-chart/ -f values/<env>/<cluster>-values.yaml 2>&1 | head -30
# Verify in helm-charts repo:
ls /path/to/devops-infra-helm-charts/helm-templates/<chartDir>/
```
**Fix:** Either:
1. Add the chart to `devops-infra-helm-charts` (work in the sister repo).
2. Correct the `chartDir` in the values file to an existing directory.
Open a PR with the fix.
---
## [C] YAML syntax error
**Cause:** Invalid YAML in a values file (e.g., bad indentation, missing quotes, stray character).
**Diagnosis:**
```bash
yamllint values/<env>/<cluster>-values.yaml
# Or for more context:
python3 -c "import yaml; yaml.safe_load(open('values/<env>/<cluster>-values.yaml'))"
```
**Common causes:**
- Stray tab character (YAML requires spaces)
- Missing `- ` prefix on an appSpec entry
- Unquoted special characters in namespace or name fields
- Trailing whitespace after a value
**Fix:** Correct the YAML syntax. Run `yamllint` and `helm template` locally before opening PR. Pre-commit `cac validate` hook should also catch this.
---
## [D] Chart template bug
**Cause:** The `generic-argo-apps-chart/templates/genericTemplate.yaml` has a nil pointer dereference, undefined function, or logic error.
**Diagnosis:**
```bash
helm template generic-argo-apps-chart/ -f values/<env>/<cluster>-values.yaml 2>&1
```
**Impact:** This breaks ALL Applications rendered by this chart, across ALL clusters. This is a fleet-wide outage risk.
**Fix:** This requires a fix to `generic-argo-apps-chart/`. This is a **HIGH RISK** operation — see [AGENT_BOUNDARIES.md](../../global/AGENT_BOUNDARIES.md). Escalate to platform team immediately via [escalation-matrix.md](../../global/escalation-matrix.md).
Do not attempt to fix chart templates without explicit platform-team sign-off and test with `helm template` against multiple values files.
---
## Preventive checks
Run before any PR that modifies values files or `generic-argo-apps-chart/`:
```bash
# Test all values files in one env:
for f in values/prd/*.yaml; do
helm template generic-argo-apps-chart/ -f "$f" > /dev/null \
&& echo "OK: $f" || echo "FAIL: $f"
done
```
docs/platform/runbooks/values-drift.md
+116
View File
@@ -0,0 +1,116 @@
# Runbook: Values Drift
> Symptom → diagnosis → remediation for inconsistent `appSpec` entries across cluster values files — same tool, different `chartDir`, `namespace`, or field values across clusters.
>
> **Layer:** 2 (Advisory). Agents diagnose and report; humans decide whether to remediate.
>
> Per AI Blitz Plan §5.2.
---
## Symptoms
- A tool behaves differently on two clusters that should be equivalent
- A chart version upgrade was applied to some clusters but not others
- `grep` across values files shows the same tool with different `chartDir` or `namespace`
- After a fleet-wide rollout, one cluster was accidentally skipped
---
## Detection
### Find clusters where a tool exists
```bash
grep -rl 'name: <tool>' values/prd/
```
### Find the chartDir each cluster uses for a tool
```bash
grep -A 4 'name: <tool>' values/prd/*.yaml | grep 'chartDir'
```
### Compare a specific tool across all prd clusters
```bash
# Show full appSpec entry for the tool on every cluster that has it:
for f in values/prd/*.yaml; do
entry=$(grep -A 5 "^\s*- name: <tool>$" "$f" 2>/dev/null)
if [ -n "$entry" ]; then
echo "=== $f ==="
echo "$entry"
fi
done
```
### Find clusters missing a tool that should be present
```bash
# List all prd values files:
all=$(ls values/prd/)
# List files that have the tool:
have=$(grep -rl 'name: <tool>' values/prd/ | xargs -I{} basename {})
# Diff:
diff <(echo "$all") <(echo "$have")
```
---
## Interpreting drift
Not all drift is wrong. Some clusters legitimately have different configurations:
| Type of drift | Usually OK? | Action |
| ------------- | ----------- | ------ |
| Different `chartDir` (different version) | Only if intentional | Align versions unless the divergence is intentional (e.g., staged rollout in progress) |
| Different `namespace` | Rarely OK | Investigate — namespace change is destructive |
| Tool absent from a cluster | Depends | Confirm whether the cluster should have the tool |
| Different `valuesDir` | Sometimes OK | Clusters can have cluster-specific overrides; confirm the override exists |
| Different `nameOverride` | Rarely OK | Should only exist for collision/length reasons |
---
## Remediation
### Align chart versions across clusters
If a tool is on different versions and should be uniform, follow [upgrade-chart-version.md](../procedures/upgrade-chart-version.md) for the lagging clusters.
### Add a missing tool to a cluster
Follow [add-tool-to-cluster.md](../procedures/add-tool-to-cluster.md).
### Document intentional drift
If the divergence is intentional (e.g., Cluster A has a special `valuesDir` for a regional override), add a comment to the values file entry:
```yaml
- name: coredns
namespace: kube-system
chartDir: coredns
valuesDir: coredns-special-region # Intentional: this cluster uses a non-standard DNS config
additionalValueFiles:
- ../../helm-templates/coredns/gcp-ase1a-values.yaml
```
---
## Periodic drift check
Run this to generate a drift report across all prd clusters for a tool:
```bash
echo "Tool: <tool>" && echo "---"
for f in values/prd/*.yaml; do
cluster=$(basename "$f" -values.yaml)
entry=$(grep -A 5 "^\s*- name: <tool>$" "$f" 2>/dev/null)
if [ -n "$entry" ]; then
chartDir=$(echo "$entry" | grep 'chartDir' | awk '{print $2}')
namespace=$(echo "$entry" | grep 'namespace' | awk '{print $2}')
echo "$cluster: chartDir=$chartDir namespace=$namespace"
else
echo "$cluster: ABSENT"
fi
done
```
docs/platform/schemas/incubator-values-schema.md
+119
View File
@@ -0,0 +1,119 @@
# Incubator Application YAML Schema
> Field-by-field documentation of the incubator Application YAML files.
>
> **File location:** Varies by env/branch:
> - prd (`main`): `incubator/prd/incubator-infra-<cluster>.yaml`
> - stg/dev (`develop`): `incubator/infra/incubator-infra-<cluster>.yaml` (or `incubator/apps/incubator-apps-<cluster>.yaml` for app workloads)
> - int (`pre-prod`): no incubator file — values only in `values/int/`
> - admin: `incubator/admin/incubator-infra-<cluster>.yaml`
>
> **Purpose:** Each incubator file is a single ArgoCD `Application` that acts as the **parent** in the App-of-Applications pattern. It points at `generic-argo-apps-chart/` and its corresponding values file. ArgoCD renders the generic chart to produce one child Application per `appSpec` entry.
---
## Full example
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: incubator-infra-k8s-central-prd-ase1
namespace: argocd-prd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: 'https://github.com/Meesho/devops-infra-argo-config'
targetRevision: main # prd=main, stg=develop, int=pre-prod
path: generic-argo-apps-chart
helm:
valueFiles:
- ../values/prd/incubator-infra-k8s-central-prd-ase1-values.yaml
destination:
name: in-cluster
namespace: argocd-prd
```
---
## `metadata`
| Field | Type | Required | Editable | Notes |
| ----- | ---- | -------- | -------- | ----- |
| `name` | string | yes | no | Must match the filename without `.yaml`. Convention: `incubator-infra-<gke-cluster-name>`. This name is an identifier — changing it breaks ArgoCD tracking. |
| `namespace` | string | yes | no | Env-specific: `argocd-prd` (prd), `argocd-dev` (stg/dev), `argocd-shared-int` (int). This is the namespace in the **admin** cluster where ArgoCD runs, not the target cluster. |
| `finalizers[0]` | string | yes | no | Always `resources-finalizer.argocd.argoproj.io`. Ensures ArgoCD deletes child Applications when the parent is deleted. Do not remove. |
### Filename ↔ name invariant (R9)
The incubator filename (without `.yaml`) **must equal** `metadata.name`. These are routing keys used by ArgoCD:
```text
prd: incubator/prd/incubator-infra-k8s-central-prd-ase1.yaml
└── metadata.name: incubator-infra-k8s-central-prd-ase1
stg: incubator/infra/incubator-infra-k8s-central-stg-ase1.yaml
└── metadata.name: incubator-infra-k8s-central-stg-ase1
```
---
## `spec.project`
| Field | Type | Required | Editable | Notes |
| ----- | ---- | -------- | -------- | ----- |
| `project` | string | yes | no | Always `default` for incubator Applications. The child Applications use `sre` or `sec` projects. |
---
## `spec.source`
| Field | Type | Required | Editable | Notes |
| ----- | ---- | -------- | -------- | ----- |
| `repoURL` | string | yes | no | Always `https://github.com/Meesho/devops-infra-argo-config`. Changing this is a Layer 3 operation. |
| `targetRevision` | string | yes | no | Environment-specific: `main` (prd), `develop` (stg), `pre-prod` (int). Changing to any other value requires sign-off. |
| `path` | string | yes | no | Always `generic-argo-apps-chart`. Points to the Helm chart in this repo. |
| `helm.valueFiles[0]` | string | yes | yes | Relative path from the chart root to the cluster's values file. Convention: `../values/<env>/incubator-infra-<cluster>-values.yaml`. The `..` is required because ArgoCD resolves paths relative to `path` (i.e., relative to `generic-argo-apps-chart/`). |
### `valueFiles` path convention
The path is relative to the chart directory (`generic-argo-apps-chart/`), so you go up one level to reach the repo root:
```text
generic-argo-apps-chart/ (chart root)
└── ../values/prd/incubator-infra-k8s-central-prd-ase1-values.yaml
└── resolves to: values/prd/incubator-infra-k8s-central-prd-ase1-values.yaml
```
---
## `spec.destination`
| Field | Type | Required | Editable | Notes |
| ----- | ---- | -------- | -------- | ----- |
| `name` | string | yes | no | Always `in-cluster`. Incubator Applications run on the **admin cluster** where ArgoCD is installed, not on the target cluster. Child Applications (from `appSpec`) deploy to the target cluster. |
| `namespace` | string | yes | no | Env-specific: `argocd-prd` (prd), `argocd-dev` (stg/dev), `argocd-shared-int` (int). This is where ArgoCD creates the child Application objects. |
---
## What this file controls vs. what it doesn't
| Controlled by this file | Controlled by the values file |
| ----------------------- | ----------------------------- |
| Which generic chart to use | Which tools deploy (`appSpec[]`) |
| Which values file to use | Target cluster name (`clusterSpec`) |
| ArgoCD namespace | Chart source repo and revision (`teamSpec`) |
| Parent Application name | Per-tool namespace, chartDir, valuesDir |
---
## Files that must exist when this file is created
| File | Where | Why |
| ---- | ----- | --- |
| `values/<env>/incubator-infra-<cluster>-values.yaml` | This repo | Referenced by `helm.valueFiles` |
| `generic-argo-apps-chart/` | This repo | The chart being rendered |
See also: [values-file-schema.md](values-file-schema.md) for the values file structure, and [add-new-cluster.md](../procedures/add-new-cluster.md) for the full cluster onboarding procedure.
docs/platform/schemas/values-file-schema.md
+171
View File
@@ -0,0 +1,171 @@
# Values File Schema — Annotated Field Reference
> Field-by-field documentation of the cluster values files used by the generic ArgoCD chart.
>
> **File location:** `values/<env>/incubator-infra-<cluster>-values.yaml`
>
> **Consumer:** `generic-argo-apps-chart/templates/genericTemplate.yaml` reads these values to render one ArgoCD `Application` per `appSpec` entry.
---
## Top-level structure
```yaml
clusterSpec: # REQUIRED — cluster identity and routing
argocdSpec: # REQUIRED — ArgoCD operational config
teamSpec: # REQUIRED — source repo, labels, team identity
appSpec: # REQUIRED — list of tools to deploy (may be empty [])
```
---
## `clusterSpec`
```yaml
clusterSpec:
destination:
server: "" # String. Always empty — name-based routing is standard.
name: "k8s-central-prd-ase1" # String. REQUIRED. GKE cluster name.
# Must match: incubator file name, helm-overrides/<cluster>/ folder,
# and teamSpec.devops.labels.cluster.
```
| Field | Type | Required | Editable | Notes |
| ----- | ---- | -------- | -------- | ----- |
| `destination.server` | string | yes | no | Always `""`. Name-based routing is used instead. |
| `destination.name` | string | yes | no | GKE cluster name. Changing this redirects ALL tools — HIGH RISK. |
---
## `argocdSpec`
```yaml
argocdSpec:
namespace: argocd-prd # String. ArgoCD namespace where Applications are created.
# prd → argocd-prd | stg/dev → argocd-dev | int → argocd-shared-int
```
| Field | Type | Required | Editable | Notes |
| ----- | ---- | -------- | -------- | ----- |
| `namespace` | string | yes | no | Env-specific: `argocd-prd` (prd), `argocd-dev` (stg/dev), `argocd-shared-int` (int). Changing is HIGH RISK. |
---
## `teamSpec`
```yaml
teamSpec:
devops: # Object. Team block — always "devops" for infra tooling.
source:
repoURL: https://github.com/Meesho/devops-infra-helm-charts # String. Helm charts source.
targetRevision: main # String. Git ref for chart source in devops-infra-helm-charts.
# prd → main | stg/dev → develop | int → pre-prod
path: helm-templates # String. Root path for charts in the source repo.
valueFiles: ../../helm-overrides/k8s-central-prd-ase1 # String. Relative path to overrides.
labels:
bu: infra # String. Business unit. Always "infra" for this repo.
team: devops # String. Maps to ArgoCD AppProject name.
env: prd # String. Environment: prd, int, dev, admin.
cluster: k8s-central-prd-ase1 # String. Must match clusterSpec.destination.name.
```
| Field | Type | Required | Editable | Notes |
| ----- | ---- | -------- | -------- | ----- |
| `source.repoURL` | string | yes | no | Chart source repo. Changing this is HIGH RISK. |
| `source.targetRevision` | string | yes | no | Env-specific: `main` (prd), `develop` (stg/dev), `pre-prod` (int). Changing to any other value is HIGH RISK. |
| `source.path` | string | yes | no | Always `helm-templates`. |
| `source.valueFiles` | string | yes | no | Relative path from chart to `helm-overrides/<cluster>`. |
| `labels.bu` | string | yes | no | Always `infra`. |
| `labels.team` | string | yes | no | Maps to AppProject. Usually `devops` → `sre` project. |
| `labels.env` | string | yes | no | Matches cluster environment. |
| `labels.cluster` | string | yes | no | Must match `clusterSpec.destination.name`. Used in Application name munging. |
---
## `appSpec` (the main editing surface)
```yaml
appSpec:
- name: keda # String. REQUIRED. Short tool name.
namespace: keda-central-prd # String. REQUIRED. Target Kubernetes namespace.
chartDir: keda # String. REQUIRED. Chart directory in helm-templates/.
valuesDir: keda # String. REQUIRED. Override directory in helm-overrides/<cluster>/.
nameOverride: "" # String. OPTIONAL. Custom Application name.
additionalValueFiles: # List[String]. OPTIONAL. Extra values files.
- ../../helm-templates/coredns/gcp-ase1a-values.yaml
```
### Field reference
| Field | Type | Required | Editable | Notes |
| ----- | ---- | -------- | -------- | ----- |
| `name` | string | yes | yes | Short tool identifier. Used in auto-generated Application name: `<name>-<mungedCluster>-<env>`. Lowercase, hyphen-separated. |
| `namespace` | string | yes | yes | Target K8s namespace. Auto-created by ArgoCD (`CreateNamespace=true`). Convention: `<tool>-<mungedCluster>` or shared namespace. |
| `chartDir` | string | yes | yes | Directory under `devops-infra-helm-charts/helm-templates/`. **Must exist.** |
| `valuesDir` | string | yes | yes | Directory under `devops-infra-helm-charts/helm-overrides/<cluster>/`. **Must exist** and contain `custom-values.yaml`. |
| `nameOverride` | string | no | yes | Overrides the auto-generated Application name. Use only for collisions or >253 char names. See R7. |
| `additionalValueFiles` | list | no | yes | Extra Helm values files merged during render. Paths are relative to the chart source. |
### How `appSpec` entries become ArgoCD Applications
The generic chart template (`genericTemplate.yaml`) iterates over `appSpec` and for each entry:
1. **Computes the Application name:** `<name>-<mungedCluster>-<env>` (or `nameOverride` if set).
2. **Sets the source:** `repoURL` + `path/<chartDir>` from `teamSpec.devops.source`.
3. **Sets the values:** `valueFiles/<valuesDir>/custom-values.yaml` + any `additionalValueFiles`.
4. **Sets the destination:** `clusterSpec.destination.name` + `namespace`.
5. **Sets the project:** from `teamSpec.devops.labels.team` (maps to AppProject).
6. **Sets syncPolicy:** `CreateNamespace=true`.
7. **Adds finalizer:** `resources-finalizer.argocd.argoproj.io`.
### Rendered Application example
For `name: keda` on `k8s-central-prd-ase1`:
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: keda-central-prd
namespace: argocd-prd
labels:
bu: infra
team: devops
env: prd
cluster: k8s-central-prd-ase1
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: devops
source:
repoURL: https://github.com/Meesho/devops-infra-helm-charts
targetRevision: main
path: helm-templates/keda
helm:
valueFiles:
- ../../helm-overrides/k8s-central-prd-ase1/keda/custom-values.yaml
destination:
name: k8s-central-prd-ase1
server: ""
namespace: keda-central-prd
syncPolicy:
syncOptions:
- CreateNamespace=true
```
---
## Cluster name munging rules
Used by the generic template to produce short Application names:
| Step | Transformation | Example |
| ---- | -------------- | ------- |
| 1 | Preserve `dp-` and `backup` (placeholder swap) | `dp-foo` stays `dp-foo` |
| 2 | Strip `p-`, `prd-`, `int-`, `dev-`, `-cluster` | `k8s-central-prd-ase1` → `k8s-central-ase1` |
| 3 | Replace `prod-ops` → `infra` | — |
| 4 | Replace `-ase1c` → `-c` | `k8s-central-prd-ase1c` → `central-c` |
| 5 | Strip `-ase1` | `k8s-central-ase1` → `k8s-central` |
| 6 | Strip `k8s-` | `k8s-central` → `central` |
**Net result:** `k8s-central-prd-ase1` → `central-prd`