added repo
This commit is contained in:
Your Name
@@ -0,0 +1,168 @@
|
||||
# Procedure: Onboard a New Cluster
|
||||
|
||||
> Step-by-step guide for adding a new Kubernetes cluster to the ArgoCD infrastructure GitOps control plane.
|
||||
>
|
||||
> **Layer:** 1-T (Tool-Mediated). Two files must be created: one incubator file and one values file.
|
||||
>
|
||||
> **Blast radius:** New cluster only. Existing clusters are unaffected.
|
||||
>
|
||||
> **Prerequisite:** The GKE cluster must already be provisioned via `terraform-google-modules` and registered as an ArgoCD cluster destination.
|
||||
|
||||
---
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Example | Where it comes from |
|
||||
| ----- | ------- | ------------------- |
|
||||
| Cluster name | `k8s-dsgpu-prd-ase1` | GKE cluster provisioning (Terraform) |
|
||||
| Environment | `prd` | Cluster naming convention |
|
||||
| ArgoCD namespace | See table below | Depends on environment |
|
||||
| Target branch | See table below | Depends on environment |
|
||||
| Initial tools | `keda`, `contour`, `external-secrets`, etc. | Platform team decision |
|
||||
|
||||
**Environment → branch / ArgoCD namespace mapping:**
|
||||
|
||||
| Environment | Branch | ArgoCD namespace | Values dir |
|
||||
| ----------- | ------ | ---------------- | ---------- |
|
||||
| prd | `main` | `argocd-prd` | `values/prd/` |
|
||||
| stg / dev | `develop` | `argocd-dev` | `values/dev/` |
|
||||
| int | `pre-prod` | `argocd-shared-int` | `values/int/` |
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Create the incubator file
|
||||
|
||||
Create `incubator/<env>/<cluster>.yaml`:
|
||||
|
||||
```yaml
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: incubator-infra-<cluster>
|
||||
namespace: <argocd-namespace> # argocd-prd | argocd-dev | argocd-shared-int
|
||||
finalizers:
|
||||
- resources-finalizer.argocd.argoproj.io
|
||||
spec:
|
||||
project: default
|
||||
source:
|
||||
repoURL: 'https://github.com/Meesho/devops-infra-argo-config'
|
||||
targetRevision: <branch> # main (prd) | develop (stg/dev) | pre-prod (int)
|
||||
path: generic-argo-apps-chart
|
||||
helm:
|
||||
valueFiles:
|
||||
- ../values/<env>/incubator-infra-<cluster>-values.yaml
|
||||
destination:
|
||||
name: in-cluster
|
||||
namespace: <argocd-namespace> # argocd-prd | argocd-dev | argocd-shared-int
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
- `metadata.name`: `incubator-infra-<cluster>` (matches the file name minus `.yaml`).
|
||||
- `spec.project`: `default` (the incubator itself runs in the ArgoCD admin cluster).
|
||||
- `spec.source.path`: Always `generic-argo-apps-chart`.
|
||||
- `helm.valueFiles`: Relative path to the values file from the chart directory.
|
||||
- `spec.destination.name`: `in-cluster` (the incubator runs on the ArgoCD admin cluster, not the target cluster).
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Create the values file
|
||||
|
||||
Create `values/<env>/incubator-infra-<cluster>-values.yaml`:
|
||||
|
||||
```yaml
|
||||
clusterSpec:
|
||||
destination:
|
||||
server: ""
|
||||
name: "<cluster>"
|
||||
|
||||
argocdSpec:
|
||||
namespace: <argocd-namespace> # argocd-prd | argocd-dev | argocd-shared-int
|
||||
|
||||
teamSpec:
|
||||
devops:
|
||||
source:
|
||||
repoURL: https://github.com/Meesho/devops-infra-helm-charts
|
||||
targetRevision: <branch> # main (prd) | develop (stg/dev) | pre-prod (int)
|
||||
path: helm-templates
|
||||
valueFiles: ../../helm-overrides/<cluster>
|
||||
labels:
|
||||
bu: infra
|
||||
team: devops
|
||||
env: <env>
|
||||
cluster: <cluster>
|
||||
|
||||
appSpec: []
|
||||
```
|
||||
|
||||
Start with an empty `appSpec` list. Add tools in a follow-up PR after the cluster is bootstrapped.
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Register the cluster in the admin cluster's values file
|
||||
|
||||
The admin cluster (`k8s-admin-prd-ase1`) may need an entry to reference the new cluster's ArgoCD Application. Check if the admin values file at `values/prd/incubator-infra-k8s-admin-prd-ase1-values.yaml` needs an `appSpec` entry for the new cluster's ArgoCD instance.
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Create external-name-service files (if zone-c)
|
||||
|
||||
For multi-zone clusters (e.g., `-ase1c`), create:
|
||||
|
||||
1. `external-name-service-incubator/<env>/<zone>/external-name-service-incubator-<cluster>.yaml`
|
||||
2. Update external-name-service values for the zone if needed.
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Validate locally
|
||||
|
||||
```bash
|
||||
# Render the incubator Application
|
||||
helm template generic-argo-apps-chart/ \
|
||||
-f values/<env>/incubator-infra-<cluster>-values.yaml
|
||||
|
||||
# Verify YAML syntax
|
||||
yamllint values/<env>/incubator-infra-<cluster>-values.yaml
|
||||
yamllint incubator/<env>/incubator-infra-<cluster>.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Open PR
|
||||
|
||||
- Both files (incubator + values) in the same PR.
|
||||
- Target branch: matches environment — `main` (prd), `develop` (stg/dev), `pre-prod` (int).
|
||||
- Required: Platform team review.
|
||||
|
||||
---
|
||||
|
||||
## Step 7: Post-merge bootstrap (manual, platform team)
|
||||
|
||||
After the PR merges:
|
||||
|
||||
1. The incubator Application auto-syncs to the ArgoCD admin cluster.
|
||||
2. ArgoCD reads the values file and renders child Applications (initially none if `appSpec` is empty).
|
||||
3. Verify in ArgoCD UI that the incubator Application is healthy.
|
||||
4. Add initial tools via [add-tool-to-cluster.md](add-tool-to-cluster.md).
|
||||
|
||||
---
|
||||
|
||||
## Naming convention reference
|
||||
|
||||
| Component | Convention | Example |
|
||||
| --------- | ---------- | ------- |
|
||||
| Incubator file | `incubator/<env>/incubator-infra-<cluster>.yaml` | `incubator/prd/incubator-infra-k8s-dsgpu-prd-ase1.yaml` |
|
||||
| Values file | `values/<env>/incubator-infra-<cluster>-values.yaml` | `values/prd/incubator-infra-k8s-dsgpu-prd-ase1-values.yaml` |
|
||||
| Incubator Application name | `incubator-infra-<cluster>` | `incubator-infra-k8s-dsgpu-prd-ase1` |
|
||||
|
||||
---
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] GKE cluster is provisioned and registered in ArgoCD
|
||||
- [ ] Incubator file created with correct name and paths
|
||||
- [ ] Values file created with correct `clusterSpec.destination.name`
|
||||
- [ ] `valueFiles` path in incubator points to correct values file
|
||||
- [ ] `teamSpec.source.valueFiles` points to correct `helm-overrides/<cluster>` directory
|
||||
- [ ] Labels match the cluster name and environment
|
||||
- [ ] Both files pass `yamllint`
|
||||
- [ ] Pre-commit hooks pass
|
||||
@@ -0,0 +1,182 @@
|
||||
# Procedure: Add, Update, or Remove a Tool from a Cluster
|
||||
|
||||
> Step-by-step guide for modifying `appSpec` entries in cluster values files.
|
||||
>
|
||||
> **Layer:** 1-T (Tool-Mediated). Use `argo-app-tool` where available; manual YAML edits acceptable with checklist.
|
||||
>
|
||||
> **Blast radius:** Single cluster, single tool. Auto-sync means changes deploy immediately after merge to the env branch (`main` for prd, `develop` for stg, `pre-prod` for int).
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before starting, confirm:
|
||||
|
||||
1. The tool's **chart** exists in `devops-infra-helm-charts/helm-templates/<chartDir>/` on the correct branch.
|
||||
2. The tool's **values override** exists in `devops-infra-helm-charts/helm-overrides/<cluster>/<valuesDir>/custom-values.yaml` on the correct branch.
|
||||
3. You know which **cluster** and **environment** the tool should deploy to.
|
||||
|
||||
Both this repo and `devops-infra-helm-charts` use the same branch convention:
|
||||
|
||||
| Environment | Branch |
|
||||
| ----------- | ------ |
|
||||
| Production (prd) | `main` |
|
||||
| Staging (stg) | `develop` |
|
||||
| Integration (int) | `pre-prod` |
|
||||
|
||||
If either the chart or override doesn't exist on the target branch, the work starts in `devops-infra-helm-charts` first. See [devops-infra-helm-charts procedures](https://github.com/Meesho/devops-infra-helm-charts).
|
||||
|
||||
---
|
||||
|
||||
## Add a new tool
|
||||
|
||||
### Step 1: Identify the target values file
|
||||
|
||||
```text
|
||||
values/<env>/incubator-infra-<cluster>-values.yaml
|
||||
```
|
||||
|
||||
Example: `values/prd/incubator-infra-k8s-central-prd-ase1-values.yaml`
|
||||
|
||||
### Step 2: Verify chart and values exist in the sister repo
|
||||
|
||||
```bash
|
||||
# Chart directory
|
||||
ls /path/to/devops-infra-helm-charts/helm-templates/<chartDir>/
|
||||
|
||||
# Values override
|
||||
ls /path/to/devops-infra-helm-charts/helm-overrides/<cluster>/<valuesDir>/custom-values.yaml
|
||||
```
|
||||
|
||||
If either is missing, stop. Create them in `devops-infra-helm-charts` first.
|
||||
|
||||
### Step 3: Compute the auto-generated Application name
|
||||
|
||||
Apply the munging rules from [coding-guidelines/infra-argo.md](../global/coding-guidelines/infra-argo.md):
|
||||
|
||||
```text
|
||||
<name>-<mungedCluster>-<env>
|
||||
```
|
||||
|
||||
Example: `kyverno` on `k8s-farmiso-prd-ase1` → `kyverno-farmiso-prd`
|
||||
|
||||
Verify:
|
||||
- Name is <= 253 characters.
|
||||
- Name doesn't collide with an existing Application (grep the values file).
|
||||
|
||||
### Step 4: Add the appSpec entry
|
||||
|
||||
Append to the `appSpec` list in the values file:
|
||||
|
||||
```yaml
|
||||
- name: kyverno
|
||||
namespace: kyverno-farmiso-prd
|
||||
chartDir: kyverno
|
||||
valuesDir: kyverno
|
||||
```
|
||||
|
||||
Only add `nameOverride` if Step 3 revealed a collision or length issue.
|
||||
|
||||
Only add `additionalValueFiles` if the tool requires region-shared overlays.
|
||||
|
||||
### Step 5: Validate locally
|
||||
|
||||
```bash
|
||||
# Render the generic chart with the updated values
|
||||
helm template generic-argo-apps-chart/ \
|
||||
-f values/prd/incubator-infra-<cluster>-values.yaml | grep -A 20 "name: kyverno"
|
||||
|
||||
# Verify no YAML syntax errors
|
||||
yamllint values/prd/incubator-infra-<cluster>-values.yaml
|
||||
```
|
||||
|
||||
### Step 6: Open PR
|
||||
|
||||
- Target branch: `main` (prd) / `develop` (stg) / `pre-prod` (int)
|
||||
- Required: Platform team review
|
||||
- Pre-commit hooks must pass (TruffleHog, CAC, Yaak)
|
||||
|
||||
### Step 7: After merge
|
||||
|
||||
ArgoCD auto-syncs from the env branch. The new Application will appear in ArgoCD within minutes. Verify in the ArgoCD UI that:
|
||||
- The Application is created with the expected name.
|
||||
- It syncs successfully.
|
||||
- The target namespace is created.
|
||||
|
||||
---
|
||||
|
||||
## Update an existing tool
|
||||
|
||||
### Change chart version
|
||||
|
||||
Update `chartDir` to point to the new chart directory:
|
||||
|
||||
```yaml
|
||||
- name: contour-internal-0
|
||||
chartDir: contour-v1.33.3 # was: contour
|
||||
```
|
||||
|
||||
Verify the new `chartDir` exists in `helm-templates/`.
|
||||
|
||||
### Change values directory
|
||||
|
||||
Update `valuesDir`:
|
||||
|
||||
```yaml
|
||||
- name: vmagent
|
||||
valuesDir: victoria-metrics-agent-new # was: victoria-metrics-agent
|
||||
```
|
||||
|
||||
Verify the new `valuesDir` exists in `helm-overrides/<cluster>/`.
|
||||
|
||||
### Change namespace
|
||||
|
||||
Update `namespace`. This is a destructive operation — ArgoCD will create the new namespace and deploy there, but the old namespace's resources are **not automatically cleaned up**.
|
||||
|
||||
```yaml
|
||||
- name: ai-gateway
|
||||
namespace: ai-gateway-prd # was: ai-gateway
|
||||
```
|
||||
|
||||
After merge, manually clean up the old namespace if no other tools use it.
|
||||
|
||||
---
|
||||
|
||||
## Remove a tool from a cluster
|
||||
|
||||
### Step 1: Delete the appSpec entry
|
||||
|
||||
Remove the entire `- name: ...` block from the values file.
|
||||
|
||||
### Step 2: Verify no other entries depend on it
|
||||
|
||||
Check if any other appSpec entries reference the same namespace or have dependencies on this tool.
|
||||
|
||||
### Step 3: Open PR
|
||||
|
||||
After merge, the ArgoCD Application will be deleted by the finalizer (`resources-finalizer.argocd.argoproj.io`), which will also clean up the deployed resources.
|
||||
|
||||
**Warning:** If the Application has `CreateNamespace=true` in syncOptions and the namespace is shared with other tools, removing the Application will NOT delete the namespace. Namespace cleanup is manual.
|
||||
|
||||
---
|
||||
|
||||
## Multi-cluster rollout
|
||||
|
||||
When adding a tool to multiple clusters:
|
||||
|
||||
1. Add the appSpec entry to each cluster's values file in the **same PR**.
|
||||
2. Verify chart/values exist for **each** cluster — valuesDir names may differ per cluster.
|
||||
3. Each cluster should have its own override directory in `devops-infra-helm-charts/helm-overrides/<cluster>/<valuesDir>/`.
|
||||
|
||||
---
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] `chartDir` exists in `devops-infra-helm-charts/helm-templates/`
|
||||
- [ ] `valuesDir` exists in `devops-infra-helm-charts/helm-overrides/<cluster>/`
|
||||
- [ ] Auto-generated Application name <= 253 characters
|
||||
- [ ] No name collision with existing appSpec entries
|
||||
- [ ] `nameOverride` used only if justified
|
||||
- [ ] `helm template` renders without errors
|
||||
- [ ] `yamllint` passes
|
||||
- [ ] Pre-commit hooks pass
|
||||
@@ -0,0 +1,118 @@
|
||||
# Procedure: Deboard a Tool from a Cluster
|
||||
|
||||
> Step-by-step guide for safely removing an `appSpec` entry from a cluster values file.
|
||||
>
|
||||
> **Layer:** 1-T (Tool-Mediated). Agent generates the diff and opens a PR.
|
||||
>
|
||||
> **Blast radius:** Single cluster. After merge, ArgoCD deletes the Application and its resources via finalizer.
|
||||
>
|
||||
> Per AI Blitz Plan §5.2.
|
||||
|
||||
---
|
||||
|
||||
## Warning: deletion is destructive
|
||||
|
||||
Removing an `appSpec` entry causes ArgoCD to delete the child Application, which triggers the `resources-finalizer.argocd.argoproj.io` to **delete all Kubernetes resources** the Application manages. This includes Deployments, Services, ConfigMaps, PVCs, and the namespace (if `CreateNamespace=true` and the namespace is not shared).
|
||||
|
||||
**Confirm with the tool owner before proceeding.**
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. Confirm the tool is no longer needed on this cluster.
|
||||
2. Confirm no other system depends on the tool's namespace (e.g., shared `victoriametrics` namespace — removing one app doesn't remove the namespace, but verify).
|
||||
3. If the tool has persistent storage (PVC), confirm data can be discarded or has been backed up.
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1: Find the appSpec entry
|
||||
|
||||
```bash
|
||||
grep -n -A 5 'name: <tool>' values/<env>/incubator-infra-<cluster>-values.yaml
|
||||
```
|
||||
|
||||
Record the line numbers of the entire entry block.
|
||||
|
||||
### Step 2: Check for namespace sharing
|
||||
|
||||
```bash
|
||||
# Does any other appSpec entry use the same namespace?
|
||||
grep 'namespace: <namespace>' values/<env>/incubator-infra-<cluster>-values.yaml
|
||||
```
|
||||
|
||||
If another entry shares the namespace, removing this entry will NOT delete the namespace — only the Application's resources. This is usually safe, but confirm.
|
||||
|
||||
### Step 3: Compute the Application name that will be deleted
|
||||
|
||||
```text
|
||||
<name>-<mungedCluster>-<env>
|
||||
```
|
||||
|
||||
Use this to verify the Application in ArgoCD before the PR merges.
|
||||
|
||||
### Step 4: Remove the appSpec entry
|
||||
|
||||
Delete the entire entry block from the values file:
|
||||
|
||||
```yaml
|
||||
# Remove this block entirely:
|
||||
- name: <tool>
|
||||
namespace: <namespace>
|
||||
chartDir: <chartDir>
|
||||
valuesDir: <valuesDir>
|
||||
```
|
||||
|
||||
Do not leave empty lines or dangling list markers.
|
||||
|
||||
### Step 5: Validate
|
||||
|
||||
```bash
|
||||
helm template generic-argo-apps-chart/ \
|
||||
-f values/<env>/incubator-infra-<cluster>-values.yaml
|
||||
|
||||
yamllint values/<env>/incubator-infra-<cluster>-values.yaml
|
||||
```
|
||||
|
||||
Confirm the removed Application no longer appears in the rendered output.
|
||||
|
||||
### Step 6: Open PR
|
||||
|
||||
- Title: `deboard <tool> from <cluster>`
|
||||
- PR body must include: what the tool was doing, why it's being removed, who confirmed the removal, and namespace cleanup plan.
|
||||
- Required: Platform team review + tool owner acknowledgment
|
||||
|
||||
### Step 7: Post-merge cleanup
|
||||
|
||||
After merge and ArgoCD sync completes:
|
||||
|
||||
1. Verify in ArgoCD that the Application has been deleted.
|
||||
2. Verify the namespace is gone (or still exists if shared — expected).
|
||||
3. If `custom-values.yaml` in `devops-infra-helm-charts` is no longer needed, clean it up in a separate PR.
|
||||
|
||||
---
|
||||
|
||||
## Special case: decommissioning a cluster
|
||||
|
||||
If you are removing **all** tools from a cluster as part of cluster decommission:
|
||||
|
||||
1. Do **not** delete the incubator file in this PR — that is a separate step requiring confirmation.
|
||||
2. First empty the `appSpec` list: `appSpec: []`
|
||||
3. After all tools are confirmed deleted, open a second PR to delete the incubator and values files.
|
||||
|
||||
See escalation matrix: [docs/global/escalation-matrix.md](../../global/escalation-matrix.md) — cluster decommission requires coordinated sign-off.
|
||||
|
||||
---
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] Tool owner confirmed removal
|
||||
- [ ] Namespace sharing checked — no unintended namespace deletion
|
||||
- [ ] Persistent storage impact assessed
|
||||
- [ ] Application name identified for post-merge verification
|
||||
- [ ] `helm template` renders without errors
|
||||
- [ ] `yamllint` passes
|
||||
- [ ] Pre-commit hooks pass
|
||||
- [ ] PR body includes removal rationale and namespace cleanup plan
|
||||
@@ -0,0 +1,126 @@
|
||||
# Procedure: Fleet-Wide Tool Rollout
|
||||
|
||||
> Step-by-step guide for adding a new tool to multiple clusters in a single PR.
|
||||
>
|
||||
> **Layer:** 1-T (Tool-Mediated). Agent generates the diff and opens a PR.
|
||||
>
|
||||
> **Blast radius:** ALL targeted clusters simultaneously. Auto-sync means all clusters deploy on merge.
|
||||
>
|
||||
> Per AI Blitz Plan §5.2. Skill: [skills/infra/fleet-wide-rollout.md](../../../skills/infra/fleet-wide-rollout.md).
|
||||
|
||||
---
|
||||
|
||||
## When to use this procedure
|
||||
|
||||
- Adding an observability tool (e.g., Coroot, Pyroscope) to a set of clusters
|
||||
- Rolling out a security policy tool (e.g., Kyverno) fleet-wide
|
||||
- Deploying a new mandatory platform component across all production clusters
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before touching any values file:
|
||||
|
||||
1. **Chart exists** in `devops-infra-helm-charts/helm-templates/<chartDir>/`.
|
||||
2. **Override values exist** for EACH target cluster: `devops-infra-helm-charts/helm-overrides/<cluster>/<valuesDir>/custom-values.yaml`. ValuesDir names may differ per cluster — verify each one.
|
||||
3. Decide whether this is a **full fleet** rollout or **subset** (specific envs, BUs, or roles).
|
||||
|
||||
If any chart or override is missing, stop. Create them in `devops-infra-helm-charts` first.
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1: Identify target clusters
|
||||
|
||||
List all values files for the target env:
|
||||
|
||||
```bash
|
||||
ls values/prd/
|
||||
ls values/int/
|
||||
```
|
||||
|
||||
Filter to the clusters you need. Common subsets:
|
||||
|
||||
| Subset | Description |
|
||||
| ------ | ----------- |
|
||||
| All prd | All files in `values/prd/` |
|
||||
| Data clusters | `datascience`, `dataengg`, `dengspark`, `dscispark`, `dsgpu` |
|
||||
| App clusters | `demand`, `supply`, `farmiso`, `central` |
|
||||
| Admin cluster | `admin-prd`, `devops-admin`, `sec-admin` |
|
||||
|
||||
### Step 2: Verify override exists for each cluster
|
||||
|
||||
```bash
|
||||
for cluster in <cluster-list>; do
|
||||
echo -n "$cluster: "
|
||||
ls /path/to/devops-infra-helm-charts/helm-overrides/$cluster/<valuesDir>/custom-values.yaml \
|
||||
&& echo "ok" || echo "MISSING"
|
||||
done
|
||||
```
|
||||
|
||||
**Halt** if any cluster is missing its override. The override must be added to `devops-infra-helm-charts` first. Only proceed when all clusters are covered.
|
||||
|
||||
### Step 3: Compute Application names for all clusters
|
||||
|
||||
For each target cluster, compute `<name>-<mungedCluster>-<env>`. Verify no name collisions with existing entries.
|
||||
|
||||
### Step 4: Add appSpec entries to all target values files
|
||||
|
||||
For each cluster values file, append:
|
||||
|
||||
```yaml
|
||||
- name: <tool>
|
||||
namespace: <namespace>
|
||||
chartDir: <chartDir>
|
||||
valuesDir: <valuesDir>
|
||||
```
|
||||
|
||||
The `valuesDir` value may differ per cluster if override directories are cluster-named.
|
||||
|
||||
### Step 5: Validate each cluster
|
||||
|
||||
```bash
|
||||
for f in values/prd/incubator-infra-k8s-<cluster>-prd-ase1-values.yaml; do
|
||||
echo "=== $f ==="
|
||||
helm template generic-argo-apps-chart/ -f "$f" | grep -c "kind: Application"
|
||||
yamllint "$f"
|
||||
done
|
||||
```
|
||||
|
||||
All files must render cleanly.
|
||||
|
||||
### Step 6: Open PR
|
||||
|
||||
- Title: `onboard <tool> to [all prd clusters | <subset description>]`
|
||||
- PR body must include:
|
||||
- Complete list of modified clusters
|
||||
- The `chartDir` and `valuesDir` used
|
||||
- Link to the `devops-infra-helm-charts` PR that added the chart/overrides (if applicable)
|
||||
- Confirmation that all override directories exist
|
||||
- Required: Platform team review
|
||||
|
||||
---
|
||||
|
||||
## Staged rollout alternative
|
||||
|
||||
If you want cluster-by-cluster rollout (to catch issues early):
|
||||
|
||||
1. Start with one low-risk cluster (e.g., `k8s-datascience-prd-ase1`).
|
||||
2. Open PR 1, merge, verify in ArgoCD.
|
||||
3. Open PR 2 with the remaining clusters.
|
||||
|
||||
Staged rollout requires multiple PRs and more human attention but reduces blast radius per deploy.
|
||||
|
||||
---
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] Chart exists in `devops-infra-helm-charts/helm-templates/`
|
||||
- [ ] Override exists for EVERY target cluster in `devops-infra-helm-charts/helm-overrides/<cluster>/`
|
||||
- [ ] All Application names computed and verified no collisions
|
||||
- [ ] `helm template` passes for every modified values file
|
||||
- [ ] `yamllint` passes for every modified values file
|
||||
- [ ] PR body lists all modified clusters
|
||||
- [ ] Pre-commit hooks pass
|
||||
@@ -0,0 +1,123 @@
|
||||
# Procedure: Upgrade a Chart Version
|
||||
|
||||
> Step-by-step guide for changing the `chartDir` of an `appSpec` entry to reference a new chart version.
|
||||
>
|
||||
> **Layer:** 1-T (Tool-Mediated). Agent generates the diff and opens a PR.
|
||||
>
|
||||
> **Blast radius:** Single cluster (if one values file) or fleet-wide (if multiple clusters). Auto-sync means changes deploy immediately after merge to the env branch (`main` for prd, `develop` for stg, `pre-prod` for int).
|
||||
>
|
||||
> Per AI Blitz Plan §5.2. Skill: [skills/infra/upgrade-chart-version.md](../../../skills/infra/upgrade-chart-version.md).
|
||||
|
||||
---
|
||||
|
||||
## When to use this procedure
|
||||
|
||||
- Upgrading a tool to a new version by switching `chartDir` (e.g., `contour` → `contour-v1.33.3`)
|
||||
- Pinning a tool back to a previous chart directory
|
||||
- Upgrading a tool across multiple clusters as part of a coordinated rollout
|
||||
|
||||
**Do not** use this procedure to change `teamSpec.source.targetRevision` — that's a HIGH RISK operation requiring explicit platform-team sign-off.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. The **new chart directory** exists in `devops-infra-helm-charts/helm-templates/<newChartDir>/`.
|
||||
2. The **override values** for the new chart version are compatible — check `devops-infra-helm-charts/helm-overrides/<cluster>/<valuesDir>/custom-values.yaml` for any new required fields.
|
||||
3. You know which clusters need upgrading.
|
||||
|
||||
---
|
||||
|
||||
## Single-cluster upgrade
|
||||
|
||||
### Step 1: Identify the target appSpec entry
|
||||
|
||||
Find the tool in the cluster's values file:
|
||||
|
||||
```bash
|
||||
grep -A 4 'name: <tool>' values/<env>/incubator-infra-<cluster>-values.yaml
|
||||
```
|
||||
|
||||
Note the current `chartDir`.
|
||||
|
||||
### Step 2: Verify the new chart directory exists
|
||||
|
||||
```bash
|
||||
ls /path/to/devops-infra-helm-charts/helm-templates/<newChartDir>/
|
||||
```
|
||||
|
||||
**Halt** if missing — the chart must be added to `devops-infra-helm-charts` first.
|
||||
|
||||
### Step 3: Check override compatibility
|
||||
|
||||
```bash
|
||||
# Review the custom-values.yaml for any changed keys in the new chart
|
||||
cat /path/to/devops-infra-helm-charts/helm-overrides/<cluster>/<valuesDir>/custom-values.yaml
|
||||
```
|
||||
|
||||
If the new chart has breaking changes (renamed keys, removed defaults), the `custom-values.yaml` in `devops-infra-helm-charts` must be updated first — that is a separate PR in the sister repo.
|
||||
|
||||
### Step 4: Update the `chartDir`
|
||||
|
||||
Change the `chartDir` field in the values file:
|
||||
|
||||
```yaml
|
||||
# Before:
|
||||
- name: contour-internal-0
|
||||
chartDir: contour
|
||||
valuesDir: contour-internal
|
||||
|
||||
# After:
|
||||
- name: contour-internal-0
|
||||
chartDir: contour-v1.33.3
|
||||
valuesDir: contour-internal
|
||||
```
|
||||
|
||||
Do not change `name`, `namespace`, or `valuesDir` unless the upgrade specifically requires it.
|
||||
|
||||
### Step 5: Validate
|
||||
|
||||
```bash
|
||||
helm template generic-argo-apps-chart/ \
|
||||
-f values/<env>/incubator-infra-<cluster>-values.yaml \
|
||||
| grep -B 2 -A 30 "name: <tool>"
|
||||
|
||||
yamllint values/<env>/incubator-infra-<cluster>-values.yaml
|
||||
```
|
||||
|
||||
Verify the rendered Application points to the new chart path.
|
||||
|
||||
### Step 6: Open PR
|
||||
|
||||
- Title: `upgrade <tool> to <newChartDir> on <cluster>`
|
||||
- Required: Platform team review
|
||||
- Pre-commit hooks must pass
|
||||
|
||||
---
|
||||
|
||||
## Multi-cluster upgrade
|
||||
|
||||
When upgrading the same tool across multiple clusters:
|
||||
|
||||
1. Update `chartDir` in **all affected values files in the same PR**.
|
||||
2. Validate each cluster's values file independently with `helm template`.
|
||||
3. Title: `upgrade <tool> to <newChartDir> across [list of clusters or "all prd clusters"]`
|
||||
|
||||
**Ordering note:** ArgoCD auto-syncs all clusters on merge. There is no cluster-by-cluster rollout order. If you need a staged rollout (one cluster at a time), open separate PRs.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
If the upgrade causes issues, create a revert PR changing `chartDir` back to the previous value. Do not force-push.
|
||||
|
||||
---
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] New `chartDir` exists in `devops-infra-helm-charts/helm-templates/`
|
||||
- [ ] `custom-values.yaml` is compatible with the new chart version
|
||||
- [ ] `helm template` renders without errors for each updated cluster
|
||||
- [ ] `yamllint` passes
|
||||
- [ ] Pre-commit hooks pass
|
||||
- [ ] For multi-cluster: all clusters validated before PR opens
|
||||
Reference in New Issue
Block a user