gitadmin/devops-infra-helm-charts-gcp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added repo

Browse Source
This commit is contained in:
Your Name
2026-08-26 03:39:42 +05:30
parent 45c25a95af
commit b8575bb8b9
6889 changed files with 1217125 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/platform/schemas/custom-values-schema.md
+379
View File
@@ -0,0 +1,379 @@
# Schema — `custom-values.yaml` (Helm override)
> Field-by-field annotation of the values file Argo CD's `valueFiles` references.
> Authoritative for `helm-overrides/<cluster>/<app>/custom-values.yaml`.
>
> Cross-reference: [coding-guidelines/helm-values.md](../../global/coding-guidelines/helm-values.md), [../../architecture.md](../../architecture.md).
---
## What this file is — and isn't
A `custom-values.yaml` is **not** a Helm chart. It is the *override* layer Argo CD merges over the chart's own `values.yaml` (or, for thin-wrapper charts, the upstream subchart's defaults).
- **Schema is owned by the chart, not by us.** Every key here must be a key the chart understands. Adding a key the chart doesn't read does nothing.
- **No multi-doc YAML.** One document per file.
- **No `apiVersion` / `kind`.** This file is values, not a Kubernetes manifest.
---
## Top-level shape (typical)
```yaml
# image registry / repository / tag
image:
registry: asia-southeast1-docker.pkg.dev
repository: meesho-devops-admin-0622/admin/sre/<image>
tag: <semver-or-sha>
pullPolicy: IfNotPresent
# release-name pinning (rarely changed once set)
fullnameOverride: <release>-<bu>-prd # e.g. kube-state-metrics-dbc-dsci-prd
# scaling
replicaCount: 3 # static, OR
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 10
targetCPUUtilizationPercentage: 70
# scheduling — bespoke per cluster
nodeSelector:
dedicated: <pool-key> # standard GKE
# cloud.google.com/compute-class: <cc> # GKE Autopilot
tolerations:
- key: dedicated
value: <pool-key>
effect: NoSchedule
# resources
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
cpu: 1
memory: 2Gi
# persistence (stateful charts only)
persistence:
enabled: true
storageClass: sc-pd-ssd # MUST exist in manifests/storageclass/
size: 100Gi
accessModes: [ReadWriteOnce]
# secrets — by reference, never inline
existingSecret: <name-managed-by-external-secrets>
# probes
livenessProbe:
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
initialDelaySeconds: 5
periodSeconds: 5
# ingress / Contour HTTPProxy (if applicable)
ingress:
enabled: false # most internal services
# chart-specific top-level keys
# (varies — read the chart's values.yaml for the full schema)
```
---
## §image
| Field | Required | Convention | Notes |
|-------|----------|------------|-------|
| `image.registry` | yes (production) | `asia-southeast1-docker.pkg.dev` | Meesho's Artifact Registry mirror. ([SANCTITY_RULES R11](../../global/SANCTITY_RULES.md)) |
| `image.repository` | yes | `meesho-devops-admin-0622/admin/sre/<image>` | Mirror path. |
| `image.tag` | yes (production) | semver, datestamp, or git SHA | Never `latest`. Never unpinned. |
| `image.pullPolicy` | no | `IfNotPresent` | `Always` only for development; pulls slow rollout. |
| `image.pullSecrets` | no | omit | Mirror is public to fleet; pull secrets are a lock-in trap. |
Note: chart authors disagree on the shape — some use `image: {repository, tag}` (no `registry`), some use a single `image: <fully-qualified>`, some split into `image.repository: <registry>/<repo>`. **Read the chart's own `values.yaml`** before structuring this block.
---
## §release identity
| Field | Required | Convention | Notes |
|-------|----------|------------|-------|
| `fullnameOverride` | varies | Set once at release creation; stable forever | Service DNS, PVC binding, ConfigMap refs depend on it. ([SANCTITY_RULES R9](../../global/SANCTITY_RULES.md)) |
| `nameOverride` | rare | Omit unless explicitly needed | |
| `commonLabels` / `commonAnnotations` | rare | Omit unless the chart documents the pattern | |
For dataplane (`db-*`) clusters: `fullnameOverride: <kind>-dbc-<bu>-prd` is the convention (e.g. `kube-state-metrics-dbc-dsci-prd`). For BU clusters, omit unless the chart's default name collides.
---
## §scaling
Charts diverge sharply here. Common shapes:
```yaml
# Pattern A — static replicas
replicaCount: 3
# Pattern B — chart-level autoscaling block
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 10
targetCPUUtilizationPercentage: 70
# Pattern C — separate HPA resource (KEDA / VictoriaMetrics)
hpa:
enabled: true
min: 3
max: 20
metrics:
- type: Resource
resource: {name: cpu, target: {type: Utilization, averageUtilization: 70}}
```
| Rule | Why |
|------|-----|
| **Don't set `replicaCount`** when `autoscaling.enabled: true`. | HPA and the chart's static deployment fight; replica count flaps. |
| **Set both `minReplicas` and `maxReplicas`** when autoscaling. | Without `min`, the HPA can scale to zero on a quiet hour. |
| **`minReplicas` ≥ 2** for any production-traffic-path workload. | Single-replica releases die on node drain. |
---
## §scheduling
The single biggest source of silent mis-deploys. Per [SANCTITY_RULES R5](../../global/SANCTITY_RULES.md):
| Cluster | Key style |
|---------|-----------|
| GKE Autopilot (`k8s-central-prd-ase1`, `k8s-dsgpu-prd-ase1`, `k8s-shared-int-ase1`) | `cloud.google.com/compute-class:` |
| Standard GKE | `dedicated:` |
```yaml
# Standard GKE — Contour internal-0 on most clusters
nodeSelector:
dedicated: contour-internal-0
tolerations:
- key: dedicated
value: contour-internal-0
effect: NoSchedule
# often a second toleration to allow scheduling onto shared nodes
- key: dedicated
value: contour-shared
effect: NoSchedule
# GKE Autopilot — same Contour internal-0 on k8s-central-prd-ase1
nodeSelector:
cloud.google.com/compute-class: contour-internal-0-cc
tolerations:
- key: cloud.google.com/compute-class
value: contour-internal-0-cc
effect: NoSchedule
- key: cloud.google.com/compute-class
value: contour-shared-cc
effect: NoSchedule
```
The full per-cluster Contour matrix lives in [`contour-nodeselector-tolerations-summary.md`](../../../contour-nodeselector-tolerations-summary.md). For non-Contour apps, copy from a sibling app on the **same** cluster.
| Field | Notes |
|-------|-------|
| `nodeSelector` | Hard requirement — no scheduling onto non-matching nodes. Use exactly one key (the cluster's pool key). |
| `tolerations` | Allow scheduling onto tainted nodes. May list multiple to allow shared nodes alongside dedicated. |
| `affinity.nodeAffinity` | Use sparingly; `nodeSelector` is sufficient for most cases here. |
| `topologySpreadConstraints` | Use for multi-zone resilience on chatty workloads (Contour, ingress-nginx). |
---
## §resources
```yaml
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
cpu: 1
memory: 2Gi
```
| Rule | Why |
|------|-----|
| **Always set `requests`** in production. | Without requests, pods are best-effort; first to evict on memory pressure. |
| **Set `limits` unless the chart explicitly recommends omitting them.** | Some sidecars (cert-manager, external-dns) deliberately skip `limits`. |
| **Don't copy resources from another cluster.** | Workload sizing is per-traffic-tier. Supply prd is not demand prd. |
| **CPU limits cause throttling**, not OOMKill. Memory limits cause OOMKill. Tune accordingly. |
---
## §persistence
```yaml
persistence:
enabled: true
storageClass: sc-pd-ssd
size: 100Gi
accessModes: [ReadWriteOnce]
```
Allowed `storageClass` values (must match a file under `manifests/storageclass/`):
| Class | Backend | Use case |
|-------|---------|----------|
| `pd-standard-retain-dr` | GCP Persistent Disk standard, retention-on-delete | DR-critical state |
| `sc-pd-ssd` | GCP PD SSD | Default for write-heavy workloads (etcd, ClickHouse) |
| `sc-pd-standard` | GCP PD standard | Default for read-heavy / archival |
| `sc-filestore-standard` | GCP Filestore | Shared volumes (Jenkins build cache, JFrog filestore) |
| Rule | Why |
|------|-----|
| **Never reference a `storageClass`** that doesn't exist in `manifests/storageclass/`. | PVC stays Pending forever. |
| **`size` must be set explicitly.** | Chart defaults are often wrong (8Gi for everything). |
| **`accessModes`** for `ReadWriteMany` requires Filestore. PD-backed classes are RWO. |
---
## §secrets
```yaml
# Chart-specific — read the chart's values.yaml
existingSecret: <name>
# OR
auth:
existingSecret: <name>
# OR
envFrom:
- secretRef: {name: <name>}
```
| Rule | Why |
|------|-----|
| **Never inline `password:`, `apiKey:`, etc.** | Pre-commit TruffleHog will catch many; it won't catch all. ([SANCTITY_RULES R4](../../global/SANCTITY_RULES.md)) |
| **The `Secret` resource is materialised** by the cluster's `external-secrets` app. If the secret name doesn't appear in `helm-overrides/<cluster>/external-secrets/`, it doesn't exist. |
| **Reference, don't author.** Adding the secret material to a Helm-rendered `Secret` template defeats the External Secrets pattern. |
---
## §probes
```yaml
livenessProbe:
httpGet: {path: /healthz, port: http}
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 3
readinessProbe:
httpGet: {path: /ready, port: http}
initialDelaySeconds: 5
periodSeconds: 5
# For slow-warm workloads (Jenkins, JFrog, ClickHouse)
startupProbe:
httpGet: {path: /healthz, port: http}
initialDelaySeconds: 60
periodSeconds: 10
failureThreshold: 60 # 10 minutes total runway
```
| Rule | Why |
|------|-----|
| **Always set `liveness` + `readiness`** for any long-running container. |
| **Use `startupProbe` for slow-warm workloads** instead of inflating `livenessProbe.initialDelaySeconds` to 600 s. |
| **Don't set the same probe to both `liveness` and `readiness`** — they have different purposes (kill vs gate). |
---
## §ingress (Contour `HTTPProxy` / `Ingress`)
For external-facing apps, ingress is via Contour `HTTPProxy`. Specifics depend on the cluster's Contour topology (multiple Contour instances per cluster — see [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md)).
- The `external-dns` annotation is the DNS-binding step — match the cluster's DNS pattern.
- TLS terminates at Contour with `cert-manager`-issued certs; reference the issuer by name.
- Internal services use `contour-internal-0` / `contour-internal-1`; external services use `contour-external` / `contour-external-1`. The class is set via the `Service` annotation, not the values file directly.
---
## End-to-end example — a typical observability-side override
`helm-overrides/k8s-supply-prd-ase1/kube-state-metrics/custom-values.yaml`:
```yaml
image:
registry: asia-southeast1-docker.pkg.dev
repository: meesho-devops-admin-0622/admin/sre/kube-state-metrics
tag: v2.10.1
replicaCount: 2
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 1Gi
nodeSelector:
dedicated: monitoring
tolerations:
- key: dedicated
value: monitoring
effect: NoSchedule
prometheus:
monitor:
enabled: true
honorLabels: true
```
---
## End-to-end example — a dataplane override
`helm-overrides/db-2516183257845181-c-1204-195038-428/victoria-metrics-agent/custom-values.yaml`:
```yaml
fullnameOverride: vmagent-dbc-supply-prd
image:
registry: asia-southeast1-docker.pkg.dev
repository: meesho-devops-admin-0622/admin/sre/victoria-metrics-agent
tag: v1.96.0
replicaCount: 1
resources:
requests:
cpu: 100m
memory: 256Mi
```
(Dataplane clusters intentionally have minimal overrides.)
---
## Validation
```bash
# Render the chart locally with this values file
helm template <release> helm-templates/<chart> \
-f helm-overrides/<cluster>/<app>/custom-values.yaml | head -80
# Dry-run diff against the live cluster (requires kubectl context + helm-diff plugin)
helm diff upgrade <release> helm-templates/<chart> \
-f helm-overrides/<cluster>/<app>/custom-values.yaml
# Lint
yamllint helm-overrides/<cluster>/<app>/custom-values.yaml
helm lint helm-templates/<chart>
# Sanity: storageClass referenced exists
yq e '.persistence.storageClass' helm-overrides/<cluster>/<app>/custom-values.yaml \
| xargs -I{} ls manifests/storageclass/{}.yaml 2>/dev/null \
|| echo "WARN: storageClass not in manifests/storageclass/"
```
docs/platform/schemas/incubator-values-schema.md
+195
View File
@@ -0,0 +1,195 @@
> Per AI Blitz Plan §platform.schemas. Layer: 1. Repo: devops-infra-helm-charts.
# Schema — Incubator-tool values + sidecar contract
> **Scope:** the values-side contract for incubator (newly-vendored) infrastructure tools that live under `helm-overrides/<cluster>/<incubator-tool>/`.
>
> **Out of scope (explicit):** the Argo CD `Application` / `ApplicationSet` manifest that registers the incubator tool with a cluster's Argo. Those manifests live in the **sister repo** `github.com/Meesho/devops-infra-argo-config`, NOT here. This file documents only what `devops-infra-helm-charts` is responsible for: the values + raw sidecar manifests Argo CD reads from this repo.
This schema complements [custom-values-schema.md](custom-values-schema.md) (general values shape) and [raw-manifest-sidecar-schema.md](raw-manifest-sidecar-schema.md) (sidecar manifest shapes). Read both first.
---
## What "incubator" means here
A tool is **incubator** while it is being trialled on one or two clusters before fleet-wide rollout. In this repo that maps to:
- A new chart directory under `helm-templates/<incubator-tool>/` (often a thin wrapper `Chart.yaml` with an upstream dep).
- An override under `helm-overrides/<cluster>/<incubator-tool>/` for the trial cluster(s) only — typically `k8s-shared-int-ase1` (integration / pre-prod) first, sometimes one BU prod cluster as a canary.
- Possibly raw sidecar manifests alongside the values file.
Once the tool graduates from incubator status, the override pattern is identical to other infra tools — the "incubator" label is operational, not structural. This schema codifies the conventions that keep early-stage adoption sane.
---
## Directory layout
```
helm-overrides/<cluster>/<incubator-tool>/
├── custom-values.yaml # Helm values (required if chart is Helm-based)
├── computeclass/ # Optional — GKE Autopilot ComputeClass
│ └── <name>-cc.yaml
├── external-dns-services/ # Optional — Service for external-dns annotation
│ └── <fqdn>.yaml
├── external-secrets/ # Optional — ExternalSecret resources
│ └── <name>.yaml
└── <incubator-specific>.yaml # Tool-specific raw manifest (CRD, ConfigMap)
```
The Argo `Application` for this directory (sister repo) determines whether files are Helm-rendered, raw-applied, or layered (see [raw-manifest-sidecar-schema.md §Layered-with-Helm vs standalone](raw-manifest-sidecar-schema.md)).
---
## `custom-values.yaml` keys an incubator chart should expose
A chart promoted to incubator status MUST surface (i.e. let the override file set without forking templates) at minimum:
| Key | Why required |
|-----|--------------|
| `image.registry`, `image.repository`, `image.tag` | Production overrides pin to `asia-southeast1-docker.pkg.dev/meesho-devops-admin-0622/admin/sre/<image>` (Meesho Artifact Registry mirror, not Docker Hub). |
| `image.pullPolicy` | Default `IfNotPresent`. |
| `resources.requests.cpu`, `.memory` | No defaults assumed; Autopilot scheduling requires explicit requests. |
| `resources.limits.cpu`, `.memory` | Same. |
| `nodeSelector` | Per-cluster scheduling (see [custom-values-schema.md](custom-values-schema.md)). |
| `tolerations` | Per-cluster taints. |
| `affinity` (optional) | Pod anti-affinity for replicated tools. |
| `serviceAccount.create`, `.name`, `.annotations` | Workload Identity — `iam.gke.io/gcp-service-account` annotation goes here. |
| `replicaCount` (or `autoscaling.{enabled,minReplicas,maxReplicas}`) | Sizing. |
| `persistence.enabled`, `.storageClass`, `.size` | If stateful. `storageClass` MUST exist in `manifests/storageclass/`. See [storageclass-priorityclass-schema.md](storageclass-priorityclass-schema.md). |
| `priorityClassName` | If the tool needs preemption priority — must reference a class in `manifests/priorityclass/<cluster>/`. |
| `podLabels`, `podAnnotations` | For prometheus.io scrape annotations and team-attribution labels. |
| `extraEnv` (or `env`) | For environment-specific knobs the chart's templates don't already accept. |
If the upstream chart doesn't expose these — that's a chart-bug. **Do not** fork the chart in `helm-templates/` to add them ([NEVER-DO list](../../../CLAUDE.md)). Either upstream-PR the chart or wrap with a small Meesho-owned chart that re-exposes the keys.
---
## Values-file conventions for incubator tools
1. **Pin every image tag.** Never `latest`, never an unpinned SHA. This is the same rule as production overrides; incubator status does not relax it.
2. **Set explicit `nodeSelector` and `tolerations`** authored from scratch using sibling apps on the same cluster. Cross-cluster copying is forbidden — see [custom-values-schema.md](custom-values-schema.md).
3. **Use `fullnameOverride` deliberately or not at all.** Once set, do not change ([NEVER-DO list](../../../CLAUDE.md)). Incubator graduations to other clusters should reuse the same `fullnameOverride` string for portability.
4. **Don't enable `autoscaling`** in the first incubator deploy. Pin `replicaCount: 1` (or 2 for HA-mandatory) until you have a load profile.
5. **Don't expose the tool externally** in the incubator phase. No `external-dns-services/` until it's promoted to a cluster's stable inventory.
6. **Annotate the values file** with a top-of-file YAML comment: `# Incubator: cluster=<...>, owner=<...>, graduation-target=<date>`. Comment is plain text; not parsed; serves as reviewer signal.
7. **Confine secrets to `ExternalSecret`** under `external-secrets/`. No inline `existingSecret:` referencing a hand-applied Secret.
---
## Raw sidecar manifests in incubator directories
Concrete shapes already documented elsewhere; this section calls out which ones routinely appear with incubators.
### `computeclass/*-cc.yaml` (GKE Autopilot only)
Required when the override's `nodeSelector` uses `cloud.google.com/compute-class: <name>` and that class doesn't already exist on the cluster. The `metadata.name` MUST equal the value referenced. See [raw-manifest-sidecar-schema.md §ComputeClass](raw-manifest-sidecar-schema.md).
Example incubator pattern:
```yaml
apiVersion: autoscaling.gke.io/v1
kind: ComputeClass
metadata:
name: <incubator-tool>-cc
namespace: <ns>
spec:
priorities:
- machineFamily: c4d
minCores: 2
minMemory: 8Gi
nodePoolAutoCreation:
enabled: true
```
Per-cluster only — never copy between clusters.
### `external-dns-services/*.yaml`
Skip in the first incubator deploy. Only add once the tool has a stable internal-only DNS need. See [raw-manifest-sidecar-schema.md §Service for external-dns binding](raw-manifest-sidecar-schema.md).
### `external-secrets/*.yaml`
Always present if the tool needs secrets. Use the existing per-cluster `SecretStore` / `ClusterSecretStore`; do not author new stores in an incubator PR. See [raw-manifest-sidecar-schema.md §ExternalSecret](raw-manifest-sidecar-schema.md).
### `elastic-cluster/argo-launch.yaml`-style operator-managed CRD launches
If the incubator tool is an operator (e.g. ECK, Pyroscope), a sidecar CRD instance often lives alongside the operator's chart values:
```
helm-overrides/<cluster>/<incubator-operator>/
├── custom-values.yaml # operator chart values
└── <crd-instance>/
└── argo-launch.yaml # the actual workload CRD instance
```
Argo CD applies both in one Application (directory loader). The CRD instance must:
- Reference an `apiVersion` whose CRD the operator has already installed.
- Pin its own image / version explicitly.
- Avoid hand-rolling values that the operator templates would otherwise compute.
---
## Argo CD — sister repo contract (for cross-reference only)
The Application that points at this directory lives in `github.com/Meesho/devops-infra-argo-config`. For incubator tools the Application typically:
- Has `syncPolicy.automated` **disabled** (manual sync — incubator-grade safety; see [argocd.md](../../global/coding-guidelines/argocd.md)).
- Is named `<incubator-tool>-<cluster>` to be unambiguous.
- Lives in the cluster's namespace under `apps/<cluster>/`.
This file does NOT instruct on authoring the Application — that PR is in the sister repo. Cross-link the sister-repo PR in the values-side PR description.
---
## Validation
Same as any Helm override. Before opening a PR:
```bash
yamllint helm-overrides/<cluster>/<incubator-tool>/custom-values.yaml
helm template <release> helm-templates/<incubator-tool> \
-f helm-overrides/<cluster>/<incubator-tool>/custom-values.yaml > /tmp/render.yaml
# Validate any sidecar manifests against the cluster's CRDs
kubectl --context=<ctx> --dry-run=server \
-f helm-overrides/<cluster>/<incubator-tool>/<sidecar>.yaml apply
```
For tool-specific CRDs that aren't in vanilla Kubernetes, prefer `--dry-run=server` (the cluster validates against the registered CRD) over `--dry-run=client` (only client-side schema, often misses required fields).
---
## Graduation: incubator → stable
When the tool has been stable for some period (usually 2–4 weeks) and is ready to fleet-roll:
1. Remove the top-of-file `# Incubator:` comment.
2. Open new override directories in target clusters — author from scratch each time, do not copy.
3. Open the matching sister-repo `Application` PRs (one per new cluster) or extend the `ApplicationSet`.
4. If the chart was a thin wrapper, audit `Chart.yaml` `dependencies[].version` is current.
5. Promote the chart's documentation in `helm-templates/<incubator-tool>/README.md` (if a fork was needed) or note in PR description that no fork was needed.
---
## Anti-patterns
1. **Forking the chart in `helm-templates/<incubator-tool>/templates/`** to add a missing values key. Wrap or upstream-PR; don't fork. See NEVER-DO.
2. **Copying the entire incubator directory across clusters** for graduation. Per-cluster scheduling is bespoke.
3. **Enabling autoscaling on day one.** No load profile = thrashy autoscaler.
4. **Promoting from `k8s-shared-int-ase1` straight to a critical BU cluster.** Insert a low-traffic prod canary first.
5. **Inlining secrets** "just for the trial." TruffleHog will block; even if it didn't, the leak is real.
6. **Setting `fullnameOverride`** without a graduation plan. The string travels — bad ones travel forever.
---
## Related
- Schema: [custom-values-schema.md](custom-values-schema.md) — general values shape.
- Schema: [raw-manifest-sidecar-schema.md](raw-manifest-sidecar-schema.md) — sidecar shapes.
- Schema: [storageclass-priorityclass-schema.md](storageclass-priorityclass-schema.md) — cluster-singleton resources.
- Procedure: [../procedures/onboard-app-to-cluster.md](../procedures/onboard-app-to-cluster.md) — the procedure used to land an incubator override.
- Procedure: [../procedures/fork-upstream-chart.md](../procedures/fork-upstream-chart.md) — only when a fork is genuinely required.
- Coding guideline: [../../global/coding-guidelines/helm-values.md](../../global/coding-guidelines/helm-values.md).
- Coding guideline: [../../global/coding-guidelines/argocd.md](../../global/coding-guidelines/argocd.md) — sister-repo Application conventions.
docs/platform/schemas/raw-manifest-sidecar-schema.md
+191
View File
@@ -0,0 +1,191 @@
# Schema — Raw-manifest sidecars in `helm-overrides/`
> Field-by-field guidance for `helm-overrides/<cluster>/<app>/<extra>.yaml` — Kubernetes manifests applied **alongside** a Helm release, not through it.
---
## What this is
Some Argo CD Applications point at a directory containing both a Helm `custom-values.yaml` *and* one or more raw Kubernetes manifests. The raw manifests are not Helm-rendered; Argo CD applies them as-is.
Common shapes seen in this repo:
| Path pattern | What it is |
|--------------|------------|
| `helm-overrides/<cluster>/<app>/computeclass/*-cc.yaml` | GKE Autopilot `ComputeClass` resource — declares a node-pool/compute-class profile referenced by `nodeSelector` in the Helm values. |
| `helm-overrides/<cluster>/<app>/external-dns-services/<svc>.yaml` | A `Service` that exists purely to carry an `external-dns.alpha.kubernetes.io/hostname` annotation, binding a DNS name to a workload. |
| `helm-overrides/<cluster>/elastic-cluster/argo-launch.yaml` | An `ElasticCluster` (ECK CRD) launched alongside the operator. |
| `helm-overrides/<cluster>/<app>/mimir-distributed/alertmanager_config.yaml` | Inlined Alertmanager config materialised as a `ConfigMap`. |
If the sister-repo `Application` for this directory has `path: helm-overrides/<cluster>/<app>/`, then **every YAML file in the directory is applied** — Argo CD's directory loader treats them as a single deployment unit.
---
## File-level conventions
| Rule | Why |
|------|-----|
| **One Kubernetes resource per file** unless they're tightly coupled (e.g. a `Service` + a `ServiceAccount` referenced by it). | Reviewer cognition; rollback granularity. |
| **Pin `apiVersion` explicitly.** | `extensions/v1beta1` and `networking.k8s.io/v1beta1` are real footguns — both are gone in current Kubernetes. |
| **Set `metadata.namespace` explicitly.** | Don't rely on the Argo `Application.spec.destination.namespace` carrying through — different cluster Argos behave differently here. |
| **Trailing newline at EOF.** |
| **No multi-doc (`---`) within one file** unless genuinely required. |
---
## Top-level shape
```yaml
apiVersion: <api-version> # e.g. v1, networking.k8s.io/v1, autoscaling.gke.io/v1
kind: <kind> # e.g. Service, ComputeClass, ConfigMap, ExternalSecret
metadata:
name: <name>
namespace: <namespace>
labels: {...} # optional
annotations: {...} # often the load-bearing field (DNS, LB)
spec:
... # kind-specific
```
---
## Common kinds in this repo
### `ComputeClass` (`autoscaling.gke.io/v1`)
GKE Autopilot uses `ComputeClass` resources to declare named node profiles. The Helm values reference them via `cloud.google.com/compute-class:` keys.
```yaml
apiVersion: autoscaling.gke.io/v1
kind: ComputeClass
metadata:
name: contour-internal-0-cc
namespace: projectcontour
spec:
priorities:
- machineFamily: c4d
minCores: 4
minMemory: 16Gi
nodePoolAutoCreation:
enabled: true
```
| Rule | Why |
|------|-----|
| **`metadata.name` MUST match the value referenced by `nodeSelector` in the Helm values.** | Otherwise pods stay `Pending`. |
| **Per-cluster.** A `ComputeClass` for `k8s-central-prd-ase1` does not exist on `k8s-supply-prd-ase1`. Don't share files. |
| **The matrix of which `ComputeClass` exists where** is recorded in [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md). |
### `Service` for `external-dns` binding
```yaml
apiVersion: v1
kind: Service
metadata:
name: <hostname>-dns
namespace: <ns>
annotations:
external-dns.alpha.kubernetes.io/hostname: <fqdn>.meeshogcp.in
cloud.google.com/load-balancer-type: Internal
spec:
type: ClusterIP
selector: {app.kubernetes.io/name: <app>}
ports:
- port: 80
targetPort: 8080
```
| Rule | Why |
|------|-----|
| **`metadata.annotations.external-dns.alpha.kubernetes.io/hostname`** is the contract. Check the cluster's existing DNS pattern before authoring. |
| **`selector` must match labels of an actual workload Pod** in the same namespace. |
| **Cloud-LB annotations** (`cloud.google.com/load-balancer-type`, etc.) — copy from a sibling on the same cluster. |
### `ExternalSecret` (per-cluster `external-secrets/`)
```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: <secret-name>
namespace: <ns>
spec:
refreshInterval: 1h
secretStoreRef:
name: gcp-sm-store # cluster-level SecretStore, defined elsewhere
kind: ClusterSecretStore
target:
name: <secret-name>
creationPolicy: Owner
data:
- secretKey: <env-key>
remoteRef:
key: <gcp-secret-name>
version: latest
```
| Rule | Why |
|------|-----|
| **`spec.target.name` is the `Secret` name** the workload's `existingSecret:` references. |
| **`spec.secretStoreRef.name`** must reference a `SecretStore` / `ClusterSecretStore` that already exists on the cluster. |
| **`creationPolicy: Owner`** is the default; `creationPolicy: Merge` is for the rare case where another tool also manages the same `Secret`. |
### `ConfigMap` for inlined config
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: <name>
namespace: <ns>
data:
<key>: |
<multi-line content>
```
| Rule | Why |
|------|-----|
| **Use `|` (literal block scalar)** for multi-line content with significant whitespace. |
| **Don't inline secrets** (TruffleHog will catch obvious ones; subtle ones slip through). |
| **Reference from the workload's chart values** via the `ConfigMap` name; don't duplicate config across files. |
### `ElasticCluster` / other CRDs
For ECK, Pyroscope, etc. — the schema is the operator's, not Kubernetes's. **Read the operator's CRD docs** before authoring; copy a sibling cluster's existing `argo-launch.yaml` first.
---
## Layered-with-Helm vs standalone
| Pattern | Argo Application points at |
|---------|----------------------------|
| Pure Helm | `path: helm-overrides/<cluster>/<app>/` with `helm.valueFiles: ['custom-values.yaml']` |
| Pure raw manifests | `path: helm-overrides/<cluster>/<app>/` with `directory.recurse: true`, no `helm:` block |
| Layered | `path: helm-overrides/<cluster>/<app>/` with `helm.valueFiles: ['custom-values.yaml']` AND extra files alongside |
Argo CD's behaviour for layered directories depends on its `directory.include` / `directory.exclude` settings — when in doubt, read the sister-repo `Application` to see exactly what gets picked up.
---
## Validation
```bash
# Validate the manifest against its API
kubectl --dry-run=client -f helm-overrides/<cluster>/<app>/<extra>.yaml apply
# Lint
yamllint helm-overrides/<cluster>/<app>/<extra>.yaml
# Confirm the Argo Application loads this path
# (read the matching file in github.com/Meesho/devops-infra-argo-config)
```
---
## Anti-patterns
1. **Inlining secrets** in a `ConfigMap` "to ship a hotfix." It's still a secret. Use `ExternalSecret`.
2. **Using `apiVersion: v1beta1`** of a CRD whose stable version exists. Pin to the highest stable.
3. **A `Service` whose `selector` doesn't match any Pod** — silent: the Service exists, DNS resolves, no endpoints.
4. **Cross-cluster cloning** of a `ComputeClass` or DNS `Service` without rewriting cluster-specific fields.
5. **Hand-rendered Helm output** dropped into a sidecar file (a release "freeze"). The chart bumps and your snapshot rots.
docs/platform/schemas/storageclass-priorityclass-schema.md
+158
View File
@@ -0,0 +1,158 @@
# Schema — `manifests/storageclass/` and `manifests/priorityclass/`
> Cluster-wide singletons. **High blast radius — every PVC / scheduling decision in the cluster is affected.**
>
> Cross-reference: [SANCTITY_RULES R10](../../global/SANCTITY_RULES.md), [coding-guidelines/helm-values.md §persistence](../../global/coding-guidelines/helm-values.md).
---
## §StorageClass (`manifests/storageclass/*.yaml`)
These are **repo-global** — one StorageClass file is applied to every cluster that consumes it. A wrong reclaim policy or volume-binding mode breaks every new PVC.
### Inventory
| File | Backend | Reclaim | Binding | Use case |
|------|---------|---------|---------|----------|
| `pd-standard-retain-dr.yaml` | GCP PD standard | `Retain` | `WaitForFirstConsumer` | DR-critical state — do not lose data on PVC delete |
| `sc-pd-ssd.yaml` | GCP PD SSD | `Delete` | `WaitForFirstConsumer` | Default for write-heavy (etcd, ClickHouse) |
| `sc-pd-standard.yaml` | GCP PD standard | `Delete` | `WaitForFirstConsumer` | Default for read-heavy / archival |
| `sc-filestore-standard.yaml` | GCP Filestore | varies | varies | Shared `ReadWriteMany` volumes (Jenkins build cache, JFrog filestore) |
### Schema
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: sc-pd-ssd # MUST match filename
annotations:
storageclass.kubernetes.io/is-default-class: "false" # exactly one default per cluster
provisioner: pd.csi.storage.gke.io
parameters:
type: pd-ssd
reclaimPolicy: Delete # Delete | Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
```
| Field | Convention | Notes |
|-------|------------|-------|
| `metadata.name` | matches filename (without `.yaml`) | Helm values reference by name. |
| `metadata.annotations."storageclass.kubernetes.io/is-default-class"` | `"false"` for all of these | Exactly one StorageClass should be `"true"` per cluster (declared elsewhere, often by Terraform/cluster-bootstrap). |
| `provisioner` | `pd.csi.storage.gke.io` (PD) or `filestore.csi.storage.gke.io` (Filestore) | GCP CSI drivers. |
| `parameters.type` | `pd-ssd`, `pd-standard`, `pd-balanced` | Cost/perf trade-off. |
| `reclaimPolicy` | `Delete` (default) or `Retain` (DR) | **Changing this on an existing StorageClass does not retroactively change PVCs.** |
| `volumeBindingMode` | `WaitForFirstConsumer` | Avoids zone-mismatch on multi-zone clusters. `Immediate` only for `ReadWriteMany`. |
| `allowVolumeExpansion` | `true` | Required for online resizing. Default false on some classes — set explicitly. |
### Hard rules
1. **Never delete a StorageClass referenced by an existing PVC.** New PVCs against the deleted class fail; existing bound PVCs survive but lose the ability to expand.
2. **Never change `reclaimPolicy` from `Delete` → `Retain`** as a "safety improvement" without auditing every PVC. Existing bound PVs keep their old policy; new PVs get the new policy. Drift.
3. **Never change `provisioner`** — that's a delete-and-recreate, not an edit. Existing PVs become orphaned.
4. **Never make a new StorageClass the cluster default** without coordinating with the platform team. A wrong default class hijacks every PVC that doesn't pin a class explicitly.
5. **Validation:** every StorageClass referenced by any `helm-overrides/*/persistence.storageClass:` must have a corresponding file here.
```bash
# Find every PVC reference
grep -rE 'storageClass(Name)?:' helm-overrides | sort -u
# Find every StorageClass file
ls manifests/storageclass/*.yaml
```
---
## §PriorityClass (`manifests/priorityclass/<cluster>/*.yaml`)
**Per-cluster** but **cluster-wide** — affects scheduling priority for every pod that references the class.
### Inventory pattern
```
manifests/priorityclass/
k8s-central-prd-ase1/
priorityclass-high.yaml
priorityclass-low.yaml
k8s-supply-prd-ase1/
priorityclass-high.yaml
priorityclass-low.yaml
…
```
Most BU clusters get a `high` and a `low` class. Specialty clusters (`mqkafka`, `dsgpu`, dataplane `db-*`) sometimes have additional tiers.
### Schema
```yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: high-priority
value: 1000000 # higher = more important
globalDefault: false # NEVER true on these — would clash with system defaults
description: "High-priority workloads — promoted ahead of general pods on resource pressure"
preemptionPolicy: PreemptLowerPriority # default; alternative is Never
```
| Field | Convention | Notes |
|-------|------------|-------|
| `metadata.name` | `high-priority`, `low-priority`, etc. | Workload pods reference by name. |
| `value` | High: 1,000,000 / Low: 100 / Critical (rare): >1,000,000,000 | Kubernetes system pods reserve `> 2,000,000,000`; don't conflict. |
| `globalDefault` | **always `false`** | A `true` here applies to every pod that doesn't pin a class — chaos. |
| `description` | one-line | For audit log. |
| `preemptionPolicy` | `PreemptLowerPriority` (default) or `Never` | `Never` for batch workloads that shouldn't kick others out. |
### Hard rules
1. **Never set `globalDefault: true`** on any PriorityClass here. The cluster's implicit default is what we want.
2. **Never raise `value`** of an existing class without auditing what gets preempted. A bump from 1,000,000 → 10,000,000 changes which pods get evicted under pressure.
3. **Never delete a PriorityClass** referenced by any workload — pods that referenced it become invalid.
4. **The same `name` must mean the same thing across clusters.** `high-priority` on supply ≠ `high-priority` on demand at the *value* level is an audit nightmare. Keep values consistent.
5. **Validation:** find every pod-spec reference:
```bash
grep -rE 'priorityClassName:' helm-overrides
```
---
## §The "PV/PVC singletons" — Jenkins / JFrog filestore
Beyond StorageClass and PriorityClass, `manifests/` also holds **per-env, one-shot** PV/PVC pairs:
```
manifests/jenkins-filestore-caching/
dev/{pv,pvc}.yaml
prd/{pv,pvc}.yaml
manifests/jenkins-gcs-caching/{pv,pvc,sc-gcs}.yaml
manifests/jfrog-filestore-data/
dev/...
prd/...
```
These are **already-created PVs being adopted** — typically because the underlying disk (Filestore mount, GCS bucket) was provisioned by Terraform or by hand. The PV file binds the existing disk to the cluster; the PVC file binds a workload to the PV.
| Rule | Why |
|------|-----|
| **Don't change `spec.csi.volumeHandle` / `spec.gcePersistentDisk.pdName`** without coordinating with Terraform. The handle is the disk identity. |
| **Don't change `spec.persistentVolumeReclaimPolicy`** from `Retain` to `Delete` on these. The disk has data on it. |
| **Don't move dev→prd or vice versa** — they reference different physical disks. |
| **Treat as platform-team review.** Any change here is a 1:1 disk operation. |
---
## Validation script
```bash
# StorageClass: every name in helm-overrides has a file here
for sc in $(grep -rhE 'storageClass(Name)?:' helm-overrides \
| sed -E 's/.*storageClass(Name)?:\s*//' \
| tr -d '"' | sort -u); do
[ -f "manifests/storageclass/${sc}.yaml" ] || echo "MISSING: $sc"
done
# PriorityClass: every name referenced has a file in the matching cluster
grep -rE 'priorityClassName:' helm-overrides
# (cross-reference manually against manifests/priorityclass/<cluster>/)
```