196 lines
11 KiB
Markdown
196 lines
11 KiB
Markdown
> Per AI Blitz Plan §platform.schemas. Layer: 1. Repo: devops-infra-helm-charts.
|
||
|
||
# Schema — Incubator-tool values + sidecar contract
|
||
|
||
> **Scope:** the values-side contract for incubator (newly-vendored) infrastructure tools that live under `helm-overrides/<cluster>/<incubator-tool>/`.
|
||
>
|
||
> **Out of scope (explicit):** the Argo CD `Application` / `ApplicationSet` manifest that registers the incubator tool with a cluster's Argo. Those manifests live in the **sister repo** `github.com/Meesho/devops-infra-argo-config`, NOT here. This file documents only what `devops-infra-helm-charts` is responsible for: the values + raw sidecar manifests Argo CD reads from this repo.
|
||
|
||
This schema complements [custom-values-schema.md](custom-values-schema.md) (general values shape) and [raw-manifest-sidecar-schema.md](raw-manifest-sidecar-schema.md) (sidecar manifest shapes). Read both first.
|
||
|
||
---
|
||
|
||
## What "incubator" means here
|
||
|
||
A tool is **incubator** while it is being trialled on one or two clusters before fleet-wide rollout. In this repo that maps to:
|
||
|
||
- A new chart directory under `helm-templates/<incubator-tool>/` (often a thin wrapper `Chart.yaml` with an upstream dep).
|
||
- An override under `helm-overrides/<cluster>/<incubator-tool>/` for the trial cluster(s) only — typically `k8s-shared-int-ase1` (integration / pre-prod) first, sometimes one BU prod cluster as a canary.
|
||
- Possibly raw sidecar manifests alongside the values file.
|
||
|
||
Once the tool graduates from incubator status, the override pattern is identical to other infra tools — the "incubator" label is operational, not structural. This schema codifies the conventions that keep early-stage adoption sane.
|
||
|
||
---
|
||
|
||
## Directory layout
|
||
|
||
```
|
||
helm-overrides/<cluster>/<incubator-tool>/
|
||
├── custom-values.yaml # Helm values (required if chart is Helm-based)
|
||
├── computeclass/ # Optional — GKE Autopilot ComputeClass
|
||
│ └── <name>-cc.yaml
|
||
├── external-dns-services/ # Optional — Service for external-dns annotation
|
||
│ └── <fqdn>.yaml
|
||
├── external-secrets/ # Optional — ExternalSecret resources
|
||
│ └── <name>.yaml
|
||
└── <incubator-specific>.yaml # Tool-specific raw manifest (CRD, ConfigMap)
|
||
```
|
||
|
||
The Argo `Application` for this directory (sister repo) determines whether files are Helm-rendered, raw-applied, or layered (see [raw-manifest-sidecar-schema.md §Layered-with-Helm vs standalone](raw-manifest-sidecar-schema.md)).
|
||
|
||
---
|
||
|
||
## `custom-values.yaml` keys an incubator chart should expose
|
||
|
||
A chart promoted to incubator status MUST surface (i.e. let the override file set without forking templates) at minimum:
|
||
|
||
| Key | Why required |
|
||
|-----|--------------|
|
||
| `image.registry`, `image.repository`, `image.tag` | Production overrides pin to `asia-southeast1-docker.pkg.dev/meesho-devops-admin-0622/admin/sre/<image>` (Meesho Artifact Registry mirror, not Docker Hub). |
|
||
| `image.pullPolicy` | Default `IfNotPresent`. |
|
||
| `resources.requests.cpu`, `.memory` | No defaults assumed; Autopilot scheduling requires explicit requests. |
|
||
| `resources.limits.cpu`, `.memory` | Same. |
|
||
| `nodeSelector` | Per-cluster scheduling (see [custom-values-schema.md](custom-values-schema.md)). |
|
||
| `tolerations` | Per-cluster taints. |
|
||
| `affinity` (optional) | Pod anti-affinity for replicated tools. |
|
||
| `serviceAccount.create`, `.name`, `.annotations` | Workload Identity — `iam.gke.io/gcp-service-account` annotation goes here. |
|
||
| `replicaCount` (or `autoscaling.{enabled,minReplicas,maxReplicas}`) | Sizing. |
|
||
| `persistence.enabled`, `.storageClass`, `.size` | If stateful. `storageClass` MUST exist in `manifests/storageclass/`. See [storageclass-priorityclass-schema.md](storageclass-priorityclass-schema.md). |
|
||
| `priorityClassName` | If the tool needs preemption priority — must reference a class in `manifests/priorityclass/<cluster>/`. |
|
||
| `podLabels`, `podAnnotations` | For prometheus.io scrape annotations and team-attribution labels. |
|
||
| `extraEnv` (or `env`) | For environment-specific knobs the chart's templates don't already accept. |
|
||
|
||
If the upstream chart doesn't expose these — that's a chart-bug. **Do not** fork the chart in `helm-templates/` to add them ([NEVER-DO list](../../../CLAUDE.md)). Either upstream-PR the chart or wrap with a small Meesho-owned chart that re-exposes the keys.
|
||
|
||
---
|
||
|
||
## Values-file conventions for incubator tools
|
||
|
||
1. **Pin every image tag.** Never `latest`, never an unpinned SHA. This is the same rule as production overrides; incubator status does not relax it.
|
||
2. **Set explicit `nodeSelector` and `tolerations`** authored from scratch using sibling apps on the same cluster. Cross-cluster copying is forbidden — see [custom-values-schema.md](custom-values-schema.md).
|
||
3. **Use `fullnameOverride` deliberately or not at all.** Once set, do not change ([NEVER-DO list](../../../CLAUDE.md)). Incubator graduations to other clusters should reuse the same `fullnameOverride` string for portability.
|
||
4. **Don't enable `autoscaling`** in the first incubator deploy. Pin `replicaCount: 1` (or 2 for HA-mandatory) until you have a load profile.
|
||
5. **Don't expose the tool externally** in the incubator phase. No `external-dns-services/` until it's promoted to a cluster's stable inventory.
|
||
6. **Annotate the values file** with a top-of-file YAML comment: `# Incubator: cluster=<...>, owner=<...>, graduation-target=<date>`. Comment is plain text; not parsed; serves as reviewer signal.
|
||
7. **Confine secrets to `ExternalSecret`** under `external-secrets/`. No inline `existingSecret:` referencing a hand-applied Secret.
|
||
|
||
---
|
||
|
||
## Raw sidecar manifests in incubator directories
|
||
|
||
Concrete shapes already documented elsewhere; this section calls out which ones routinely appear with incubators.
|
||
|
||
### `computeclass/*-cc.yaml` (GKE Autopilot only)
|
||
|
||
Required when the override's `nodeSelector` uses `cloud.google.com/compute-class: <name>` and that class doesn't already exist on the cluster. The `metadata.name` MUST equal the value referenced. See [raw-manifest-sidecar-schema.md §ComputeClass](raw-manifest-sidecar-schema.md).
|
||
|
||
Example incubator pattern:
|
||
|
||
```yaml
|
||
apiVersion: autoscaling.gke.io/v1
|
||
kind: ComputeClass
|
||
metadata:
|
||
name: <incubator-tool>-cc
|
||
namespace: <ns>
|
||
spec:
|
||
priorities:
|
||
- machineFamily: c4d
|
||
minCores: 2
|
||
minMemory: 8Gi
|
||
nodePoolAutoCreation:
|
||
enabled: true
|
||
```
|
||
|
||
Per-cluster only — never copy between clusters.
|
||
|
||
### `external-dns-services/*.yaml`
|
||
|
||
Skip in the first incubator deploy. Only add once the tool has a stable internal-only DNS need. See [raw-manifest-sidecar-schema.md §Service for external-dns binding](raw-manifest-sidecar-schema.md).
|
||
|
||
### `external-secrets/*.yaml`
|
||
|
||
Always present if the tool needs secrets. Use the existing per-cluster `SecretStore` / `ClusterSecretStore`; do not author new stores in an incubator PR. See [raw-manifest-sidecar-schema.md §ExternalSecret](raw-manifest-sidecar-schema.md).
|
||
|
||
### `elastic-cluster/argo-launch.yaml`-style operator-managed CRD launches
|
||
|
||
If the incubator tool is an operator (e.g. ECK, Pyroscope), a sidecar CRD instance often lives alongside the operator's chart values:
|
||
|
||
```
|
||
helm-overrides/<cluster>/<incubator-operator>/
|
||
├── custom-values.yaml # operator chart values
|
||
└── <crd-instance>/
|
||
└── argo-launch.yaml # the actual workload CRD instance
|
||
```
|
||
|
||
Argo CD applies both in one Application (directory loader). The CRD instance must:
|
||
|
||
- Reference an `apiVersion` whose CRD the operator has already installed.
|
||
- Pin its own image / version explicitly.
|
||
- Avoid hand-rolling values that the operator templates would otherwise compute.
|
||
|
||
---
|
||
|
||
## Argo CD — sister repo contract (for cross-reference only)
|
||
|
||
The Application that points at this directory lives in `github.com/Meesho/devops-infra-argo-config`. For incubator tools the Application typically:
|
||
|
||
- Has `syncPolicy.automated` **disabled** (manual sync — incubator-grade safety; see [argocd.md](../../global/coding-guidelines/argocd.md)).
|
||
- Is named `<incubator-tool>-<cluster>` to be unambiguous.
|
||
- Lives in the cluster's namespace under `apps/<cluster>/`.
|
||
|
||
This file does NOT instruct on authoring the Application — that PR is in the sister repo. Cross-link the sister-repo PR in the values-side PR description.
|
||
|
||
---
|
||
|
||
## Validation
|
||
|
||
Same as any Helm override. Before opening a PR:
|
||
|
||
```bash
|
||
yamllint helm-overrides/<cluster>/<incubator-tool>/custom-values.yaml
|
||
|
||
helm template <release> helm-templates/<incubator-tool> \
|
||
-f helm-overrides/<cluster>/<incubator-tool>/custom-values.yaml > /tmp/render.yaml
|
||
|
||
# Validate any sidecar manifests against the cluster's CRDs
|
||
kubectl --context=<ctx> --dry-run=server \
|
||
-f helm-overrides/<cluster>/<incubator-tool>/<sidecar>.yaml apply
|
||
```
|
||
|
||
For tool-specific CRDs that aren't in vanilla Kubernetes, prefer `--dry-run=server` (the cluster validates against the registered CRD) over `--dry-run=client` (only client-side schema, often misses required fields).
|
||
|
||
---
|
||
|
||
## Graduation: incubator → stable
|
||
|
||
When the tool has been stable for some period (usually 2–4 weeks) and is ready to fleet-roll:
|
||
|
||
1. Remove the top-of-file `# Incubator:` comment.
|
||
2. Open new override directories in target clusters — author from scratch each time, do not copy.
|
||
3. Open the matching sister-repo `Application` PRs (one per new cluster) or extend the `ApplicationSet`.
|
||
4. If the chart was a thin wrapper, audit `Chart.yaml` `dependencies[].version` is current.
|
||
5. Promote the chart's documentation in `helm-templates/<incubator-tool>/README.md` (if a fork was needed) or note in PR description that no fork was needed.
|
||
|
||
---
|
||
|
||
## Anti-patterns
|
||
|
||
1. **Forking the chart in `helm-templates/<incubator-tool>/templates/`** to add a missing values key. Wrap or upstream-PR; don't fork. See NEVER-DO.
|
||
2. **Copying the entire incubator directory across clusters** for graduation. Per-cluster scheduling is bespoke.
|
||
3. **Enabling autoscaling on day one.** No load profile = thrashy autoscaler.
|
||
4. **Promoting from `k8s-shared-int-ase1` straight to a critical BU cluster.** Insert a low-traffic prod canary first.
|
||
5. **Inlining secrets** "just for the trial." TruffleHog will block; even if it didn't, the leak is real.
|
||
6. **Setting `fullnameOverride`** without a graduation plan. The string travels — bad ones travel forever.
|
||
|
||
---
|
||
|
||
## Related
|
||
|
||
- Schema: [custom-values-schema.md](custom-values-schema.md) — general values shape.
|
||
- Schema: [raw-manifest-sidecar-schema.md](raw-manifest-sidecar-schema.md) — sidecar shapes.
|
||
- Schema: [storageclass-priorityclass-schema.md](storageclass-priorityclass-schema.md) — cluster-singleton resources.
|
||
- Procedure: [../procedures/onboard-app-to-cluster.md](../procedures/onboard-app-to-cluster.md) — the procedure used to land an incubator override.
|
||
- Procedure: [../procedures/fork-upstream-chart.md](../procedures/fork-upstream-chart.md) — only when a fork is genuinely required.
|
||
- Coding guideline: [../../global/coding-guidelines/helm-values.md](../../global/coding-guidelines/helm-values.md).
|
||
- Coding guideline: [../../global/coding-guidelines/argocd.md](../../global/coding-guidelines/argocd.md) — sister-repo Application conventions.
|