192 lines
7.3 KiB
Markdown
192 lines
7.3 KiB
Markdown
# Schema — Raw-manifest sidecars in `helm-overrides/`
|
|
|
|
> Field-by-field guidance for `helm-overrides/<cluster>/<app>/<extra>.yaml` — Kubernetes manifests applied **alongside** a Helm release, not through it.
|
|
|
|
---
|
|
|
|
## What this is
|
|
|
|
Some Argo CD Applications point at a directory containing both a Helm `custom-values.yaml` *and* one or more raw Kubernetes manifests. The raw manifests are not Helm-rendered; Argo CD applies them as-is.
|
|
|
|
Common shapes seen in this repo:
|
|
|
|
| Path pattern | What it is |
|
|
|--------------|------------|
|
|
| `helm-overrides/<cluster>/<app>/computeclass/*-cc.yaml` | GKE Autopilot `ComputeClass` resource — declares a node-pool/compute-class profile referenced by `nodeSelector` in the Helm values. |
|
|
| `helm-overrides/<cluster>/<app>/external-dns-services/<svc>.yaml` | A `Service` that exists purely to carry an `external-dns.alpha.kubernetes.io/hostname` annotation, binding a DNS name to a workload. |
|
|
| `helm-overrides/<cluster>/elastic-cluster/argo-launch.yaml` | An `ElasticCluster` (ECK CRD) launched alongside the operator. |
|
|
| `helm-overrides/<cluster>/<app>/mimir-distributed/alertmanager_config.yaml` | Inlined Alertmanager config materialised as a `ConfigMap`. |
|
|
|
|
If the sister-repo `Application` for this directory has `path: helm-overrides/<cluster>/<app>/`, then **every YAML file in the directory is applied** — Argo CD's directory loader treats them as a single deployment unit.
|
|
|
|
---
|
|
|
|
## File-level conventions
|
|
|
|
| Rule | Why |
|
|
|------|-----|
|
|
| **One Kubernetes resource per file** unless they're tightly coupled (e.g. a `Service` + a `ServiceAccount` referenced by it). | Reviewer cognition; rollback granularity. |
|
|
| **Pin `apiVersion` explicitly.** | `extensions/v1beta1` and `networking.k8s.io/v1beta1` are real footguns — both are gone in current Kubernetes. |
|
|
| **Set `metadata.namespace` explicitly.** | Don't rely on the Argo `Application.spec.destination.namespace` carrying through — different cluster Argos behave differently here. |
|
|
| **Trailing newline at EOF.** |
|
|
| **No multi-doc (`---`) within one file** unless genuinely required. |
|
|
|
|
---
|
|
|
|
## Top-level shape
|
|
|
|
```yaml
|
|
apiVersion: <api-version> # e.g. v1, networking.k8s.io/v1, autoscaling.gke.io/v1
|
|
kind: <kind> # e.g. Service, ComputeClass, ConfigMap, ExternalSecret
|
|
metadata:
|
|
name: <name>
|
|
namespace: <namespace>
|
|
labels: {...} # optional
|
|
annotations: {...} # often the load-bearing field (DNS, LB)
|
|
spec:
|
|
... # kind-specific
|
|
```
|
|
|
|
---
|
|
|
|
## Common kinds in this repo
|
|
|
|
### `ComputeClass` (`autoscaling.gke.io/v1`)
|
|
|
|
GKE Autopilot uses `ComputeClass` resources to declare named node profiles. The Helm values reference them via `cloud.google.com/compute-class:` keys.
|
|
|
|
```yaml
|
|
apiVersion: autoscaling.gke.io/v1
|
|
kind: ComputeClass
|
|
metadata:
|
|
name: contour-internal-0-cc
|
|
namespace: projectcontour
|
|
spec:
|
|
priorities:
|
|
- machineFamily: c4d
|
|
minCores: 4
|
|
minMemory: 16Gi
|
|
nodePoolAutoCreation:
|
|
enabled: true
|
|
```
|
|
|
|
| Rule | Why |
|
|
|------|-----|
|
|
| **`metadata.name` MUST match the value referenced by `nodeSelector` in the Helm values.** | Otherwise pods stay `Pending`. |
|
|
| **Per-cluster.** A `ComputeClass` for `k8s-central-prd-ase1` does not exist on `k8s-supply-prd-ase1`. Don't share files. |
|
|
| **The matrix of which `ComputeClass` exists where** is recorded in [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md). |
|
|
|
|
### `Service` for `external-dns` binding
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: <hostname>-dns
|
|
namespace: <ns>
|
|
annotations:
|
|
external-dns.alpha.kubernetes.io/hostname: <fqdn>.meeshogcp.in
|
|
cloud.google.com/load-balancer-type: Internal
|
|
spec:
|
|
type: ClusterIP
|
|
selector: {app.kubernetes.io/name: <app>}
|
|
ports:
|
|
- port: 80
|
|
targetPort: 8080
|
|
```
|
|
|
|
| Rule | Why |
|
|
|------|-----|
|
|
| **`metadata.annotations.external-dns.alpha.kubernetes.io/hostname`** is the contract. Check the cluster's existing DNS pattern before authoring. |
|
|
| **`selector` must match labels of an actual workload Pod** in the same namespace. |
|
|
| **Cloud-LB annotations** (`cloud.google.com/load-balancer-type`, etc.) — copy from a sibling on the same cluster. |
|
|
|
|
### `ExternalSecret` (per-cluster `external-secrets/`)
|
|
|
|
```yaml
|
|
apiVersion: external-secrets.io/v1beta1
|
|
kind: ExternalSecret
|
|
metadata:
|
|
name: <secret-name>
|
|
namespace: <ns>
|
|
spec:
|
|
refreshInterval: 1h
|
|
secretStoreRef:
|
|
name: gcp-sm-store # cluster-level SecretStore, defined elsewhere
|
|
kind: ClusterSecretStore
|
|
target:
|
|
name: <secret-name>
|
|
creationPolicy: Owner
|
|
data:
|
|
- secretKey: <env-key>
|
|
remoteRef:
|
|
key: <gcp-secret-name>
|
|
version: latest
|
|
```
|
|
|
|
| Rule | Why |
|
|
|------|-----|
|
|
| **`spec.target.name` is the `Secret` name** the workload's `existingSecret:` references. |
|
|
| **`spec.secretStoreRef.name`** must reference a `SecretStore` / `ClusterSecretStore` that already exists on the cluster. |
|
|
| **`creationPolicy: Owner`** is the default; `creationPolicy: Merge` is for the rare case where another tool also manages the same `Secret`. |
|
|
|
|
### `ConfigMap` for inlined config
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: <name>
|
|
namespace: <ns>
|
|
data:
|
|
<key>: |
|
|
<multi-line content>
|
|
```
|
|
|
|
| Rule | Why |
|
|
|------|-----|
|
|
| **Use `|` (literal block scalar)** for multi-line content with significant whitespace. |
|
|
| **Don't inline secrets** (TruffleHog will catch obvious ones; subtle ones slip through). |
|
|
| **Reference from the workload's chart values** via the `ConfigMap` name; don't duplicate config across files. |
|
|
|
|
### `ElasticCluster` / other CRDs
|
|
|
|
For ECK, Pyroscope, etc. — the schema is the operator's, not Kubernetes's. **Read the operator's CRD docs** before authoring; copy a sibling cluster's existing `argo-launch.yaml` first.
|
|
|
|
---
|
|
|
|
## Layered-with-Helm vs standalone
|
|
|
|
| Pattern | Argo Application points at |
|
|
|---------|----------------------------|
|
|
| Pure Helm | `path: helm-overrides/<cluster>/<app>/` with `helm.valueFiles: ['custom-values.yaml']` |
|
|
| Pure raw manifests | `path: helm-overrides/<cluster>/<app>/` with `directory.recurse: true`, no `helm:` block |
|
|
| Layered | `path: helm-overrides/<cluster>/<app>/` with `helm.valueFiles: ['custom-values.yaml']` AND extra files alongside |
|
|
|
|
Argo CD's behaviour for layered directories depends on its `directory.include` / `directory.exclude` settings — when in doubt, read the sister-repo `Application` to see exactly what gets picked up.
|
|
|
|
---
|
|
|
|
## Validation
|
|
|
|
```bash
|
|
# Validate the manifest against its API
|
|
kubectl --dry-run=client -f helm-overrides/<cluster>/<app>/<extra>.yaml apply
|
|
|
|
# Lint
|
|
yamllint helm-overrides/<cluster>/<app>/<extra>.yaml
|
|
|
|
# Confirm the Argo Application loads this path
|
|
# (read the matching file in github.com/Meesho/devops-infra-argo-config)
|
|
```
|
|
|
|
---
|
|
|
|
## Anti-patterns
|
|
|
|
1. **Inlining secrets** in a `ConfigMap` "to ship a hotfix." It's still a secret. Use `ExternalSecret`.
|
|
2. **Using `apiVersion: v1beta1`** of a CRD whose stable version exists. Pin to the highest stable.
|
|
3. **A `Service` whose `selector` doesn't match any Pod** — silent: the Service exists, DNS resolves, no endpoints.
|
|
4. **Cross-cluster cloning** of a `ComputeClass` or DNS `Service` without rewriting cluster-specific fields.
|
|
5. **Hand-rendered Helm output** dropped into a sidecar file (a release "freeze"). The chart bumps and your snapshot rots.
|