159 lines
7.3 KiB
Markdown
159 lines
7.3 KiB
Markdown
# Schema — `manifests/storageclass/` and `manifests/priorityclass/`
|
|
|
|
> Cluster-wide singletons. **High blast radius — every PVC / scheduling decision in the cluster is affected.**
|
|
>
|
|
> Cross-reference: [SANCTITY_RULES R10](../../global/SANCTITY_RULES.md), [coding-guidelines/helm-values.md §persistence](../../global/coding-guidelines/helm-values.md).
|
|
|
|
---
|
|
|
|
## §StorageClass (`manifests/storageclass/*.yaml`)
|
|
|
|
These are **repo-global** — one StorageClass file is applied to every cluster that consumes it. A wrong reclaim policy or volume-binding mode breaks every new PVC.
|
|
|
|
### Inventory
|
|
|
|
| File | Backend | Reclaim | Binding | Use case |
|
|
|------|---------|---------|---------|----------|
|
|
| `pd-standard-retain-dr.yaml` | GCP PD standard | `Retain` | `WaitForFirstConsumer` | DR-critical state — do not lose data on PVC delete |
|
|
| `sc-pd-ssd.yaml` | GCP PD SSD | `Delete` | `WaitForFirstConsumer` | Default for write-heavy (etcd, ClickHouse) |
|
|
| `sc-pd-standard.yaml` | GCP PD standard | `Delete` | `WaitForFirstConsumer` | Default for read-heavy / archival |
|
|
| `sc-filestore-standard.yaml` | GCP Filestore | varies | varies | Shared `ReadWriteMany` volumes (Jenkins build cache, JFrog filestore) |
|
|
|
|
### Schema
|
|
|
|
```yaml
|
|
apiVersion: storage.k8s.io/v1
|
|
kind: StorageClass
|
|
metadata:
|
|
name: sc-pd-ssd # MUST match filename
|
|
annotations:
|
|
storageclass.kubernetes.io/is-default-class: "false" # exactly one default per cluster
|
|
provisioner: pd.csi.storage.gke.io
|
|
parameters:
|
|
type: pd-ssd
|
|
reclaimPolicy: Delete # Delete | Retain
|
|
volumeBindingMode: WaitForFirstConsumer
|
|
allowVolumeExpansion: true
|
|
```
|
|
|
|
| Field | Convention | Notes |
|
|
|-------|------------|-------|
|
|
| `metadata.name` | matches filename (without `.yaml`) | Helm values reference by name. |
|
|
| `metadata.annotations."storageclass.kubernetes.io/is-default-class"` | `"false"` for all of these | Exactly one StorageClass should be `"true"` per cluster (declared elsewhere, often by Terraform/cluster-bootstrap). |
|
|
| `provisioner` | `pd.csi.storage.gke.io` (PD) or `filestore.csi.storage.gke.io` (Filestore) | GCP CSI drivers. |
|
|
| `parameters.type` | `pd-ssd`, `pd-standard`, `pd-balanced` | Cost/perf trade-off. |
|
|
| `reclaimPolicy` | `Delete` (default) or `Retain` (DR) | **Changing this on an existing StorageClass does not retroactively change PVCs.** |
|
|
| `volumeBindingMode` | `WaitForFirstConsumer` | Avoids zone-mismatch on multi-zone clusters. `Immediate` only for `ReadWriteMany`. |
|
|
| `allowVolumeExpansion` | `true` | Required for online resizing. Default false on some classes — set explicitly. |
|
|
|
|
### Hard rules
|
|
|
|
1. **Never delete a StorageClass referenced by an existing PVC.** New PVCs against the deleted class fail; existing bound PVCs survive but lose the ability to expand.
|
|
2. **Never change `reclaimPolicy` from `Delete` → `Retain`** as a "safety improvement" without auditing every PVC. Existing bound PVs keep their old policy; new PVs get the new policy. Drift.
|
|
3. **Never change `provisioner`** — that's a delete-and-recreate, not an edit. Existing PVs become orphaned.
|
|
4. **Never make a new StorageClass the cluster default** without coordinating with the platform team. A wrong default class hijacks every PVC that doesn't pin a class explicitly.
|
|
5. **Validation:** every StorageClass referenced by any `helm-overrides/*/persistence.storageClass:` must have a corresponding file here.
|
|
|
|
```bash
|
|
# Find every PVC reference
|
|
grep -rE 'storageClass(Name)?:' helm-overrides | sort -u
|
|
|
|
# Find every StorageClass file
|
|
ls manifests/storageclass/*.yaml
|
|
```
|
|
|
|
---
|
|
|
|
## §PriorityClass (`manifests/priorityclass/<cluster>/*.yaml`)
|
|
|
|
**Per-cluster** but **cluster-wide** — affects scheduling priority for every pod that references the class.
|
|
|
|
### Inventory pattern
|
|
|
|
```
|
|
manifests/priorityclass/
|
|
k8s-central-prd-ase1/
|
|
priorityclass-high.yaml
|
|
priorityclass-low.yaml
|
|
k8s-supply-prd-ase1/
|
|
priorityclass-high.yaml
|
|
priorityclass-low.yaml
|
|
…
|
|
```
|
|
|
|
Most BU clusters get a `high` and a `low` class. Specialty clusters (`mqkafka`, `dsgpu`, dataplane `db-*`) sometimes have additional tiers.
|
|
|
|
### Schema
|
|
|
|
```yaml
|
|
apiVersion: scheduling.k8s.io/v1
|
|
kind: PriorityClass
|
|
metadata:
|
|
name: high-priority
|
|
value: 1000000 # higher = more important
|
|
globalDefault: false # NEVER true on these — would clash with system defaults
|
|
description: "High-priority workloads — promoted ahead of general pods on resource pressure"
|
|
preemptionPolicy: PreemptLowerPriority # default; alternative is Never
|
|
```
|
|
|
|
| Field | Convention | Notes |
|
|
|-------|------------|-------|
|
|
| `metadata.name` | `high-priority`, `low-priority`, etc. | Workload pods reference by name. |
|
|
| `value` | High: 1,000,000 / Low: 100 / Critical (rare): >1,000,000,000 | Kubernetes system pods reserve `> 2,000,000,000`; don't conflict. |
|
|
| `globalDefault` | **always `false`** | A `true` here applies to every pod that doesn't pin a class — chaos. |
|
|
| `description` | one-line | For audit log. |
|
|
| `preemptionPolicy` | `PreemptLowerPriority` (default) or `Never` | `Never` for batch workloads that shouldn't kick others out. |
|
|
|
|
### Hard rules
|
|
|
|
1. **Never set `globalDefault: true`** on any PriorityClass here. The cluster's implicit default is what we want.
|
|
2. **Never raise `value`** of an existing class without auditing what gets preempted. A bump from 1,000,000 → 10,000,000 changes which pods get evicted under pressure.
|
|
3. **Never delete a PriorityClass** referenced by any workload — pods that referenced it become invalid.
|
|
4. **The same `name` must mean the same thing across clusters.** `high-priority` on supply ≠ `high-priority` on demand at the *value* level is an audit nightmare. Keep values consistent.
|
|
5. **Validation:** find every pod-spec reference:
|
|
```bash
|
|
grep -rE 'priorityClassName:' helm-overrides
|
|
```
|
|
|
|
---
|
|
|
|
## §The "PV/PVC singletons" — Jenkins / JFrog filestore
|
|
|
|
Beyond StorageClass and PriorityClass, `manifests/` also holds **per-env, one-shot** PV/PVC pairs:
|
|
|
|
```
|
|
manifests/jenkins-filestore-caching/
|
|
dev/{pv,pvc}.yaml
|
|
prd/{pv,pvc}.yaml
|
|
manifests/jenkins-gcs-caching/{pv,pvc,sc-gcs}.yaml
|
|
manifests/jfrog-filestore-data/
|
|
dev/...
|
|
prd/...
|
|
```
|
|
|
|
These are **already-created PVs being adopted** — typically because the underlying disk (Filestore mount, GCS bucket) was provisioned by Terraform or by hand. The PV file binds the existing disk to the cluster; the PVC file binds a workload to the PV.
|
|
|
|
| Rule | Why |
|
|
|------|-----|
|
|
| **Don't change `spec.csi.volumeHandle` / `spec.gcePersistentDisk.pdName`** without coordinating with Terraform. The handle is the disk identity. |
|
|
| **Don't change `spec.persistentVolumeReclaimPolicy`** from `Retain` to `Delete` on these. The disk has data on it. |
|
|
| **Don't move dev→prd or vice versa** — they reference different physical disks. |
|
|
| **Treat as platform-team review.** Any change here is a 1:1 disk operation. |
|
|
|
|
---
|
|
|
|
## Validation script
|
|
|
|
```bash
|
|
# StorageClass: every name in helm-overrides has a file here
|
|
for sc in $(grep -rhE 'storageClass(Name)?:' helm-overrides \
|
|
| sed -E 's/.*storageClass(Name)?:\s*//' \
|
|
| tr -d '"' | sort -u); do
|
|
[ -f "manifests/storageclass/${sc}.yaml" ] || echo "MISSING: $sc"
|
|
done
|
|
|
|
# PriorityClass: every name referenced has a file in the matching cluster
|
|
grep -rE 'priorityClassName:' helm-overrides
|
|
# (cross-reference manually against manifests/priorityclass/<cluster>/)
|
|
```
|