A new cluster directory rather than edits to k8s-admin-prd-ase1, so no
homelab value is ever reused for GCP by accident. Charts are the ones
already vendored here (gitea 12.7.0, argo-cd 7.7.23, cert-manager
v1.20.1); only the values are new. Verified with helm template.
What differs from the homelab, and why:
- gitea: storageClass standard-rwo, and Recreate for a different reason
than the homelab's LevelDB lock — three nodes and a ReadWriteOnce disk
mean a rolling update's new pod waits forever on Multi-Attach. The
admin password comes from a Secret created at bootstrap instead of the
chart's published default, which would otherwise be live on a public
IP. Registration is disabled and webhooks are limited to private
ranges, for the same reason.
- argocd: single ingress host (no Tailscale), and the homelab's Ingress
health override is dropped, since Contour writes real load balancer
status here. server and repoServer autoscale 1-3 on CPU; the chart
omits replicas when autoscaling is on, so the HPA and ArgoCD's own
self-management do not fight over the count. Memory is deliberately
not a scaling metric: Go does not return memory promptly, so a memory
target scales up and never back down.
- cert-manager: written fresh, not copied. The homelab file was never
adapted from the fleet — it pulls from a private Meesho registry and
pins pods to a node pool that does not exist here. The chart's own
values.yaml carries that registry too, so imageRegistry and
imageNamespace are overridden back to upstream's quay.io/jetstack.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
toolshed provisions a per-app Redis ACL user, scoped to its own key
prefix, on request (internal/dbprovision.EnsureRedisUser). Nothing in this
cluster ran Redis — Harbor's internal one is Harbor's and is not ACL
configured — so there was nowhere for that to point.
Hand-written rather than vendoring Bitnami's, same reasoning as the
sibling postgresql chart: Broadcom has been retiring and freezing images
behind that repo (infra issue #4, where it broke Contour twice), and Redis
publishes no official chart either.
The authentication design is the part worth reading before changing
anything. Redis is started with an ACL file and NO requirepass, and that
distinction is a security property rather than a style choice:
- toolshed persists provisioned users with ACL SAVE, which requires an
aclfile. Without it every provisioned user is lost on the next restart.
- But ACL SAVE also serialises the default user. With requirepass, the
saved entry comes back as `user default on nopass ~* &* +@all`, and
after the next restart the ACL file wins — leaving Redis open to
UNAUTHENTICATED access with full permissions. Verified directly: with
requirepass, the restarted server answered an unauthenticated PING with
PONG and served a key.
So the default user is defined in the ACL file instead, seeded once by an
init container that deliberately never overwrites an existing file —
overwriting would delete every user toolshed had provisioned into it,
reintroducing the same lockout from the other end. The documented
consequence is that rotating the admin password in Vault does not
propagate on its own; that needs ACL SETUSER default + ACL SAVE against
the running server.
Sized for a node at its ceiling: 32Mi requested, 96Mi limit, maxmemory
48mb. The limit sits above maxmemory on purpose, so Redis reaches its own
eviction policy rather than being OOM-killed, which would lose the whole
instance instead of the coldest keys. Snapshotting is off — what must
survive a restart is the ACL file, which ACL SAVE writes independently of
RDB, and cached values are by definition reconstructible.
allkeys-lru because this backs a connection kind called "cache" and
eviction under pressure is that contract; values.yaml says plainly that an
app using Redis as its only copy of something wants noeviction instead.
Verified: helm template, then a real deploy to a k3d cluster — provisioned
users through toolshed's own code, deleted the pod, and confirmed all five
came back with their key patterns intact, the init container declined to
overwrite, unauthenticated access got NOAUTH, and a user writing outside
its prefix got NOPERM.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Provisioned rather than built by hand in Grafana's UI — same reasoning
as the datasource: survives a pod restart, and a git diff shows what
changed. Four rows: ingress (Envoy total RPS + connections + response
class breakdown), service level (CPU/memory by namespace, filterable
via a $namespace template variable, plus a current-usage table), cluster
utilization (used vs actual node capacity, not an assumed limit), and
total resources (cores/memory/pods/disk).
Two scrape gaps found and fixed to make this possible, both in vmagent:
- Contour's own ingress Envoy (projectcontour namespace — the actual
data plane for everything routed through this homelab, hostPort
80/443) was not being scraped at all. Confirmed live: Cilium's
separate embedded Envoy (kube-system, its own L7 policy proxy) was
already flowing, via the annotation-based kubernetes-pods job — which
is what first showed envoy_* metrics existed in this cluster at all —
but Contour's Envoy carries no such annotation. Added an explicit job
targeting the projectcontour namespace by container port (8002, the
official chart's fixed Envoy metrics port) rather than guessing at
pod labels this cluster's auto-detected object names may not match.
- node-exporter, deployed two commits ago, was never actually being
scraped either: confirmed live that kubernetes-service-endpoints
(role: endpointslice, keyed on the Service's scrape annotation — where
that chart puts it) finds nothing in this cluster at all, not merely
down. Rather than chase why, added the same fix as Envoy: target the
pod directly by its declared container port (9100).
Verified against the live deployment (queried through vmui) before
writing a single panel: envoy_http_downstream_rq_total,
envoy_http_downstream_rq_xx, container_cpu_usage_seconds_total,
container_memory_working_set_bytes, machine_cpu_cores and
machine_memory_bytes all confirmed present with real data. The one
exception is the "Disk free" panel, which depends on the node-exporter
scrape fix landing in this same change — noted in the values file's own
comment as unverified until it actually deploys.
Also verified with `helm template`: the dashboard JSON round-trips
through the YAML values file and the chart's own ConfigMap templating
intact (19 panels both times), and vmagent's scrape_configs list still
carries all 8 chart defaults plus both new jobs — nothing lost by using
extraScrapeConfigs instead of overriding the full list by hand.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Grafana: real dashboards over VictoriaMetrics. Provisioned rather than
clicked through: a VictoriaMetrics datasource (type: prometheus — VM
speaks that query API, which is the whole point of it existing) is
baked into the chart's own datasource-provisioning config, so a fresh
install has it working with no manual setup. Admin credentials from
Vault via ExternalSecret (secretstores/grafana-admin-credentials.yaml
in devops-infra-argo-config), same pattern as every other admin
credential in this project — never plaintext in this repo. 1Gi
local-path PVC for dashboards/Grafana's own state (VictoriaMetrics
holds the actual metric data, not this). Dual LAN+Tailscale Ingress
hosts, same convention as everything externally reachable here.
Found and fixed while vendoring: helm-templates/grafana already held a
fully-vendored old Grafana chart (v6.58.7, appVersion 10.0.3) from the
original Meesho monorepo import (commit b8575bb) — generic production
config (fullnameOverride: grafana-infra-prd, GKE-shaped RBAC/PSP
defaults) unrelated to this homelab, same class of leftover as
victoria-metrics-single's collision two commits ago. Removed and
re-vendored fresh (10.5.15) as a thin wrapper, matching every other
official-chart component in this repo now.
vmui: VictoriaMetrics' own built-in UI (ad-hoc PromQL + graphs, no
saved dashboards — what Grafana is for) is served on the same
pod/port, so exposing it cost one ingress block on the
victoria-metrics-single values already committed. No new component,
no new RAM.
Verified with `helm template` against the real charts for both
components individually (Grafana: admin env vars correctly reference
the ExternalSecret's keys, datasource ConfigMap renders the intended
VictoriaMetrics URL, PVC/resources/ingress hosts all match; vmui:
ingress renders both hostnames pointing at the existing Service's named
http port) and again for the whole generic-argo-apps-chart appSpec
list — 12 Applications render, including grafana.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Lower RAM/disk for the same metric volume on an 8GB single node — VM's
own compression is the entire reason it exists as a project. Speaks
Prometheus's own query API (/api/v1/query), so nothing downstream
needed to change beyond the URL it points at: same PromQL, same
kubernetes-nodes-cadvisor-sourced container_cpu_usage_seconds_total /
container_memory_working_set_bytes toolshed's metrics work
(docs/PRODUCT-ARCHITECTURE.md step 5) already targets.
Three releases, matching the one-release-per-component convention
already used everywhere in this repo, since victoria-metrics-single has
no bundled scraper/exporter subcharts the way the Prometheus chart did:
- victoria-metrics-single: the TSDB + query engine. 3Gi/local-path,
7-day retention, resources capped at 512Mi — same trim reasoning as
the Prometheus server it replaces.
- vmagent: the scraper. Its default scrape_configs already includes
kubernetes-nodes-cadvisor and kubernetes-service-endpoints (the
chart's own comment says "COPY from Prometheus helm chart") — nothing
to override there, only remoteWrite pointed at victoria-metrics-single
and trimmed resources.
- node-exporter: pulled out of the removed Prometheus chart's bundled
subchart into its own standalone release, since victoria-metrics-single
has no equivalent. Same trim as before (32Mi/64Mi), same
prometheus.io/scrape annotation vmagent's default scrape config
already looks for.
Found and fixed while vendoring: helm-templates/victoria-metrics-single
already existed in this repo — a leftover GKE-targeted vendored copy
from the original Meesho monorepo import (commit b8575bb), with its own
templates/values.yaml for a different chart entirely. `mkdir -p` on an
already-existing directory silently did nothing, so the first vendoring
attempt left that old tree sitting alongside the new one and rendered
two ServiceAccounts/Services/StatefulSets with colliding names. Removed
outright rather than adapted, same reasoning as the ~35 charts
devops-helm-charts removed in its own prune — see this chart's own
Chart.yaml for the full note.
Verified with `helm template` against the real charts and these values
for all three releases individually, and again for the whole
generic-argo-apps-chart appSpec list (11 Applications render, including
the three new ones and nothing orphaned from the old prometheus entry).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
The other three disabled subcharts (alertmanager/kube-state-metrics/
pushgateway) stay off — none of them do anything cAdvisor's per-container
metrics don't already cover for this deployment's actual purpose.
node-exporter is different: it's the one thing here that would report on
the node itself (disk, memory, load) rather than any particular
container, which matters on an 8GB single node already near its ceiling.
One DaemonSet pod, hostNetwork rather than a Service, resources capped
at 32Mi/64Mi — light enough that turning it on doesn't meaningfully
touch the budget the other three would have.
Verified with `helm template`: the DaemonSet renders with hostNetwork/
hostPID true and the trimmed resources, and its Service carries the
`prometheus.io/scrape: true` annotation the server's already-enabled
kubernetes-service-endpoints job picks up automatically — no separate
scrape config needed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Vendored prometheus-community/prometheus 29.27.1 the same way
Contour/ArgoCD/Vault/Gitea/Harbor/Jenkins already are here — a thin
Chart.yaml dependency plus a committed .tgz — rather than hand-written
like postgresql, which has no official chart to vendor.
Server only: alertmanager, kube-state-metrics, prometheus-node-exporter
and prometheus-pushgateway are all enabled by default in this chart and
all disabled here. None are needed for what actually consumes this —
toolshed's per-app CPU/memory metrics read straight from the chart's
built-in kubernetes-nodes-cadvisor scrape job (kubelet's own cAdvisor
endpoint) — and each is its own pod on a node that was already at its
8GB ceiling before this.
Trimmed for the same ceiling: 3Gi PVC on local-path (not the chart's
8Gi default), 7-day retention (not 15), resources capped at 512Mi.
nameOverride pinned to exactly "prometheus" matters more here than for
any other component pinning it: the chart's server Service renders as
"<release-name>-server", so this is what makes it "prometheus-server"
— the exact hostname toolshed's PROMETHEUS_URL was already seeded to
point at, before this existed, so the connection is already correct
the day this deploys.
Verified with `helm template` against the real chart and these values:
server-only object set (ClusterRole, ClusterRoleBinding, ConfigMap,
Deployment, PVC, Service, ServiceAccount — nothing from the four
disabled subcharts), and the rendered PVC/retention/resources/Service
name all match what's written above.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Not Bitnami's: that registry has been actively unstable here (it broke
Contour twice, infra issue #4) and PostgreSQL publishes no official chart.
A single StatefulSet, PVC and Service is small enough that owning it costs
less than depending on an unstable repackage.
Credentials come from an existing Secret rather than being generated by
the chart — a chart that generates its own password regenerates it on
every render and silently locks you out of the existing volume.
Details that matter and are easy to get wrong:
- PGDATA is a subdirectory of the mount, not the mount itself. initdb
refuses to run in a directory that already has contents.
- Probes run through a shell. Kubernetes does not expand $(VAR) inside
exec probe commands, only in command/args.
- fsGroup 70 so the volume stays writable after the entrypoint drops
from root to the postgres user on the Alpine variant.
- shared_buffers cut to 32MB from PostgreSQL's 128MB default. The node
has 8GB and was at its ceiling before this.
Verified with helm template.
Was hardcoded to webapp/demo-go-app — any future app added to this
project needed its own policy line. Wildcarding to webapp/* means any
app onboarded into that project (project: webapp in its Application
manifest) is automatically covered, no RBAC change needed per app.
Platform-level apps on the default project are unaffected.
Webhook delivery to jenkins.192.168.1.7.nip.io failed: "webhook can
only call allowed HTTP servers (check your security.ALLOWED_HOST_LIST
setting)". Gitea's SSRF protection blocks outbound webhook calls to
private/internal IPs by default. Set to "*" rather than a narrow
allowlist — every host on this homelab is on the same private LAN, so
a per-host allowlist would just need updating every time a new
service needs webhook access, matching the lightweight security
posture already used elsewhere (ArgoCD --insecure, plain HTTP
throughout).
kubernetes was manually updated to 4437.v3a_18554d3f32 via the Jenkins
UI (the pin still said 4353.vb_47977da_9417) — this combination,
paired with kubernetes-client-api:7.3.1-256.v788a_0b_787114, is what
got a real build through checkout successfully. Pinning it here so a
future restart/reprovision reinstalls the version that's actually
been proven to work, not the untested original pin.
The kubernetes plugin fix worked — agent connected, checkout
succeeded, pipeline ran through 3 stages. Next failure:
loadConfig.groovy calls readYaml (to parse config.yaml), but that
step isn't in the chart's default plugin list at all, so it was never
installed: "No such DSL method 'readYaml' found among steps [...]".
Adds pipeline-utility-steps, which provides readYaml/writeYaml/
readJSON etc.
kubernetes:4353.vb_47977da_9417 requires kubernetes-client-api >=
7.3.1-256.v788a_0b_787114 (confirmed via plugins.jenkins.io), but that
plugin was never explicitly pinned in installPlugins — left to
resolve on its own at image-build time, it landed on an older,
incompatible version. Every build agent launch then crashed with
NoSuchMethodError: ConfigBuilder.withMasterUrl(String) inside
Reaper.preLaunch -> KubernetesCloud.connect ->
KubernetesFactoryAdapter.createClient: pods provisioned fine at the
Kubernetes API level, but the controller could never actually connect
an agent to them, so every demo-go-app build hung forever at "Still
waiting to schedule task".
demo-go-app moved from ArgoCD's default project to the new webapp
project (devops-argo-config#7de2f8b). ArgoCD RBAC is keyed
<project>/<app-name>, so the jenkins-ci sync/get policy has to move
with it or syncArgoApp's scoped token starts getting silent 403s on
its next sync.
Adds accounts.jenkins-ci (apiKey) and a repo-scoped RBAC policy
(sync/get on default/demo-go-app only) so devops-lib's syncArgoApp
stage can trigger syncs without full admin access.
