Removes the cause of the diff failures that ServerSideDiff worked around:
v2.13.8 diffs against a Kubernetes schema compiled into its own binary,
and this cluster is newer than that schema.
Three things needed real changes, none of them mechanical:
- applicationSet.enabled no longer exists, and there is no replacement
gate — unlike dex and notifications, the ApplicationSet controller's
Deployment carries no conditional at all. Helm ignores unknown keys, so
carrying the old value forward would have quietly started a controller
nothing here uses. replicas: 0 is the only lever.
- The image tag pin is dropped rather than moved to v3.5.2. The chart's
appVersion governs, so image and chart cannot drift; a pin outliving its
chart is close to the failure being fixed here.
- server.insecure moved from server.extraArgs to configs.params, which is
what the chart renders into argocd-cmd-params-cm. Caught by rendering:
an earlier version of this commit deleted the extra arg on the strength
of a comment claiming configs.params already set it, which it did not —
Argo CD would have served its own TLS behind Contour and produced a
redirect loop.
Verified in the rendered output: image v3.5.2 and no v2.13.8 anywhere,
server.insecure true, dex and notifications absent, ApplicationSet at zero
replicas, both HPAs intact with replicas omitted, ingress on Contour, repo
Secrets on cluster DNS.
Behaviour changes in v3 that apply here, none needing a values change:
logs RBAC is now enforced (jenkins-ci only syncs), update/delete no longer
inherit to sub-resources, and resource tracking moves from labels to
annotations, so the first sync re-stamps every managed resource.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
toolshed provisions a per-app Redis ACL user, scoped to its own key
prefix, on request (internal/dbprovision.EnsureRedisUser). Nothing in this
cluster ran Redis — Harbor's internal one is Harbor's and is not ACL
configured — so there was nowhere for that to point.
Hand-written rather than vendoring Bitnami's, same reasoning as the
sibling postgresql chart: Broadcom has been retiring and freezing images
behind that repo (infra issue #4, where it broke Contour twice), and Redis
publishes no official chart either.
The authentication design is the part worth reading before changing
anything. Redis is started with an ACL file and NO requirepass, and that
distinction is a security property rather than a style choice:
- toolshed persists provisioned users with ACL SAVE, which requires an
aclfile. Without it every provisioned user is lost on the next restart.
- But ACL SAVE also serialises the default user. With requirepass, the
saved entry comes back as `user default on nopass ~* &* +@all`, and
after the next restart the ACL file wins — leaving Redis open to
UNAUTHENTICATED access with full permissions. Verified directly: with
requirepass, the restarted server answered an unauthenticated PING with
PONG and served a key.
So the default user is defined in the ACL file instead, seeded once by an
init container that deliberately never overwrites an existing file —
overwriting would delete every user toolshed had provisioned into it,
reintroducing the same lockout from the other end. The documented
consequence is that rotating the admin password in Vault does not
propagate on its own; that needs ACL SETUSER default + ACL SAVE against
the running server.
Sized for a node at its ceiling: 32Mi requested, 96Mi limit, maxmemory
48mb. The limit sits above maxmemory on purpose, so Redis reaches its own
eviction policy rather than being OOM-killed, which would lose the whole
instance instead of the coldest keys. Snapshotting is off — what must
survive a restart is the ACL file, which ACL SAVE writes independently of
RDB, and cached values are by definition reconstructible.
allkeys-lru because this backs a connection kind called "cache" and
eviction under pressure is that contract; values.yaml says plainly that an
app using Redis as its only copy of something wants noeviction instead.
Verified: helm template, then a real deploy to a k3d cluster — provisioned
users through toolshed's own code, deleted the pod, and confirmed all five
came back with their key patterns intact, the init container declined to
overwrite, unauthenticated access got NOAUTH, and a user writing outside
its prefix got NOPERM.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Grafana: real dashboards over VictoriaMetrics. Provisioned rather than
clicked through: a VictoriaMetrics datasource (type: prometheus — VM
speaks that query API, which is the whole point of it existing) is
baked into the chart's own datasource-provisioning config, so a fresh
install has it working with no manual setup. Admin credentials from
Vault via ExternalSecret (secretstores/grafana-admin-credentials.yaml
in devops-infra-argo-config), same pattern as every other admin
credential in this project — never plaintext in this repo. 1Gi
local-path PVC for dashboards/Grafana's own state (VictoriaMetrics
holds the actual metric data, not this). Dual LAN+Tailscale Ingress
hosts, same convention as everything externally reachable here.
Found and fixed while vendoring: helm-templates/grafana already held a
fully-vendored old Grafana chart (v6.58.7, appVersion 10.0.3) from the
original Meesho monorepo import (commit b8575bb) — generic production
config (fullnameOverride: grafana-infra-prd, GKE-shaped RBAC/PSP
defaults) unrelated to this homelab, same class of leftover as
victoria-metrics-single's collision two commits ago. Removed and
re-vendored fresh (10.5.15) as a thin wrapper, matching every other
official-chart component in this repo now.
vmui: VictoriaMetrics' own built-in UI (ad-hoc PromQL + graphs, no
saved dashboards — what Grafana is for) is served on the same
pod/port, so exposing it cost one ingress block on the
victoria-metrics-single values already committed. No new component,
no new RAM.
Verified with `helm template` against the real charts for both
components individually (Grafana: admin env vars correctly reference
the ExternalSecret's keys, datasource ConfigMap renders the intended
VictoriaMetrics URL, PVC/resources/ingress hosts all match; vmui:
ingress renders both hostnames pointing at the existing Service's named
http port) and again for the whole generic-argo-apps-chart appSpec
list — 12 Applications render, including grafana.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Lower RAM/disk for the same metric volume on an 8GB single node — VM's
own compression is the entire reason it exists as a project. Speaks
Prometheus's own query API (/api/v1/query), so nothing downstream
needed to change beyond the URL it points at: same PromQL, same
kubernetes-nodes-cadvisor-sourced container_cpu_usage_seconds_total /
container_memory_working_set_bytes toolshed's metrics work
(docs/PRODUCT-ARCHITECTURE.md step 5) already targets.
Three releases, matching the one-release-per-component convention
already used everywhere in this repo, since victoria-metrics-single has
no bundled scraper/exporter subcharts the way the Prometheus chart did:
- victoria-metrics-single: the TSDB + query engine. 3Gi/local-path,
7-day retention, resources capped at 512Mi — same trim reasoning as
the Prometheus server it replaces.
- vmagent: the scraper. Its default scrape_configs already includes
kubernetes-nodes-cadvisor and kubernetes-service-endpoints (the
chart's own comment says "COPY from Prometheus helm chart") — nothing
to override there, only remoteWrite pointed at victoria-metrics-single
and trimmed resources.
- node-exporter: pulled out of the removed Prometheus chart's bundled
subchart into its own standalone release, since victoria-metrics-single
has no equivalent. Same trim as before (32Mi/64Mi), same
prometheus.io/scrape annotation vmagent's default scrape config
already looks for.
Found and fixed while vendoring: helm-templates/victoria-metrics-single
already existed in this repo — a leftover GKE-targeted vendored copy
from the original Meesho monorepo import (commit b8575bb), with its own
templates/values.yaml for a different chart entirely. `mkdir -p` on an
already-existing directory silently did nothing, so the first vendoring
attempt left that old tree sitting alongside the new one and rendered
two ServiceAccounts/Services/StatefulSets with colliding names. Removed
outright rather than adapted, same reasoning as the ~35 charts
devops-helm-charts removed in its own prune — see this chart's own
Chart.yaml for the full note.
Verified with `helm template` against the real charts and these values
for all three releases individually, and again for the whole
generic-argo-apps-chart appSpec list (11 Applications render, including
the three new ones and nothing orphaned from the old prometheus entry).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Vendored prometheus-community/prometheus 29.27.1 the same way
Contour/ArgoCD/Vault/Gitea/Harbor/Jenkins already are here — a thin
Chart.yaml dependency plus a committed .tgz — rather than hand-written
like postgresql, which has no official chart to vendor.
Server only: alertmanager, kube-state-metrics, prometheus-node-exporter
and prometheus-pushgateway are all enabled by default in this chart and
all disabled here. None are needed for what actually consumes this —
toolshed's per-app CPU/memory metrics read straight from the chart's
built-in kubernetes-nodes-cadvisor scrape job (kubelet's own cAdvisor
endpoint) — and each is its own pod on a node that was already at its
8GB ceiling before this.
Trimmed for the same ceiling: 3Gi PVC on local-path (not the chart's
8Gi default), 7-day retention (not 15), resources capped at 512Mi.
nameOverride pinned to exactly "prometheus" matters more here than for
any other component pinning it: the chart's server Service renders as
"<release-name>-server", so this is what makes it "prometheus-server"
— the exact hostname toolshed's PROMETHEUS_URL was already seeded to
point at, before this existed, so the connection is already correct
the day this deploys.
Verified with `helm template` against the real chart and these values:
server-only object set (ClusterRole, ClusterRoleBinding, ConfigMap,
Deployment, PVC, Service, ServiceAccount — nothing from the four
disabled subcharts), and the rendered PVC/retention/resources/Service
name all match what's written above.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Not Bitnami's: that registry has been actively unstable here (it broke
Contour twice, infra issue #4) and PostgreSQL publishes no official chart.
A single StatefulSet, PVC and Service is small enough that owning it costs
less than depending on an unstable repackage.
Credentials come from an existing Secret rather than being generated by
the chart — a chart that generates its own password regenerates it on
every render and silently locks you out of the existing volume.
Details that matter and are easy to get wrong:
- PGDATA is a subdirectory of the mount, not the mount itself. initdb
refuses to run in a directory that already has contents.
- Probes run through a shell. Kubernetes does not expand $(VAR) inside
exec probe commands, only in command/args.
- fsGroup 70 so the volume stays writable after the entrypoint drops
from root to the postgres user on the Alpine variant.
- shared_buffers cut to 32MB from PostgreSQL's 128MB default. The node
has 8GB and was at its ceiling before this.
Verified with helm template.
