Grafana: real dashboards over VictoriaMetrics. Provisioned rather than
clicked through: a VictoriaMetrics datasource (type: prometheus — VM
speaks that query API, which is the whole point of it existing) is
baked into the chart's own datasource-provisioning config, so a fresh
install has it working with no manual setup. Admin credentials from
Vault via ExternalSecret (secretstores/grafana-admin-credentials.yaml
in devops-infra-argo-config), same pattern as every other admin
credential in this project — never plaintext in this repo. 1Gi
local-path PVC for dashboards/Grafana's own state (VictoriaMetrics
holds the actual metric data, not this). Dual LAN+Tailscale Ingress
hosts, same convention as everything externally reachable here.
Found and fixed while vendoring: helm-templates/grafana already held a
fully-vendored old Grafana chart (v6.58.7, appVersion 10.0.3) from the
original Meesho monorepo import (commit b8575bb) — generic production
config (fullnameOverride: grafana-infra-prd, GKE-shaped RBAC/PSP
defaults) unrelated to this homelab, same class of leftover as
victoria-metrics-single's collision two commits ago. Removed and
re-vendored fresh (10.5.15) as a thin wrapper, matching every other
official-chart component in this repo now.
vmui: VictoriaMetrics' own built-in UI (ad-hoc PromQL + graphs, no
saved dashboards — what Grafana is for) is served on the same
pod/port, so exposing it cost one ingress block on the
victoria-metrics-single values already committed. No new component,
no new RAM.
Verified with `helm template` against the real charts for both
components individually (Grafana: admin env vars correctly reference
the ExternalSecret's keys, datasource ConfigMap renders the intended
VictoriaMetrics URL, PVC/resources/ingress hosts all match; vmui:
ingress renders both hostnames pointing at the existing Service's named
http port) and again for the whole generic-argo-apps-chart appSpec
list — 12 Applications render, including grafana.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Lower RAM/disk for the same metric volume on an 8GB single node — VM's
own compression is the entire reason it exists as a project. Speaks
Prometheus's own query API (/api/v1/query), so nothing downstream
needed to change beyond the URL it points at: same PromQL, same
kubernetes-nodes-cadvisor-sourced container_cpu_usage_seconds_total /
container_memory_working_set_bytes toolshed's metrics work
(docs/PRODUCT-ARCHITECTURE.md step 5) already targets.
Three releases, matching the one-release-per-component convention
already used everywhere in this repo, since victoria-metrics-single has
no bundled scraper/exporter subcharts the way the Prometheus chart did:
- victoria-metrics-single: the TSDB + query engine. 3Gi/local-path,
7-day retention, resources capped at 512Mi — same trim reasoning as
the Prometheus server it replaces.
- vmagent: the scraper. Its default scrape_configs already includes
kubernetes-nodes-cadvisor and kubernetes-service-endpoints (the
chart's own comment says "COPY from Prometheus helm chart") — nothing
to override there, only remoteWrite pointed at victoria-metrics-single
and trimmed resources.
- node-exporter: pulled out of the removed Prometheus chart's bundled
subchart into its own standalone release, since victoria-metrics-single
has no equivalent. Same trim as before (32Mi/64Mi), same
prometheus.io/scrape annotation vmagent's default scrape config
already looks for.
Found and fixed while vendoring: helm-templates/victoria-metrics-single
already existed in this repo — a leftover GKE-targeted vendored copy
from the original Meesho monorepo import (commit b8575bb), with its own
templates/values.yaml for a different chart entirely. `mkdir -p` on an
already-existing directory silently did nothing, so the first vendoring
attempt left that old tree sitting alongside the new one and rendered
two ServiceAccounts/Services/StatefulSets with colliding names. Removed
outright rather than adapted, same reasoning as the ~35 charts
devops-helm-charts removed in its own prune — see this chart's own
Chart.yaml for the full note.
Verified with `helm template` against the real charts and these values
for all three releases individually, and again for the whole
generic-argo-apps-chart appSpec list (11 Applications render, including
the three new ones and nothing orphaned from the old prometheus entry).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
The other three disabled subcharts (alertmanager/kube-state-metrics/
pushgateway) stay off — none of them do anything cAdvisor's per-container
metrics don't already cover for this deployment's actual purpose.
node-exporter is different: it's the one thing here that would report on
the node itself (disk, memory, load) rather than any particular
container, which matters on an 8GB single node already near its ceiling.
One DaemonSet pod, hostNetwork rather than a Service, resources capped
at 32Mi/64Mi — light enough that turning it on doesn't meaningfully
touch the budget the other three would have.
Verified with `helm template`: the DaemonSet renders with hostNetwork/
hostPID true and the trimmed resources, and its Service carries the
`prometheus.io/scrape: true` annotation the server's already-enabled
kubernetes-service-endpoints job picks up automatically — no separate
scrape config needed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Vendored prometheus-community/prometheus 29.27.1 the same way
Contour/ArgoCD/Vault/Gitea/Harbor/Jenkins already are here — a thin
Chart.yaml dependency plus a committed .tgz — rather than hand-written
like postgresql, which has no official chart to vendor.
Server only: alertmanager, kube-state-metrics, prometheus-node-exporter
and prometheus-pushgateway are all enabled by default in this chart and
all disabled here. None are needed for what actually consumes this —
toolshed's per-app CPU/memory metrics read straight from the chart's
built-in kubernetes-nodes-cadvisor scrape job (kubelet's own cAdvisor
endpoint) — and each is its own pod on a node that was already at its
8GB ceiling before this.
Trimmed for the same ceiling: 3Gi PVC on local-path (not the chart's
8Gi default), 7-day retention (not 15), resources capped at 512Mi.
nameOverride pinned to exactly "prometheus" matters more here than for
any other component pinning it: the chart's server Service renders as
"<release-name>-server", so this is what makes it "prometheus-server"
— the exact hostname toolshed's PROMETHEUS_URL was already seeded to
point at, before this existed, so the connection is already correct
the day this deploys.
Verified with `helm template` against the real chart and these values:
server-only object set (ClusterRole, ClusterRoleBinding, ConfigMap,
Deployment, PVC, Service, ServiceAccount — nothing from the four
disabled subcharts), and the rendered PVC/retention/resources/Service
name all match what's written above.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Not Bitnami's: that registry has been actively unstable here (it broke
Contour twice, infra issue #4) and PostgreSQL publishes no official chart.
A single StatefulSet, PVC and Service is small enough that owning it costs
less than depending on an unstable repackage.
Credentials come from an existing Secret rather than being generated by
the chart — a chart that generates its own password regenerates it on
every render and silently locks you out of the existing volume.
Details that matter and are easy to get wrong:
- PGDATA is a subdirectory of the mount, not the mount itself. initdb
refuses to run in a directory that already has contents.
- Probes run through a shell. Kubernetes does not expand $(VAR) inside
exec probe commands, only in command/args.
- fsGroup 70 so the volume stays writable after the entrypoint drops
from root to the postgres user on the Alpine variant.
- shared_buffers cut to 32MB from PostgreSQL's 128MB default. The node
has 8GB and was at its ceiling before this.
Verified with helm template.
Was hardcoded to webapp/demo-go-app — any future app added to this
project needed its own policy line. Wildcarding to webapp/* means any
app onboarded into that project (project: webapp in its Application
manifest) is automatically covered, no RBAC change needed per app.
Platform-level apps on the default project are unaffected.
Webhook delivery to jenkins.192.168.1.7.nip.io failed: "webhook can
only call allowed HTTP servers (check your security.ALLOWED_HOST_LIST
setting)". Gitea's SSRF protection blocks outbound webhook calls to
private/internal IPs by default. Set to "*" rather than a narrow
allowlist — every host on this homelab is on the same private LAN, so
a per-host allowlist would just need updating every time a new
service needs webhook access, matching the lightweight security
posture already used elsewhere (ArgoCD --insecure, plain HTTP
throughout).
kubernetes was manually updated to 4437.v3a_18554d3f32 via the Jenkins
UI (the pin still said 4353.vb_47977da_9417) — this combination,
paired with kubernetes-client-api:7.3.1-256.v788a_0b_787114, is what
got a real build through checkout successfully. Pinning it here so a
future restart/reprovision reinstalls the version that's actually
been proven to work, not the untested original pin.
The kubernetes plugin fix worked — agent connected, checkout
succeeded, pipeline ran through 3 stages. Next failure:
loadConfig.groovy calls readYaml (to parse config.yaml), but that
step isn't in the chart's default plugin list at all, so it was never
installed: "No such DSL method 'readYaml' found among steps [...]".
Adds pipeline-utility-steps, which provides readYaml/writeYaml/
readJSON etc.
kubernetes:4353.vb_47977da_9417 requires kubernetes-client-api >=
7.3.1-256.v788a_0b_787114 (confirmed via plugins.jenkins.io), but that
plugin was never explicitly pinned in installPlugins — left to
resolve on its own at image-build time, it landed on an older,
incompatible version. Every build agent launch then crashed with
NoSuchMethodError: ConfigBuilder.withMasterUrl(String) inside
Reaper.preLaunch -> KubernetesCloud.connect ->
KubernetesFactoryAdapter.createClient: pods provisioned fine at the
Kubernetes API level, but the controller could never actually connect
an agent to them, so every demo-go-app build hung forever at "Still
waiting to schedule task".
demo-go-app moved from ArgoCD's default project to the new webapp
project (devops-argo-config#7de2f8b). ArgoCD RBAC is keyed
<project>/<app-name>, so the jenkins-ci sync/get policy has to move
with it or syncArgoApp's scoped token starts getting silent 403s on
its next sync.
Adds accounts.jenkins-ci (apiKey) and a repo-scoped RBAC policy
(sync/get on default/demo-go-app only) so devops-lib's syncArgoApp
stage can trigger syncs without full admin access.
