Vendored prometheus-community/prometheus 29.27.1 the same way
Contour/ArgoCD/Vault/Gitea/Harbor/Jenkins already are here — a thin
Chart.yaml dependency plus a committed .tgz — rather than hand-written
like postgresql, which has no official chart to vendor.
Server only: alertmanager, kube-state-metrics, prometheus-node-exporter
and prometheus-pushgateway are all enabled by default in this chart and
all disabled here. None are needed for what actually consumes this —
toolshed's per-app CPU/memory metrics read straight from the chart's
built-in kubernetes-nodes-cadvisor scrape job (kubelet's own cAdvisor
endpoint) — and each is its own pod on a node that was already at its
8GB ceiling before this.
Trimmed for the same ceiling: 3Gi PVC on local-path (not the chart's
8Gi default), 7-day retention (not 15), resources capped at 512Mi.
nameOverride pinned to exactly "prometheus" matters more here than for
any other component pinning it: the chart's server Service renders as
"<release-name>-server", so this is what makes it "prometheus-server"
— the exact hostname toolshed's PROMETHEUS_URL was already seeded to
point at, before this existed, so the connection is already correct
the day this deploys.
Verified with `helm template` against the real chart and these values:
server-only object set (ClusterRole, ClusterRoleBinding, ConfigMap,
Deployment, PVC, Service, ServiceAccount — nothing from the four
disabled subcharts), and the rendered PVC/retention/resources/Service
name all match what's written above.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Not Bitnami's: that registry has been actively unstable here (it broke
Contour twice, infra issue #4) and PostgreSQL publishes no official chart.
A single StatefulSet, PVC and Service is small enough that owning it costs
less than depending on an unstable repackage.
Credentials come from an existing Secret rather than being generated by
the chart — a chart that generates its own password regenerates it on
every render and silently locks you out of the existing volume.
Details that matter and are easy to get wrong:
- PGDATA is a subdirectory of the mount, not the mount itself. initdb
refuses to run in a directory that already has contents.
- Probes run through a shell. Kubernetes does not expand $(VAR) inside
exec probe commands, only in command/args.
- fsGroup 70 so the volume stays writable after the entrypoint drops
from root to the postgres user on the Alpine variant.
- shared_buffers cut to 32MB from PostgreSQL's 128MB default. The node
has 8GB and was at its ceiling before this.
Verified with helm template.
Was hardcoded to webapp/demo-go-app — any future app added to this
project needed its own policy line. Wildcarding to webapp/* means any
app onboarded into that project (project: webapp in its Application
manifest) is automatically covered, no RBAC change needed per app.
Platform-level apps on the default project are unaffected.
Webhook delivery to jenkins.192.168.1.7.nip.io failed: "webhook can
only call allowed HTTP servers (check your security.ALLOWED_HOST_LIST
setting)". Gitea's SSRF protection blocks outbound webhook calls to
private/internal IPs by default. Set to "*" rather than a narrow
allowlist — every host on this homelab is on the same private LAN, so
a per-host allowlist would just need updating every time a new
service needs webhook access, matching the lightweight security
posture already used elsewhere (ArgoCD --insecure, plain HTTP
throughout).
kubernetes was manually updated to 4437.v3a_18554d3f32 via the Jenkins
UI (the pin still said 4353.vb_47977da_9417) — this combination,
paired with kubernetes-client-api:7.3.1-256.v788a_0b_787114, is what
got a real build through checkout successfully. Pinning it here so a
future restart/reprovision reinstalls the version that's actually
been proven to work, not the untested original pin.
The kubernetes plugin fix worked — agent connected, checkout
succeeded, pipeline ran through 3 stages. Next failure:
loadConfig.groovy calls readYaml (to parse config.yaml), but that
step isn't in the chart's default plugin list at all, so it was never
installed: "No such DSL method 'readYaml' found among steps [...]".
Adds pipeline-utility-steps, which provides readYaml/writeYaml/
readJSON etc.
kubernetes:4353.vb_47977da_9417 requires kubernetes-client-api >=
7.3.1-256.v788a_0b_787114 (confirmed via plugins.jenkins.io), but that
plugin was never explicitly pinned in installPlugins — left to
resolve on its own at image-build time, it landed on an older,
incompatible version. Every build agent launch then crashed with
NoSuchMethodError: ConfigBuilder.withMasterUrl(String) inside
Reaper.preLaunch -> KubernetesCloud.connect ->
KubernetesFactoryAdapter.createClient: pods provisioned fine at the
Kubernetes API level, but the controller could never actually connect
an agent to them, so every demo-go-app build hung forever at "Still
waiting to schedule task".
demo-go-app moved from ArgoCD's default project to the new webapp
project (devops-argo-config#7de2f8b). ArgoCD RBAC is keyed
<project>/<app-name>, so the jenkins-ci sync/get policy has to move
with it or syncArgoApp's scoped token starts getting silent 403s on
its next sync.
Adds accounts.jenkins-ci (apiKey) and a repo-scoped RBAC policy
(sync/get on default/demo-go-app only) so devops-lib's syncArgoApp
stage can trigger syncs without full admin access.
