toolshed deploys an app that asked for a persistent volume as a StatefulSet
with a volumeClaimTemplate rather than a Deployment
(internal/deploy.Client.ensureStatefulSet). Without this the deployer gets
"forbidden" the moment anyone creates one — the two-repositories drift
internal/deploy/kubernetes.go's own package doc warns about, and the same
way the custom-domains Ingress rights were missed until after that feature
shipped.
Delete is included deliberately, not for tidiness: a Deployment and a
StatefulSet share the app's selector, so switching an app between stateless
and stateful must remove whichever controller it no longer is, or both stay
alive fighting over the same pods.
persistentvolumeclaims is read-only. The claims are created by the
StatefulSet's own volumeClaimTemplates, never directly by toolshed, and
deleting one would destroy an app's data — so there is no reason for this
credential to be able to.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Registers the new hand-written redis chart (devops-infra-helm-charts,
separate commit) and the ExternalSecret feeding its admin password from
Vault. Own namespace, addressed over cluster DNS like every other platform
component here:
redis.redis.svc.cluster.local:6379
Only one consumer for the credential, unlike the Postgres one next door:
the server itself, to seed its ACL file on first boot. toolshed's api gets
it from the connection an operator configures in the dashboard, encrypted
in toolshed's own database — so there is deliberately no second
ExternalSecret into the toolshed namespace.
Order matters: put the password in Vault at secret/toolshed/redis before
syncing, or the init container sits in CreateContainerConfigError. The
exact command, the reason the password must be alphanumeric (it is written
into an ACL directive where a space or quote would split it), and the
manual rotation procedure are all recorded in the ExternalSecret's own
header.
Nothing here needs to change for Postgres: toolshed's managed database
add-on points at the existing postgresql.postgres.svc.cluster.local, whose
POSTGRES_USER is the initdb superuser and so already has the CREATEDB and
CREATEROLE that provisioning needs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Overdue: this belongs with toolshed's custom-domains feature, which has
already merged and shipped. Without it, deployer gets "forbidden" the
moment anyone actually adds a custom domain — the exact failure mode
internal/deploy/kubernetes.go's own package doc warns about, since its
resource list and this file are unsynchronised copies in two repositories.
Scoped as a namespaced Role/RoleBinding rather than widening the
ClusterRole above it. A custom domain's Ingress and TLS secret live in the
gateway's own namespace and never in an app's, so this only ever targets
one fixed namespace — unlike the per-app namespaces the ClusterRole
necessarily spans. Secrets are get/list/watch/delete only: cert-manager
writes them, deployer just reads readiness and cleans up on removal.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Companion change to toolshed's new multi-replica/autoscaling feature
(internal/deploy.Client.ensureAutoscaler in the toolshed repo) — without
this, deployer's own attempt to create a HorizontalPodAutoscaler for any
app with autoscaling enabled fails with "forbidden" the first time
someone actually uses the feature, exactly the failure mode
internal/deploy/kubernetes.go's own package doc comment warns about
for these two unsynchronized copies of deployer's permission list.
Kept in sync with toolshed's own deploy/helm/toolshed/templates/rbac.yaml,
which received the identical addition.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Deliberately separate from harbor-robot-dockerconfig, which is mounted
into build pods running arbitrary user Dockerfiles and is scoped to
push+pull only. This one is held by builder itself — a trusted platform
service that never executes user code directly — and carries a
permission the other should never have: delete. A leaked build-pod
credential can never delete anything this way, and a leaked cleanup
credential can never push.
Used by toolshed's cleanupImage (internal/builder/builder.go) to remove
an app's Harbor repository when the app itself is deleted. Nothing
destructive happens until the Vault path this pulls from is actually
populated — builder logs and skips that step otherwise, per its own
code comment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Matches the chart-side addition in devops-infra-helm-charts. The
ExternalSecret needs its Vault path populated before either this or
the grafana Application syncs — see that file's own header for the
exact vault kv put command, same requirement postgres and every other
admin credential in this repo already has.
app-of-secretstores.yaml is automated (prune: true), so this new
secretstore file needs no separate manual sync of its own — only that
app-of-apps wrapper, same as any other new file under secretstores/.
Verified with `helm template` against generic-argo-apps-chart and this
values file: 12 Applications render, grafana present and correctly
formed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Matches the chart-side replacement in devops-infra-helm-charts. Three
Applications now where there was one, since victoria-metrics-single
bundles no scraper or exporter the way the Prometheus chart did.
Sync order matters here specifically: victoria-metrics-single needs to
exist before vmagent's remote_write target resolves, though ArgoCD sync
failures are naturally retried so this only affects how quickly it
converges, not whether it does.
Verified with `helm template` against generic-argo-apps-chart and this
values file: 11 Applications render, the three new ones present and
correctly formed, nothing left pointing at the removed prometheus
chartDir.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Points at the chart and values just added to devops-infra-helm-charts.
nameOverride pinned to "prometheus" so the Helm release name matches
what the chart's server Service naming depends on
("<release-name>-server") — see that repo's commit for the full
reasoning, since this file is the one place that decision has to be
made correctly for toolshed's already-seeded PROMETHEUS_URL to resolve.
Needs the usual two syncs to actually deploy: this values file only
takes effect once incubator-infra-k8s-admin-prd-ase1 is synced (which
creates/updates the child "prometheus" Application object), and that
child Application then needs its own sync to actually create anything
in the cluster.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Environment variables are delivered through a Secret so values never appear
in the pod spec, but the ClusterRole was never given the resource — so every
deploy with configuration failed on a forbidden error, minutes after the
change looked fine.
Granted without list or watch. Kubernetes RBAC cannot scope a ClusterRole to
a namespace pattern, so this necessarily covers every namespace; withholding
list at least stops deployer enumerating the cluster's secrets, leaving only
access by a name it already knows. That narrows the blast radius rather than
removing it, and is called out in the manifest.
The proper fix, once there are tenants who are not the operator, is a
RoleBinding created per app namespace. That requires deployer to be able to
create RoleBindings, which is its own escalation path and wants deciding
deliberately rather than being slipped in here.
Scoped to the resource kinds toolshed creates for an app — not
cluster-admin, not a wildcard. This is the one credential in the system
whose compromise means the cluster, so what it can do should fit on one
screen.
Here rather than in toolshed's chart because these are cluster-scoped and
that Application runs in the webapp project, which permits only Namespace.
Widening webapp would have given every demo app in it the ability to create
cluster-wide RBAC.
A dedicated Jenkins user with an API token, not the admin account. Needs
Job/Create as well as Build and Read, because builder creates an app's job
on first deploy — which is what guarantees the SCM polling trigger exists
and so that a webhook can ever find it (issue #30).
Only api gets the private half. The gateway is given the public half in
plain Helm values, and that asymmetry is the point: the gateway
terminates traffic for every deployed app, so holding only a verification
key means compromising it cannot forge a session for anyone.
The two halves must be installed together — a private key here that does
not match the public key in the gateway's values means every session api
issues is rejected and every app redirects to login forever.
One Vault path (secret/toolshed/postgres) into two namespaces: the server
needs the credential to initialise, api needs it to connect. Kept in one
file because it is one credential with two consumers — split, they drift,
and drift surfaces as an authentication error that looks nothing like a
config mistake.
The api-side Secret also carries a composed libpq URL, so host, port and
database name live in one place instead of across several environment
variables that can disagree.
Postgres gets its own namespace rather than living inside toolshed, so it
is addressed over cluster DNS like any other platform component and
outlives whatever consumes it.
No longer needed now that anonymous pull works — pairs with
devops-helm-charts#efc2c4f dropping the corresponding pullSecret
reference from demo-go-app's values. app-of-secretstores has automated
prune, so this deletion applies on its own on next sync.
Both needed for the registry hostname switch to
harbor.192.168.1.7.nip.io (devops-lib#3ffa2d8):
- harbor-robot-dockerconfig's auths key was hardcoded to
harbor-core.harbor.svc.cluster.local — docker matches stored
credentials by exact registry hostname, so this had to move with
the image reference or push would go back to unauthorized.
- New demo-go-app-image-pull-secret, in the demo-go-app namespace
(imagePullSecrets must be same-namespace as the pod), for the
Deployment's own pull — separate from harbor-robot-dockerconfig,
which only ever lived in the jenkins namespace for the build pod's
push. Same underlying robot account/Vault path.
extra-manifests/priorityclass-high-priority.yaml: cluster-side
PriorityClass backing the app chart's hardcoded priorityClassName.
secretstores/argocd-jenkins-ci-token.yaml: ExternalSecret for the
scoped ArgoCD account devops-lib's syncArgoApp stage authenticates
with.
secretstores/harbor-robot-dockerconfig.yaml: ExternalSecret building a
dockerconfigjson from the Harbor robot account for image pulls.
