Add a Redis chart for toolshed's managed cache add-on
toolshed provisions a per-app Redis ACL user, scoped to its own key prefix, on request (internal/dbprovision.EnsureRedisUser). Nothing in this cluster ran Redis — Harbor's internal one is Harbor's and is not ACL configured — so there was nowhere for that to point. Hand-written rather than vendoring Bitnami's, same reasoning as the sibling postgresql chart: Broadcom has been retiring and freezing images behind that repo (infra issue #4, where it broke Contour twice), and Redis publishes no official chart either. The authentication design is the part worth reading before changing anything. Redis is started with an ACL file and NO requirepass, and that distinction is a security property rather than a style choice: - toolshed persists provisioned users with ACL SAVE, which requires an aclfile. Without it every provisioned user is lost on the next restart. - But ACL SAVE also serialises the default user. With requirepass, the saved entry comes back as `user default on nopass ~* &* +@all`, and after the next restart the ACL file wins — leaving Redis open to UNAUTHENTICATED access with full permissions. Verified directly: with requirepass, the restarted server answered an unauthenticated PING with PONG and served a key. So the default user is defined in the ACL file instead, seeded once by an init container that deliberately never overwrites an existing file — overwriting would delete every user toolshed had provisioned into it, reintroducing the same lockout from the other end. The documented consequence is that rotating the admin password in Vault does not propagate on its own; that needs ACL SETUSER default + ACL SAVE against the running server. Sized for a node at its ceiling: 32Mi requested, 96Mi limit, maxmemory 48mb. The limit sits above maxmemory on purpose, so Redis reaches its own eviction policy rather than being OOM-killed, which would lose the whole instance instead of the coldest keys. Snapshotting is off — what must survive a restart is the ACL file, which ACL SAVE writes independently of RDB, and cached values are by definition reconstructible. allkeys-lru because this backs a connection kind called "cache" and eviction under pressure is that contract; values.yaml says plainly that an app using Redis as its only copy of something wants noeviction instead. Verified: helm template, then a real deploy to a k3d cluster — provisioned users through toolshed's own code, deleted the pod, and confirmed all five came back with their key patterns intact, the init container declined to overwrite, unauthenticated access got NOAUTH, and a user writing outside its prefix got NOPERM. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
This commit is contained in:
Mukul Sharma
co-authored by
Claude Opus 5
Claude Opus 5
parent
0c68312765
commit
76b4ddd2de
@@ -0,0 +1,50 @@
|
||||
# Redis backing toolshed's managed cache add-on.
|
||||
#
|
||||
# Deployed as shared infrastructure in its own namespace rather than inside
|
||||
# the toolshed namespace, so it is addressed over cluster DNS like any other
|
||||
# platform component and its lifecycle is independent of the application
|
||||
# that happens to be its first consumer:
|
||||
#
|
||||
# redis.redis.svc.cluster.local:6379
|
||||
#
|
||||
# The admin password comes from Vault through External Secrets — see
|
||||
# devops-infra-argo-config/secretstores/toolshed-redis-credentials.yaml. The
|
||||
# Secret must exist before this pod can start; a missing Secret leaves the
|
||||
# init container in CreateContainerConfigError rather than failing in a way
|
||||
# that explains itself.
|
||||
#
|
||||
# Read values.yaml's `config` block before changing anything about
|
||||
# authentication here. The absence of `requirepass` is deliberate and
|
||||
# security-relevant, not an oversight.
|
||||
|
||||
fullnameOverride: redis
|
||||
|
||||
image:
|
||||
repository: redis
|
||||
tag: "7-alpine"
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
existingSecret: redis-credentials
|
||||
|
||||
persistence:
|
||||
enabled: true
|
||||
storageClass: local-path
|
||||
# Holds the ACL file and nothing else worth keeping — snapshotting is off
|
||||
# (see config.save). 1Gi is already far more than needed; local-path
|
||||
# cannot resize in place, so it is sized up front rather than tightly.
|
||||
size: 1Gi
|
||||
|
||||
config:
|
||||
# The node has 8GB and was at its ceiling before Postgres was added; the
|
||||
# demo apps were scaled to zero to make room for that. 48mb is a real
|
||||
# cache for a handful of small internal tools and costs little.
|
||||
maxmemory: 48mb
|
||||
maxmemoryPolicy: allkeys-lru
|
||||
save: ""
|
||||
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
memory: 96Mi
|
||||
@@ -0,0 +1,23 @@
|
||||
apiVersion: v2
|
||||
name: redis
|
||||
description: |
|
||||
Single-instance Redis for this homelab, backing toolshed's managed cache
|
||||
add-on (internal/dbprovision) — toolshed provisions a per-app ACL user
|
||||
scoped to its own key prefix on request.
|
||||
|
||||
Hand-written rather than vendoring Bitnami's chart, for the same reason
|
||||
the sibling postgresql chart is: Broadcom has been retiring and freezing
|
||||
images behind that repo (claude.md infra issue #4, where it broke Contour
|
||||
twice), and Redis publishes no official Helm chart of its own.
|
||||
|
||||
Authentication is defined entirely by the ACL file, with no requirepass.
|
||||
That is not a style choice — see values.yaml, where the reasoning is
|
||||
recorded alongside the setting it explains. Getting it wrong leaves the
|
||||
server open to unauthenticated access after its first restart.
|
||||
|
||||
Not highly available and not intended to be. One replica, one PVC, no
|
||||
replication, no sentinel. On a single-node cluster those would be
|
||||
theatre.
|
||||
type: application
|
||||
version: 0.1.0
|
||||
appVersion: "7"
|
||||
@@ -0,0 +1,23 @@
|
||||
{{- $name := .Values.fullnameOverride | default "redis" -}}
|
||||
# ClusterIP only. Nothing outside the cluster should reach Redis, and there
|
||||
# is no Ingress here on purpose — Contour terminates HTTP, and exposing
|
||||
# Redis's wire protocol through it is neither possible nor wanted.
|
||||
#
|
||||
# Consumers address this as:
|
||||
# {{ $name }}.{{ .Release.Namespace }}.svc.cluster.local:{{ .Values.service.port }}
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ $name }}
|
||||
namespace: {{ .Release.Namespace }}
|
||||
labels:
|
||||
app: {{ $name }}
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app: {{ $name }}
|
||||
ports:
|
||||
- name: redis
|
||||
port: {{ .Values.service.port }}
|
||||
targetPort: redis
|
||||
protocol: TCP
|
||||
@@ -0,0 +1,127 @@
|
||||
{{- $name := .Values.fullnameOverride | default "redis" -}}
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: {{ $name }}
|
||||
namespace: {{ .Release.Namespace }}
|
||||
labels:
|
||||
app: {{ $name }}
|
||||
spec:
|
||||
serviceName: {{ $name }}
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: {{ $name }}
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: {{ $name }}
|
||||
spec:
|
||||
securityContext:
|
||||
# The official image runs as the redis user (uid 999 on the Alpine
|
||||
# variant). fsGroup makes the provisioned volume group-writable so
|
||||
# Redis can write the ACL file it is given — without it ACL SAVE
|
||||
# fails at provisioning time with a permission error.
|
||||
fsGroup: 999
|
||||
terminationGracePeriodSeconds: 30
|
||||
initContainers:
|
||||
# Seeds the ACL file with the default (admin) user on first boot
|
||||
# only. Redis will not start with an --aclfile that does not exist,
|
||||
# and the default user has to be defined there rather than by
|
||||
# requirepass — see the long note in values.yaml for why that
|
||||
# distinction is a security property and not a preference.
|
||||
#
|
||||
# Never overwrites an existing file. That file is rewritten by ACL
|
||||
# SAVE every time toolshed provisions an app user, so recreating it
|
||||
# on every pod start would silently delete every provisioned user
|
||||
# and lock those apps out — the exact failure this whole design
|
||||
# exists to prevent, reintroduced from the other end.
|
||||
#
|
||||
# Consequence worth knowing: rotating the admin password in Vault
|
||||
# does NOT propagate here, because this only ever runs against a
|
||||
# missing file. Rotating means `ACL SETUSER default >newpassword`
|
||||
# followed by `ACL SAVE` against the running server.
|
||||
- name: seed-acl
|
||||
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
|
||||
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
||||
env:
|
||||
- name: REDIS_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: {{ .Values.existingSecret }}
|
||||
key: {{ .Values.secretKeys.password }}
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
set -e
|
||||
if [ -f /data/users.acl ]; then
|
||||
echo "ACL file already present; leaving it alone."
|
||||
exit 0
|
||||
fi
|
||||
echo "user default on >$REDIS_PASSWORD ~* &* +@all" > /data/users.acl
|
||||
chmod 600 /data/users.acl
|
||||
echo "Seeded ACL file with the default user."
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
containers:
|
||||
- name: redis
|
||||
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
|
||||
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
||||
args:
|
||||
- redis-server
|
||||
- --aclfile
|
||||
- /data/users.acl
|
||||
- --maxmemory
|
||||
- {{ .Values.config.maxmemory | quote }}
|
||||
- --maxmemory-policy
|
||||
- {{ .Values.config.maxmemoryPolicy | quote }}
|
||||
- --save
|
||||
- {{ .Values.config.save | quote }}
|
||||
ports:
|
||||
- name: redis
|
||||
containerPort: 6379
|
||||
protocol: TCP
|
||||
# Authenticated probes: with the ACL file in place an
|
||||
# unauthenticated PING is correctly refused with NOAUTH, so a
|
||||
# bare `redis-cli ping` would mark a perfectly healthy server as
|
||||
# failing. Run through a shell so the environment expands —
|
||||
# Kubernetes does not substitute $(VAR) inside exec probe
|
||||
# commands.
|
||||
env:
|
||||
- name: REDIS_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: {{ .Values.existingSecret }}
|
||||
key: {{ .Values.secretKeys.password }}
|
||||
readinessProbe:
|
||||
exec:
|
||||
command: ["sh", "-c", 'redis-cli --no-auth-warning -a "$REDIS_PASSWORD" ping | grep -q PONG']
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 6
|
||||
livenessProbe:
|
||||
exec:
|
||||
command: ["sh", "-c", 'redis-cli --no-auth-warning -a "$REDIS_PASSWORD" ping | grep -q PONG']
|
||||
initialDelaySeconds: 20
|
||||
periodSeconds: 20
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 6
|
||||
resources:
|
||||
{{- toYaml .Values.resources | nindent 12 }}
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
{{- if .Values.persistence.enabled }}
|
||||
volumeClaimTemplates:
|
||||
- metadata:
|
||||
name: data
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
storageClassName: {{ .Values.persistence.storageClass | quote }}
|
||||
resources:
|
||||
requests:
|
||||
storage: {{ .Values.persistence.size | quote }}
|
||||
{{- end }}
|
||||
@@ -0,0 +1,84 @@
|
||||
# Chart defaults. Real configuration lives in
|
||||
# helm-overrides/k8s-admin-prd-ase1/redis/custom-values.yaml.
|
||||
|
||||
fullnameOverride: redis
|
||||
|
||||
image:
|
||||
# Pulled from Docker Hub, like every other infra component here (gitea,
|
||||
# vault, harbor, postgresql). The base-images mirror in Harbor exists to
|
||||
# remove Docker Hub from the *application build* path — it is not in play
|
||||
# for platform components.
|
||||
repository: redis
|
||||
tag: "7-alpine"
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
# Name of the Secret holding the admin password. Created by External
|
||||
# Secrets from Vault, not by this chart — a chart that generates its own
|
||||
# password regenerates it on every render, which would rewrite the ACL file
|
||||
# and lock every already-provisioned app out of its own data.
|
||||
existingSecret: redis-credentials
|
||||
secretKeys:
|
||||
password: password
|
||||
|
||||
service:
|
||||
port: 6379
|
||||
|
||||
persistence:
|
||||
enabled: true
|
||||
# local-path-provisioner, this cluster's default StorageClass. Small: this
|
||||
# holds the ACL file and (if enabled) an RDB snapshot, not a dataset of
|
||||
# any size — maxmemory below is the real ceiling on what Redis will hold.
|
||||
# The volume is not resizable in place with this provisioner, so it is
|
||||
# sized up front.
|
||||
storageClass: local-path
|
||||
size: 1Gi
|
||||
|
||||
config:
|
||||
# ACL FILE, NOT requirepass. This distinction is load-bearing and easy to
|
||||
# "simplify" into a security hole, so it is written down here rather than
|
||||
# left to be rediscovered:
|
||||
#
|
||||
# toolshed provisions per-app users with ACL SETUSER, and persists them
|
||||
# with ACL SAVE (internal/dbprovision.EnsureRedisUser) — without that
|
||||
# save, every provisioned user is lost on the next restart and every app
|
||||
# using Redis fails to authenticate with credentials that still look
|
||||
# valid. ACL SAVE requires an aclfile; that is why one is configured.
|
||||
#
|
||||
# But ACL SAVE also writes the *default* user's state to that file. With
|
||||
# `requirepass` set and the default user defined only by it, the saved
|
||||
# entry comes back as `user default on nopass ~* &* +@all` — and after
|
||||
# the next restart the ACL file wins, leaving Redis accepting
|
||||
# UNAUTHENTICATED connections with full access. Verified directly, not
|
||||
# inferred: with requirepass the restarted server answered an
|
||||
# unauthenticated PING with PONG and served a key.
|
||||
#
|
||||
# Defining the default user in the ACL file instead (seeded by the init
|
||||
# container, see the StatefulSet) keeps its password across every
|
||||
# subsequent ACL SAVE — the same restart then correctly answers
|
||||
# `NOAUTH Authentication required.`
|
||||
#
|
||||
# If you ever add `requirepass` here, you reintroduce that hole.
|
||||
maxmemory: 48mb
|
||||
# allkeys-lru, because this backs a connection kind literally called
|
||||
# "cache" and eviction under pressure is that contract. An app using
|
||||
# Redis as its only copy of something wants noeviction instead — at
|
||||
# which case writes start failing when full rather than data silently
|
||||
# disappearing. Neither is safe for every use; this one matches the name.
|
||||
maxmemoryPolicy: allkeys-lru
|
||||
# Snapshotting off. What must survive a restart is the ACL file, which is
|
||||
# written by ACL SAVE independently of RDB/AOF. Cached values are by
|
||||
# definition reconstructible, and on a node at its memory ceiling a
|
||||
# background save's copy-on-write spike is a real risk for no benefit.
|
||||
save: ""
|
||||
|
||||
# Tuned for a node with 8GB total that is already near its ceiling. The
|
||||
# request is what the scheduler reserves; the limit is sized above
|
||||
# maxmemory so Redis hits its own eviction policy rather than being
|
||||
# OOM-killed by the kernel, which loses the whole instance instead of the
|
||||
# coldest keys.
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
memory: 96Mi
|
||||
Reference in New Issue
Block a user