devops-base-images-gcp/README.md

devops-base-images (GKE)

Owns every image the build pipeline pulls, in Harbor's base-images project:

• Mirrored language base images (images.txt), so builds — devops-lib's buildDocker.groovy fallback templates — never depend on Docker Hub at build time, and what ships is the leanest official variant per language.
• build-tools (build-tools.Dockerfile), the image the build pod runs its own shell steps in.

Both are built or mirrored by hand from here, and devops-lib only ever references the results by tag. build-tools lives here rather than in devops-lib because it is the same kind of artefact as the mirrors, and because a Jenkins job cannot build the image its own build pod needs.

GKE counterpart of the homelab repo of the same name. The manifest and the script are identical; what differs is the registry host and, more substantially, TLS. The homelab's Harbor speaks plain HTTP and its dind pod passes --insecure-registry. Here Harbor has a real certificate issued from a private CA, so instead the CA must be installed into the Docker daemon's trust store.

Not wired into any pipeline. This is deliberately manual and occasional — re-run when images.txt changes (a new language, a version bump, or picking up an upstream base-image update), using a throwaway kubectl DinD pod, since neither the nodes nor your laptop necessarily has Docker.

One-time setup

1. Create the Harbor project. Public: these are re-hosted public images, and public means no pull credentials need wiring into any build.

Harbor UI → New Project → name base-images → check Public → Create.

Or via the API. --cacert is needed because the certificate is signed by the private CA, which your laptop has no reason to trust:

TF=~/Documents/localgit/gcp/toolshed-gke-infra/envs/prod/10-infra
curl --cacert <(terraform -chdir=$TF output -raw registry_ca_cert_pem) \
  -X POST -u admin:'<harbor admin password>' -H "Content-Type: application/json" \
  -d '{"project_name":"base-images","metadata":{"public":"true"}}' \
  "https://harbor.35.238.248.203.nip.io/api/v2.0/projects"

2. Make sure the CA is available in the cluster. The registry-ca ConfigMap in the jenkins namespace holds the CA's public certificate and is managed by devops-infra-argo-config-gcp (extra-manifests/). Check:

kubectl -n jenkins get configmap registry-ca

If app-of-extra-manifests has not synced yet, create it directly — it is a public certificate, not a secret:

kubectl -n jenkins create configmap registry-ca \
  --from-file=ca.crt=<(terraform -chdir=$TF output -raw registry_ca_cert_pem)

3. Run the mirror. Note where the CA is mounted: dockerd looks for /etc/docker/certs.d/<registry host>/ca.crt, and the path must contain the registry hostname exactly. It goes in the dind container, not docker-cli — dockerd performs the push, the CLI only talks to it over TCP.

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: base-images-mirror
  namespace: jenkins
data:
  images.txt: |
    golang:1.22-alpine golang:1.22-alpine
    golang:1.24-alpine golang:1.24-alpine
    alpine:3.20 alpine:3.20
    node:20-alpine node:20-alpine
    python:3.12-alpine python:3.12-alpine
    maven:3-eclipse-temurin-21-alpine maven:3-eclipse-temurin-21-alpine
    eclipse-temurin:21-jre-alpine eclipse-temurin:21-jre-alpine
    php:8.3-cli-alpine php:8.3-cli-alpine
    docker:27-cli docker:27-cli
  mirror.sh: |
    #!/bin/sh
    set -eu
    REGISTRY="harbor.35.238.248.203.nip.io"
    PROJECT="base-images"
    while read -r src target; do
        [ -z "$src" ] && continue
        case "$src" in \#*) continue ;; esac
        dest="${REGISTRY}/${PROJECT}/${target}"
        echo "=== ${src} -> ${dest} ==="
        docker pull "$src"
        docker tag "$src" "$dest"
        docker push "$dest"
    done < images.txt
    echo "=== done ==="
---
apiVersion: v1
kind: Pod
metadata:
  name: base-images-mirror
  namespace: jenkins
spec:
  restartPolicy: Never
  containers:
    - name: docker
      image: docker:27-dind
      securityContext:
        privileged: true
      # No --insecure-registry, unlike the homelab: Harbor here has a real
      # certificate. The CA below is what makes dockerd accept it.
      env:
        - name: DOCKER_TLS_CERTDIR
          value: ""
      volumeMounts:
        - name: docker-graph-storage
          mountPath: /var/lib/docker
        - name: registry-ca
          mountPath: /etc/docker/certs.d/harbor.35.238.248.203.nip.io
          readOnly: true
    - name: docker-cli
      image: docker:27-cli
      command: ["cat"]
      tty: true
      env:
        - name: DOCKER_HOST
          value: tcp://localhost:2375
      volumeMounts:
        - name: mirror
          mountPath: /mirror
  volumes:
    - name: docker-graph-storage
      emptyDir: {}
    - name: mirror
      configMap:
        name: base-images-mirror
        defaultMode: 0755
    - name: registry-ca
      configMap:
        name: registry-ca
EOF

Deliberately no harbor-robot-dockerconfig mount. That robot is scoped to the apps-registry project only, and making base-images public grants anonymous pull, never push — push always needs credentials scoped to the project. For something run this rarely, logging in with the Harbor admin account inside the pod beats provisioning another robot:

kubectl wait --for=condition=Ready pod/base-images-mirror -n jenkins --timeout=180s
kubectl exec -it -n jenkins base-images-mirror -c docker-cli -- sh -c '
  for i in $(seq 1 30); do docker info >/dev/null 2>&1 && break; sleep 2; done
  docker login harbor.35.238.248.203.nip.io
'

(enter the Harbor admin username and password when prompted)

kubectl exec -n jenkins base-images-mirror -c docker-cli -- sh -c '
  cd /mirror && sh mirror.sh
'
kubectl delete pod base-images-mirror -n jenkins
kubectl delete configmap base-images-mirror -n jenkins

Pulls come from Docker Hub, which the nodes reach through Cloud NAT — the nodes have no public IPs of their own.

4. Verify:

curl --cacert <(terraform -chdir=$TF output -raw registry_ca_cert_pem) \
  -s https://harbor.35.238.248.203.nip.io/v2/_catalog

Should list base-images/golang, base-images/alpine, base-images/node, base-images/python, base-images/maven, base-images/eclipse-temurin and base-images/php.

If the push fails with a certificate error, the mount path is the usual cause: it must be /etc/docker/certs.d/harbor.35.238.248.203.nip.io/ca.crt exactly, hostname included, in the dind container.

Building build-tools

Do this after the mirror, since it builds FROM a mirrored image.

The pod is the same shape as the mirror pod — privileged dind, the CA mounted so pushes are trusted, a docker-cli sidecar — with the Dockerfile supplied through a ConfigMap. Create it from the file in this repo so the two cannot drift:

kubectl -n jenkins create configmap build-tools-src \
  --from-file=Dockerfile=build-tools.Dockerfile

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: build-tools-build
  namespace: jenkins
spec:
  restartPolicy: Never
  containers:
    - name: docker
      image: docker:27-dind
      securityContext:
        privileged: true
      env:
        - name: DOCKER_TLS_CERTDIR
          value: ""
      volumeMounts:
        - name: docker-graph-storage
          mountPath: /var/lib/docker
        - name: registry-ca
          mountPath: /etc/docker/certs.d/harbor.35.238.248.203.nip.io
          readOnly: true
    - name: docker-cli
      image: docker:27-cli
      command: ["cat"]
      tty: true
      env:
        - name: DOCKER_HOST
          value: tcp://localhost:2375
        # The classic builder. BuildKit wants to write its own state under
        # /root/.docker, which is read-only wherever the push credentials are
        # mounted as a file — the homelab hit exactly this.
        - name: DOCKER_BUILDKIT
          value: "0"
      volumeMounts:
        - name: src
          mountPath: /src
  volumes:
    - name: docker-graph-storage
      emptyDir: {}
    - name: src
      configMap:
        name: build-tools-src
    - name: registry-ca
      configMap:
        name: registry-ca
EOF

Then log in and build. The tag is deliberate: bump it rather than overwriting, because devops-lib-gcp's dind-pod.yaml pins this tag and an overwritten tag rolls out silently on the next pod start.

kubectl wait --for=condition=Ready pod/build-tools-build -n jenkins --timeout=180s
kubectl exec -it -n jenkins build-tools-build -c docker-cli -- sh -c '
  for i in $(seq 1 30); do docker info >/dev/null 2>&1 && break; sleep 2; done
  docker login harbor.35.238.248.203.nip.io'

kubectl exec -n jenkins build-tools-build -c docker-cli -- sh -c '
  cd /src &&
  docker build -t harbor.35.238.248.203.nip.io/base-images/build-tools:1 . &&
  docker push harbor.35.238.248.203.nip.io/base-images/build-tools:1'

kubectl delete pod build-tools-build -n jenkins
kubectl delete configmap build-tools-src -n jenkins

Verify it has what the stages expect:

kubectl run bt --rm -it --restart=Never \
  --image=harbor.35.238.248.203.nip.io/base-images/build-tools:1 \
  -- sh -c 'git --version && yq --version && python3 -V && bash --version | head -1'

That pull needs no credentials — base-images is public — and the nodes already trust the CA, which is what makes an ordinary pod able to run it.

Adding a new image or updating a version

Add a line to images.txt, commit, push, then re-run step 3 — the ConfigMap's copy of images.txt has to be updated to match, since the pod reads that rather than this repo.