devops-base-images-gcp/README.md

devops-base-images (GKE)

Mirrors minimized base images into Harbor's base-images project, so language builds — devops-lib's buildDocker.groovy fallback templates — never depend on Docker Hub at build time, and the images that ship are the leanest official variant for each language.

GKE counterpart of the homelab repo of the same name. The manifest and the script are identical; what differs is the registry host and, more substantially, TLS. The homelab's Harbor speaks plain HTTP and its dind pod passes --insecure-registry. Here Harbor has a real certificate issued from a private CA, so instead the CA must be installed into the Docker daemon's trust store.

Not wired into any pipeline. This is deliberately manual and occasional — re-run when images.txt changes (a new language, a version bump, or picking up an upstream base-image update), using a throwaway kubectl DinD pod, since neither the nodes nor your laptop necessarily has Docker.

One-time setup

1. Create the Harbor project. Public: these are re-hosted public images, and public means no pull credentials need wiring into any build.

Harbor UI → New Project → name base-images → check Public → Create.

Or via the API. --cacert is needed because the certificate is signed by the private CA, which your laptop has no reason to trust:

TF=~/Documents/localgit/gcp/toolshed-gke-infra/envs/prod/10-infra
curl --cacert <(terraform -chdir=$TF output -raw registry_ca_cert_pem) \
  -X POST -u admin:'<harbor admin password>' -H "Content-Type: application/json" \
  -d '{"project_name":"base-images","metadata":{"public":"true"}}' \
  "https://harbor.35.238.248.203.nip.io/api/v2.0/projects"

2. Make sure the CA is available in the cluster. The registry-ca ConfigMap in the jenkins namespace holds the CA's public certificate and is managed by devops-infra-argo-config-gcp (extra-manifests/). Check:

kubectl -n jenkins get configmap registry-ca

If app-of-extra-manifests has not synced yet, create it directly — it is a public certificate, not a secret:

kubectl -n jenkins create configmap registry-ca \
  --from-file=ca.crt=<(terraform -chdir=$TF output -raw registry_ca_cert_pem)

3. Run the mirror. Note where the CA is mounted: dockerd looks for /etc/docker/certs.d/<registry host>/ca.crt, and the path must contain the registry hostname exactly. It goes in the dind container, not docker-cli — dockerd performs the push, the CLI only talks to it over TCP.

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: base-images-mirror
  namespace: jenkins
data:
  images.txt: |
    golang:1.22-alpine golang:1.22-alpine
    golang:1.24-alpine golang:1.24-alpine
    alpine:3.20 alpine:3.20
    node:20-alpine node:20-alpine
    python:3.12-alpine python:3.12-alpine
    maven:3-eclipse-temurin-21-alpine maven:3-eclipse-temurin-21-alpine
    eclipse-temurin:21-jre-alpine eclipse-temurin:21-jre-alpine
    php:8.3-cli-alpine php:8.3-cli-alpine
  mirror.sh: |
    #!/bin/sh
    set -eu
    REGISTRY="harbor.35.238.248.203.nip.io"
    PROJECT="base-images"
    while read -r src target; do
        [ -z "$src" ] && continue
        case "$src" in \#*) continue ;; esac
        dest="${REGISTRY}/${PROJECT}/${target}"
        echo "=== ${src} -> ${dest} ==="
        docker pull "$src"
        docker tag "$src" "$dest"
        docker push "$dest"
    done < images.txt
    echo "=== done ==="
---
apiVersion: v1
kind: Pod
metadata:
  name: base-images-mirror
  namespace: jenkins
spec:
  restartPolicy: Never
  containers:
    - name: docker
      image: docker:27-dind
      securityContext:
        privileged: true
      # No --insecure-registry, unlike the homelab: Harbor here has a real
      # certificate. The CA below is what makes dockerd accept it.
      env:
        - name: DOCKER_TLS_CERTDIR
          value: ""
      volumeMounts:
        - name: docker-graph-storage
          mountPath: /var/lib/docker
        - name: registry-ca
          mountPath: /etc/docker/certs.d/harbor.35.238.248.203.nip.io
          readOnly: true
    - name: docker-cli
      image: docker:27-cli
      command: ["cat"]
      tty: true
      env:
        - name: DOCKER_HOST
          value: tcp://localhost:2375
      volumeMounts:
        - name: mirror
          mountPath: /mirror
  volumes:
    - name: docker-graph-storage
      emptyDir: {}
    - name: mirror
      configMap:
        name: base-images-mirror
        defaultMode: 0755
    - name: registry-ca
      configMap:
        name: registry-ca
EOF

Deliberately no harbor-robot-dockerconfig mount. That robot is scoped to the homelab project only, and making base-images public grants anonymous pull, never push — push always needs credentials scoped to the project. For something run this rarely, logging in with the Harbor admin account inside the pod beats provisioning another robot:

kubectl wait --for=condition=Ready pod/base-images-mirror -n jenkins --timeout=180s
kubectl exec -it -n jenkins base-images-mirror -c docker-cli -- sh -c '
  for i in $(seq 1 30); do docker info >/dev/null 2>&1 && break; sleep 2; done
  docker login harbor.35.238.248.203.nip.io
'

(enter the Harbor admin username and password when prompted)

kubectl exec -n jenkins base-images-mirror -c docker-cli -- sh -c '
  cd /mirror && sh mirror.sh
'
kubectl delete pod base-images-mirror -n jenkins
kubectl delete configmap base-images-mirror -n jenkins

Pulls come from Docker Hub, which the nodes reach through Cloud NAT — the nodes have no public IPs of their own.

4. Verify:

curl --cacert <(terraform -chdir=$TF output -raw registry_ca_cert_pem) \
  -s https://harbor.35.238.248.203.nip.io/v2/_catalog

Should list base-images/golang, base-images/alpine, base-images/node, base-images/python, base-images/maven, base-images/eclipse-temurin and base-images/php.

If the push fails with a certificate error, the mount path is the usual cause: it must be /etc/docker/certs.d/harbor.35.238.248.203.nip.io/ca.crt exactly, hostname included, in the dind container.

Adding a new image or updating a version

Add a line to images.txt, commit, push, then re-run step 3 — the ConfigMap's copy of images.txt has to be updated to match, since the pod reads that rather than this repo.