devops-base-images-gcp/README.md

# devops-base-images (GKE)

Mirrors minimized base images into Harbor's `base-images` project, so
language builds — `devops-lib`'s `buildDocker.groovy` fallback templates —
never depend on Docker Hub at build time, and the images that ship are the
leanest official variant for each language.

GKE counterpart of the homelab repo of the same name. The manifest and the
script are identical; what differs is the registry host and, more
substantially, **TLS**. The homelab's Harbor speaks plain HTTP and its dind
pod passes `--insecure-registry`. Here Harbor has a real certificate issued
from a private CA, so instead the CA must be installed into the Docker
daemon's trust store.

Not wired into any pipeline. This is deliberately manual and occasional —
re-run when `images.txt` changes (a new language, a version bump, or picking
up an upstream base-image update), using a throwaway `kubectl` DinD pod,
since neither the nodes nor your laptop necessarily has Docker.

## One-time setup

**1. Create the Harbor project.** Public: these are re-hosted public images,
and public means no pull credentials need wiring into any build.

Harbor UI → New Project → name `base-images` → check **Public** → Create.

Or via the API. `--cacert` is needed because the certificate is signed by
the private CA, which your laptop has no reason to trust:

```
TF=~/Documents/localgit/gcp/toolshed-gke-infra/envs/prod/10-infra
curl --cacert <(terraform -chdir=$TF output -raw registry_ca_cert_pem) \
  -X POST -u admin:'<harbor admin password>' -H "Content-Type: application/json" \
  -d '{"project_name":"base-images","metadata":{"public":"true"}}' \
  "https://harbor.35.238.248.203.nip.io/api/v2.0/projects"
```

**2. Make sure the CA is available in the cluster.** The `registry-ca`
ConfigMap in the `jenkins` namespace holds the CA's public certificate and
is managed by `devops-infra-argo-config-gcp` (`extra-manifests/`). Check:

```
kubectl -n jenkins get configmap registry-ca
```

If `app-of-extra-manifests` has not synced yet, create it directly — it is
a public certificate, not a secret:

```
kubectl -n jenkins create configmap registry-ca \
  --from-file=ca.crt=<(terraform -chdir=$TF output -raw registry_ca_cert_pem)
```

**3. Run the mirror.** Note where the CA is mounted: dockerd looks for
`/etc/docker/certs.d/<registry host>/ca.crt`, and the path must contain the
registry hostname exactly. It goes in the **dind** container, not
`docker-cli` — dockerd performs the push, the CLI only talks to it over TCP.

```
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: base-images-mirror
  namespace: jenkins
data:
  images.txt: |
    golang:1.22-alpine golang:1.22-alpine
    golang:1.24-alpine golang:1.24-alpine
    alpine:3.20 alpine:3.20
    node:20-alpine node:20-alpine
    python:3.12-alpine python:3.12-alpine
    maven:3-eclipse-temurin-21-alpine maven:3-eclipse-temurin-21-alpine
    eclipse-temurin:21-jre-alpine eclipse-temurin:21-jre-alpine
    php:8.3-cli-alpine php:8.3-cli-alpine
  mirror.sh: |
    #!/bin/sh
    set -eu
    REGISTRY="harbor.35.238.248.203.nip.io"
    PROJECT="base-images"
    while read -r src target; do
        [ -z "$src" ] && continue
        case "$src" in \#*) continue ;; esac
        dest="${REGISTRY}/${PROJECT}/${target}"
        echo "=== ${src} -> ${dest} ==="
        docker pull "$src"
        docker tag "$src" "$dest"
        docker push "$dest"
    done < images.txt
    echo "=== done ==="
---
apiVersion: v1
kind: Pod
metadata:
  name: base-images-mirror
  namespace: jenkins
spec:
  restartPolicy: Never
  containers:
    - name: docker
      image: docker:27-dind
      securityContext:
        privileged: true
      # No --insecure-registry, unlike the homelab: Harbor here has a real
      # certificate. The CA below is what makes dockerd accept it.
      env:
        - name: DOCKER_TLS_CERTDIR
          value: ""
      volumeMounts:
        - name: docker-graph-storage
          mountPath: /var/lib/docker
        - name: registry-ca
          mountPath: /etc/docker/certs.d/harbor.35.238.248.203.nip.io
          readOnly: true
    - name: docker-cli
      image: docker:27-cli
      command: ["cat"]
      tty: true
      env:
        - name: DOCKER_HOST
          value: tcp://localhost:2375
      volumeMounts:
        - name: mirror
          mountPath: /mirror
  volumes:
    - name: docker-graph-storage
      emptyDir: {}
    - name: mirror
      configMap:
        name: base-images-mirror
        defaultMode: 0755
    - name: registry-ca
      configMap:
        name: registry-ca
EOF
```

**Deliberately no `harbor-robot-dockerconfig` mount.** That robot is scoped
to the `homelab` project only, and making `base-images` public grants
anonymous *pull*, never push — push always needs credentials scoped to the
project. For something run this rarely, logging in with the Harbor admin
account inside the pod beats provisioning another robot:

```
kubectl wait --for=condition=Ready pod/base-images-mirror -n jenkins --timeout=180s
kubectl exec -it -n jenkins base-images-mirror -c docker-cli -- sh -c '
  for i in $(seq 1 30); do docker info >/dev/null 2>&1 && break; sleep 2; done
  docker login harbor.35.238.248.203.nip.io
'
```
(enter the Harbor admin username and password when prompted)

```
kubectl exec -n jenkins base-images-mirror -c docker-cli -- sh -c '
  cd /mirror && sh mirror.sh
'
kubectl delete pod base-images-mirror -n jenkins
kubectl delete configmap base-images-mirror -n jenkins
```

Pulls come from Docker Hub, which the nodes reach through Cloud NAT — the
nodes have no public IPs of their own.

**4. Verify:**

```
curl --cacert <(terraform -chdir=$TF output -raw registry_ca_cert_pem) \
  -s https://harbor.35.238.248.203.nip.io/v2/_catalog
```

Should list `base-images/golang`, `base-images/alpine`, `base-images/node`,
`base-images/python`, `base-images/maven`, `base-images/eclipse-temurin` and
`base-images/php`.

If the push fails with a certificate error, the mount path is the usual
cause: it must be `/etc/docker/certs.d/harbor.35.238.248.203.nip.io/ca.crt`
exactly, hostname included, in the dind container.

## Adding a new image or updating a version

Add a line to `images.txt`, commit, push, then re-run step 3 — the
ConfigMap's copy of `images.txt` has to be updated to match, since the pod
reads that rather than this repo.