devops-base-images-gcp/README.md

# devops-base-images (GKE)

Owns every image the build pipeline pulls, in Harbor's `base-images`
project:

- **Mirrored language base images** (`images.txt`), so builds — `devops-lib`'s
  `buildDocker.groovy` fallback templates — never depend on Docker Hub at
  build time, and what ships is the leanest official variant per language.
- **`build-tools`** (`build-tools.Dockerfile`), the image the build pod runs
  its own shell steps in.

Both are built or mirrored by hand from here, and `devops-lib` only ever
references the results by tag. `build-tools` lives here rather than in
devops-lib because it is the same kind of artefact as the mirrors, and
because a Jenkins job cannot build the image its own build pod needs.

GKE counterpart of the homelab repo of the same name. The manifest and the
script are identical; what differs is the registry host and, more
substantially, **TLS**. The homelab's Harbor speaks plain HTTP and its dind
pod passes `--insecure-registry`. Here Harbor has a real certificate issued
from a private CA, so instead the CA must be installed into the Docker
daemon's trust store.

Not wired into any pipeline. This is deliberately manual and occasional —
re-run when `images.txt` changes (a new language, a version bump, or picking
up an upstream base-image update), using a throwaway `kubectl` DinD pod,
since neither the nodes nor your laptop necessarily has Docker.

## One-time setup

**1. Create the Harbor project.** Public: these are re-hosted public images,
and public means no pull credentials need wiring into any build.

Harbor UI → New Project → name `base-images` → check **Public** → Create.

Or via the API. `--cacert` is needed because the certificate is signed by
the private CA, which your laptop has no reason to trust:

```
TF=~/Documents/localgit/gcp/toolshed-gke-infra/envs/prod/10-infra
curl --cacert <(terraform -chdir=$TF output -raw registry_ca_cert_pem) \
  -X POST -u admin:'<harbor admin password>' -H "Content-Type: application/json" \
  -d '{"project_name":"base-images","metadata":{"public":"true"}}' \
  "https://harbor.35.238.248.203.nip.io/api/v2.0/projects"
```

**2. Make sure the CA is available in the cluster.** The `registry-ca`
ConfigMap in the `jenkins` namespace holds the CA's public certificate and
is managed by `devops-infra-argo-config-gcp` (`extra-manifests/`). Check:

```
kubectl -n jenkins get configmap registry-ca
```

If `app-of-extra-manifests` has not synced yet, create it directly — it is
a public certificate, not a secret:

```
kubectl -n jenkins create configmap registry-ca \
  --from-file=ca.crt=<(terraform -chdir=$TF output -raw registry_ca_cert_pem)
```

**3. Run the mirror.** Note where the CA is mounted: dockerd looks for
`/etc/docker/certs.d/<registry host>/ca.crt`, and the path must contain the
registry hostname exactly. It goes in the **dind** container, not
`docker-cli` — dockerd performs the push, the CLI only talks to it over TCP.

```
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: base-images-mirror
  namespace: jenkins
data:
  images.txt: |
    golang:1.22-alpine golang:1.22-alpine
    golang:1.24-alpine golang:1.24-alpine
    alpine:3.20 alpine:3.20
    node:20-alpine node:20-alpine
    python:3.12-alpine python:3.12-alpine
    maven:3-eclipse-temurin-21-alpine maven:3-eclipse-temurin-21-alpine
    eclipse-temurin:21-jre-alpine eclipse-temurin:21-jre-alpine
    php:8.3-cli-alpine php:8.3-cli-alpine
    docker:27-cli docker:27-cli
  mirror.sh: |
    #!/bin/sh
    set -eu
    REGISTRY="harbor.35.238.248.203.nip.io"
    PROJECT="base-images"
    while read -r src target; do
        [ -z "$src" ] && continue
        case "$src" in \#*) continue ;; esac
        dest="${REGISTRY}/${PROJECT}/${target}"
        echo "=== ${src} -> ${dest} ==="
        docker pull "$src"
        docker tag "$src" "$dest"
        docker push "$dest"
    done < images.txt
    echo "=== done ==="
---
apiVersion: v1
kind: Pod
metadata:
  name: base-images-mirror
  namespace: jenkins
spec:
  restartPolicy: Never
  containers:
    - name: docker
      image: docker:27-dind
      securityContext:
        privileged: true
      # No --insecure-registry, unlike the homelab: Harbor here has a real
      # certificate. The CA below is what makes dockerd accept it.
      env:
        - name: DOCKER_TLS_CERTDIR
          value: ""
      volumeMounts:
        - name: docker-graph-storage
          mountPath: /var/lib/docker
        - name: registry-ca
          mountPath: /etc/docker/certs.d/harbor.35.238.248.203.nip.io
          readOnly: true
    - name: docker-cli
      image: docker:27-cli
      command: ["cat"]
      tty: true
      env:
        - name: DOCKER_HOST
          value: tcp://localhost:2375
      volumeMounts:
        - name: mirror
          mountPath: /mirror
  volumes:
    - name: docker-graph-storage
      emptyDir: {}
    - name: mirror
      configMap:
        name: base-images-mirror
        defaultMode: 0755
    - name: registry-ca
      configMap:
        name: registry-ca
EOF
```

**Deliberately no `harbor-robot-dockerconfig` mount.** That robot is scoped
to the `apps-registry` project only, and making `base-images` public grants
anonymous *pull*, never push — push always needs credentials scoped to the
project. For something run this rarely, logging in with the Harbor admin
account inside the pod beats provisioning another robot:

```
kubectl wait --for=condition=Ready pod/base-images-mirror -n jenkins --timeout=180s
kubectl exec -it -n jenkins base-images-mirror -c docker-cli -- sh -c '
  for i in $(seq 1 30); do docker info >/dev/null 2>&1 && break; sleep 2; done
  docker login harbor.35.238.248.203.nip.io
'
```
(enter the Harbor admin username and password when prompted)

```
kubectl exec -n jenkins base-images-mirror -c docker-cli -- sh -c '
  cd /mirror && sh mirror.sh
'
kubectl delete pod base-images-mirror -n jenkins
kubectl delete configmap base-images-mirror -n jenkins
```

Pulls come from Docker Hub, which the nodes reach through Cloud NAT — the
nodes have no public IPs of their own.

**4. Verify:**

```
curl --cacert <(terraform -chdir=$TF output -raw registry_ca_cert_pem) \
  -s https://harbor.35.238.248.203.nip.io/v2/_catalog
```

Should list `base-images/golang`, `base-images/alpine`, `base-images/node`,
`base-images/python`, `base-images/maven`, `base-images/eclipse-temurin` and
`base-images/php`.

If the push fails with a certificate error, the mount path is the usual
cause: it must be `/etc/docker/certs.d/harbor.35.238.248.203.nip.io/ca.crt`
exactly, hostname included, in the dind container.

## Building `build-tools`

Do this after the mirror, since it builds `FROM` a mirrored image.

The pod is the same shape as the mirror pod — privileged dind, the CA
mounted so pushes are trusted, a docker-cli sidecar — with the Dockerfile
supplied through a ConfigMap. Create it from the file in this repo so the
two cannot drift:

```
kubectl -n jenkins create configmap build-tools-src \
  --from-file=Dockerfile=build-tools.Dockerfile

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: build-tools-build
  namespace: jenkins
spec:
  restartPolicy: Never
  containers:
    - name: docker
      image: docker:27-dind
      securityContext:
        privileged: true
      env:
        - name: DOCKER_TLS_CERTDIR
          value: ""
      volumeMounts:
        - name: docker-graph-storage
          mountPath: /var/lib/docker
        - name: registry-ca
          mountPath: /etc/docker/certs.d/harbor.35.238.248.203.nip.io
          readOnly: true
    - name: docker-cli
      image: docker:27-cli
      command: ["cat"]
      tty: true
      env:
        - name: DOCKER_HOST
          value: tcp://localhost:2375
        # The classic builder. BuildKit wants to write its own state under
        # /root/.docker, which is read-only wherever the push credentials are
        # mounted as a file — the homelab hit exactly this.
        - name: DOCKER_BUILDKIT
          value: "0"
      volumeMounts:
        - name: src
          mountPath: /src
  volumes:
    - name: docker-graph-storage
      emptyDir: {}
    - name: src
      configMap:
        name: build-tools-src
    - name: registry-ca
      configMap:
        name: registry-ca
EOF
```

Then log in and build. The tag is deliberate: **bump it rather than
overwriting**, because `devops-lib-gcp`'s `dind-pod.yaml` pins this tag and
an overwritten tag rolls out silently on the next pod start.

```
kubectl wait --for=condition=Ready pod/build-tools-build -n jenkins --timeout=180s
kubectl exec -it -n jenkins build-tools-build -c docker-cli -- sh -c '
  for i in $(seq 1 30); do docker info >/dev/null 2>&1 && break; sleep 2; done
  docker login harbor.35.238.248.203.nip.io'

kubectl exec -n jenkins build-tools-build -c docker-cli -- sh -c '
  cd /src &&
  docker build -t harbor.35.238.248.203.nip.io/base-images/build-tools:1 . &&
  docker push harbor.35.238.248.203.nip.io/base-images/build-tools:1'

kubectl delete pod build-tools-build -n jenkins
kubectl delete configmap build-tools-src -n jenkins
```

Verify it has what the stages expect:

```
kubectl run bt --rm -it --restart=Never \
  --image=harbor.35.238.248.203.nip.io/base-images/build-tools:1 \
  -- sh -c 'git --version && yq --version && python3 -V && bash --version | head -1'
```

That pull needs no credentials — `base-images` is public — and the nodes
already trust the CA, which is what makes an ordinary pod able to run it.

## Adding a new image or updating a version

Add a line to `images.txt`, commit, push, then re-run step 3 — the
ConfigMap's copy of `images.txt` has to be updated to match, since the pod
reads that rather than this repo.