From b38af45c316ddedd760086aefd771d8065aa5359 Mon Sep 17 00:00:00 2001 From: Mukul Sharma Date: Sun, 13 Sep 2026 00:58:55 +0530 Subject: [PATCH] GKE: mirror base images into this cluster's Harbor over TLS Same manifest and same script as the homelab repo; what differs is the registry host and, substantially, TLS. The homelab's Harbor speaks plain HTTP and its mirror pod passes --insecure-registry. Harbor here serves a real certificate issued from the private CA Terraform created, so the pod instead mounts that CA into the dind container at /etc/docker/certs.d//ca.crt. It goes in the dind container specifically: dockerd performs the push, while docker-cli only talks to it over TCP. The README's project-creation and verification curls now use https and --cacert, since a laptop has no reason to trust this CA either. Still no robot credentials in the pod, for the same reason as the homelab's: the jenkins robot is scoped to the homelab project, and making base-images public grants anonymous pull but never push. A one-off admin docker login beats provisioning another robot for something run this rarely. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N --- README.md | 129 +++++++++++++++++++++++++++++++++++------------------- mirror.sh | 21 ++++++--- 2 files changed, 98 insertions(+), 52 deletions(-) diff --git a/README.md b/README.md index e87f0f8..d606070 100644 --- a/README.md +++ b/README.md @@ -1,35 +1,60 @@ -# devops-base-images +# devops-base-images (GKE) -One-time-setup repo for mirroring minimized base images into Harbor's -`base-images` project, so language builds (via `devops-lib`'s -`buildDocker.groovy` fallback templates) never depend on Docker Hub at -build time, and the images that actually ship are the leanest official -variant available for each language. +Mirrors minimized base images into Harbor's `base-images` project, so +language builds — `devops-lib`'s `buildDocker.groovy` fallback templates — +never depend on Docker Hub at build time, and the images that ship are the +leanest official variant for each language. -Not wired into any Jenkins pipeline — this is deliberately a manual, -occasional operation (re-run when `images.txt` changes: a new language, -a version bump, or picking up an upstream base-image update), using the -same one-off `kubectl`-based DinD pod pattern `devops-lib`'s -`build-tools.Dockerfile` bootstrap already uses, since the VM itself may -not have Docker installed directly. +GKE counterpart of the homelab repo of the same name. The manifest and the +script are identical; what differs is the registry host and, more +substantially, **TLS**. The homelab's Harbor speaks plain HTTP and its dind +pod passes `--insecure-registry`. Here Harbor has a real certificate issued +from a private CA, so instead the CA must be installed into the Docker +daemon's trust store. + +Not wired into any pipeline. This is deliberately manual and occasional — +re-run when `images.txt` changes (a new language, a version bump, or picking +up an upstream base-image update), using a throwaway `kubectl` DinD pod, +since neither the nodes nor your laptop necessarily has Docker. ## One-time setup -**1. Create the Harbor project** (public — these are just re-hosted -public images, no confidentiality concern, and public avoids needing any -pull credentials wired into every build): +**1. Create the Harbor project.** Public: these are re-hosted public images, +and public means no pull credentials need wiring into any build. Harbor UI → New Project → name `base-images` → check **Public** → Create. -Or via API: +Or via the API. `--cacert` is needed because the certificate is signed by +the private CA, which your laptop has no reason to trust: + ``` -curl -X POST -u admin:'' -H "Content-Type: application/json" \ +TF=~/Documents/localgit/gcp/toolshed-gke-infra/envs/prod/10-infra +curl --cacert <(terraform -chdir=$TF output -raw registry_ca_cert_pem) \ + -X POST -u admin:'' -H "Content-Type: application/json" \ -d '{"project_name":"base-images","metadata":{"public":"true"}}' \ - "http://harbor.192.168.1.7.nip.io/api/v2.0/projects" + "https://harbor.35.238.248.203.nip.io/api/v2.0/projects" ``` -**2. Run the mirror**, via a throwaway DinD pod (same shape as the -`build-tools` image bootstrap): +**2. Make sure the CA is available in the cluster.** The `registry-ca` +ConfigMap in the `jenkins` namespace holds the CA's public certificate and +is managed by `devops-infra-argo-config-gcp` (`extra-manifests/`). Check: + +``` +kubectl -n jenkins get configmap registry-ca +``` + +If `app-of-extra-manifests` has not synced yet, create it directly — it is +a public certificate, not a secret: + +``` +kubectl -n jenkins create configmap registry-ca \ + --from-file=ca.crt=<(terraform -chdir=$TF output -raw registry_ca_cert_pem) +``` + +**3. Run the mirror.** Note where the CA is mounted: dockerd looks for +`/etc/docker/certs.d//ca.crt`, and the path must contain the +registry hostname exactly. It goes in the **dind** container, not +`docker-cli` — dockerd performs the push, the CLI only talks to it over TCP. ``` cat <<'EOF' | kubectl apply -f - @@ -41,6 +66,7 @@ metadata: data: images.txt: | golang:1.22-alpine golang:1.22-alpine + golang:1.24-alpine golang:1.24-alpine alpine:3.20 alpine:3.20 node:20-alpine node:20-alpine python:3.12-alpine python:3.12-alpine @@ -50,7 +76,7 @@ data: mirror.sh: | #!/bin/sh set -eu - REGISTRY="harbor.192.168.1.7.nip.io" + REGISTRY="harbor.35.238.248.203.nip.io" PROJECT="base-images" while read -r src target; do [ -z "$src" ] && continue @@ -75,14 +101,17 @@ spec: image: docker:27-dind securityContext: privileged: true - args: - - "--insecure-registry=harbor.192.168.1.7.nip.io" + # No --insecure-registry, unlike the homelab: Harbor here has a real + # certificate. The CA below is what makes dockerd accept it. env: - name: DOCKER_TLS_CERTDIR value: "" volumeMounts: - name: docker-graph-storage mountPath: /var/lib/docker + - name: registry-ca + mountPath: /etc/docker/certs.d/harbor.35.238.248.203.nip.io + readOnly: true - name: docker-cli image: docker:27-cli command: ["cat"] @@ -100,26 +129,26 @@ spec: configMap: name: base-images-mirror defaultMode: 0755 + - name: registry-ca + configMap: + name: registry-ca EOF ``` -**No `harbor-robot-dockerconfig` mount here on purpose** — that robot -account is scoped only to the `homelab` project (push+pull there -specifically); it has no grant on `base-images` at all, and making -`base-images` public only grants anonymous *pull*, never push. Push -always needs real credentials scoped to that project regardless. For a -one-off manual task like this, simplest is just logging in with your -Harbor admin account directly inside the pod, rather than provisioning -a whole new robot account for something run this rarely: +**Deliberately no `harbor-robot-dockerconfig` mount.** That robot is scoped +to the `homelab` project only, and making `base-images` public grants +anonymous *pull*, never push — push always needs credentials scoped to the +project. For something run this rarely, logging in with the Harbor admin +account inside the pod beats provisioning another robot: ``` -kubectl wait --for=condition=Ready pod/base-images-mirror -n jenkins --timeout=120s +kubectl wait --for=condition=Ready pod/base-images-mirror -n jenkins --timeout=180s kubectl exec -it -n jenkins base-images-mirror -c docker-cli -- sh -c ' for i in $(seq 1 30); do docker info >/dev/null 2>&1 && break; sleep 2; done - docker login harbor.192.168.1.7.nip.io + docker login harbor.35.238.248.203.nip.io ' ``` -(enter your Harbor admin username/password when prompted) +(enter the Harbor admin username and password when prompted) ``` kubectl exec -n jenkins base-images-mirror -c docker-cli -- sh -c ' @@ -129,16 +158,26 @@ kubectl delete pod base-images-mirror -n jenkins kubectl delete configmap base-images-mirror -n jenkins ``` -**3. Verify:** -``` -curl -s http://harbor.192.168.1.7.nip.io/v2/_catalog -``` -Should list `base-images/golang`, `base-images/alpine`, -`base-images/node`, `base-images/python`, -`base-images/maven`, `base-images/eclipse-temurin`, `base-images/php`. +Pulls come from Docker Hub, which the nodes reach through Cloud NAT — the +nodes have no public IPs of their own. -## Adding a new image / updating a version +**4. Verify:** -Add a line to `images.txt`, commit, push, then re-run step 2 above (the -ConfigMap's `images.txt` needs updating to match — copy the current file -content in, same as the initial setup). +``` +curl --cacert <(terraform -chdir=$TF output -raw registry_ca_cert_pem) \ + -s https://harbor.35.238.248.203.nip.io/v2/_catalog +``` + +Should list `base-images/golang`, `base-images/alpine`, `base-images/node`, +`base-images/python`, `base-images/maven`, `base-images/eclipse-temurin` and +`base-images/php`. + +If the push fails with a certificate error, the mount path is the usual +cause: it must be `/etc/docker/certs.d/harbor.35.238.248.203.nip.io/ca.crt` +exactly, hostname included, in the dind container. + +## Adding a new image or updating a version + +Add a line to `images.txt`, commit, push, then re-run step 3 — the +ConfigMap's copy of `images.txt` has to be updated to match, since the pod +reads that rather than this repo. diff --git a/mirror.sh b/mirror.sh index 781e32d..28f8a75 100755 --- a/mirror.sh +++ b/mirror.sh @@ -1,15 +1,22 @@ #!/bin/sh # Reads images.txt and pulls/retags/pushes each entry into Harbor's # base-images project. Meant to run inside a docker-capable container -# with push credentials already available (docker-cli + dind sidecar, -# harbor-robot-dockerconfig mounted at /root/.docker/config.json) — same -# pattern devops-lib's build-tools.Dockerfile bootstrap already uses, see -# README.md's one-off kubectl pod commands. Not run through any Jenkins -# pipeline — genuinely one-time/occasional, re-run by hand when -# images.txt changes. +# with push credentials already available. On this cluster that means a +# docker-cli + dind pod where you have run `docker login` as the Harbor +# admin: the jenkins robot account is scoped to the homelab project and has +# no grant on base-images, and making base-images public grants anonymous +# pull, never push. +# +# dind must also trust the registry CA (mounted at +# /etc/docker/certs.d//ca.crt) — Harbor here serves real TLS, +# unlike the homelab, where the equivalent pod passes --insecure-registry. +# See README.md for the exact pod manifest. +# +# Not run through any Jenkins pipeline — genuinely occasional, re-run by +# hand when images.txt changes. set -eu -REGISTRY="${REGISTRY:-harbor.192.168.1.7.nip.io}" +REGISTRY="${REGISTRY:-harbor.35.238.248.203.nip.io}" PROJECT="${PROJECT:-base-images}" while read -r src target; do