gitadmin/devops-base-images-gcp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

GKE: mirror base images into this cluster's Harbor over TLS

Browse Source 
Same manifest and same script as the homelab repo; what differs is the
registry host and, substantially, TLS.

The homelab's Harbor speaks plain HTTP and its mirror pod passes
--insecure-registry. Harbor here serves a real certificate issued from the
private CA Terraform created, so the pod instead mounts that CA into the
dind container at /etc/docker/certs.d/<registry host>/ca.crt. It goes in
the dind container specifically: dockerd performs the push, while
docker-cli only talks to it over TCP.

The README's project-creation and verification curls now use https and
--cacert, since a laptop has no reason to trust this CA either.

Still no robot credentials in the pod, for the same reason as the
homelab's: the jenkins robot is scoped to the homelab project, and making
base-images public grants anonymous pull but never push. A one-off admin
docker login beats provisioning another robot for something run this
rarely.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
This commit is contained in:
Mukul Sharma
2026-09-13 00:58:55 +05:30
co-authored by Claude Opus 5
parent 8d875aa704
commit b38af45c31
2 changed files with 98 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+84 -45
View File
@@ -1,35 +1,60 @@
# devops-base-images # devops-base-images (GKE)
One-time-setup repo for mirroring minimized base images into Harbor's Mirrors minimized base images into Harbor's `base-images` project, so
`base-images` project, so language builds (via `devops-lib`'s language builds — `devops-lib`'s `buildDocker.groovy` fallback templates —
`buildDocker.groovy` fallback templates) never depend on Docker Hub at never depend on Docker Hub at build time, and the images that ship are the
build time, and the images that actually ship are the leanest official leanest official variant for each language.
variant available for each language.
Not wired into any Jenkins pipeline — this is deliberately a manual, GKE counterpart of the homelab repo of the same name. The manifest and the
occasional operation (re-run when `images.txt` changes: a new language, script are identical; what differs is the registry host and, more
a version bump, or picking up an upstream base-image update), using the substantially, **TLS**. The homelab's Harbor speaks plain HTTP and its dind
same one-off `kubectl`-based DinD pod pattern `devops-lib`'s pod passes `--insecure-registry`. Here Harbor has a real certificate issued
`build-tools.Dockerfile` bootstrap already uses, since the VM itself may from a private CA, so instead the CA must be installed into the Docker
not have Docker installed directly. daemon's trust store.
Not wired into any pipeline. This is deliberately manual and occasional —
re-run when `images.txt` changes (a new language, a version bump, or picking
up an upstream base-image update), using a throwaway `kubectl` DinD pod,
since neither the nodes nor your laptop necessarily has Docker.
## One-time setup ## One-time setup
**1. Create the Harbor project** (public — these are just re-hosted **1. Create the Harbor project.** Public: these are re-hosted public images,
public images, no confidentiality concern, and public avoids needing any and public means no pull credentials need wiring into any build.
pull credentials wired into every build):
Harbor UI → New Project → name `base-images` → check **Public** → Create. Harbor UI → New Project → name `base-images` → check **Public** → Create.
Or via API: Or via the API. `--cacert` is needed because the certificate is signed by
the private CA, which your laptop has no reason to trust:
``` ```
curl -X POST -u admin:'<harbor admin password>' -H "Content-Type: application/json" \ TF=~/Documents/localgit/gcp/toolshed-gke-infra/envs/prod/10-infra
curl --cacert <(terraform -chdir=$TF output -raw registry_ca_cert_pem) \
-X POST -u admin:'<harbor admin password>' -H "Content-Type: application/json" \
-d '{"project_name":"base-images","metadata":{"public":"true"}}' \ -d '{"project_name":"base-images","metadata":{"public":"true"}}' \
"http://harbor.192.168.1.7.nip.io/api/v2.0/projects" "https://harbor.35.238.248.203.nip.io/api/v2.0/projects"
``` ```
**2. Run the mirror**, via a throwaway DinD pod (same shape as the **2. Make sure the CA is available in the cluster.** The `registry-ca`
`build-tools` image bootstrap): ConfigMap in the `jenkins` namespace holds the CA's public certificate and
is managed by `devops-infra-argo-config-gcp` (`extra-manifests/`). Check:
```
kubectl -n jenkins get configmap registry-ca
```
If `app-of-extra-manifests` has not synced yet, create it directly — it is
a public certificate, not a secret:
```
kubectl -n jenkins create configmap registry-ca \
--from-file=ca.crt=<(terraform -chdir=$TF output -raw registry_ca_cert_pem)
```
**3. Run the mirror.** Note where the CA is mounted: dockerd looks for
`/etc/docker/certs.d/<registry host>/ca.crt`, and the path must contain the
registry hostname exactly. It goes in the **dind** container, not
`docker-cli` — dockerd performs the push, the CLI only talks to it over TCP.
``` ```
cat <<'EOF' | kubectl apply -f - cat <<'EOF' | kubectl apply -f -
@@ -41,6 +66,7 @@ metadata:
data: data:
images.txt: | images.txt: |
golang:1.22-alpine golang:1.22-alpine golang:1.22-alpine golang:1.22-alpine
golang:1.24-alpine golang:1.24-alpine
alpine:3.20 alpine:3.20 alpine:3.20 alpine:3.20
node:20-alpine node:20-alpine node:20-alpine node:20-alpine
python:3.12-alpine python:3.12-alpine python:3.12-alpine python:3.12-alpine
@@ -50,7 +76,7 @@ data:
mirror.sh: | mirror.sh: |
#!/bin/sh #!/bin/sh
set -eu set -eu
REGISTRY="harbor.192.168.1.7.nip.io" REGISTRY="harbor.35.238.248.203.nip.io"
PROJECT="base-images" PROJECT="base-images"
while read -r src target; do while read -r src target; do
[ -z "$src" ] && continue [ -z "$src" ] && continue
@@ -75,14 +101,17 @@ spec:
image: docker:27-dind image: docker:27-dind
securityContext: securityContext:
privileged: true privileged: true
args: # No --insecure-registry, unlike the homelab: Harbor here has a real
- "--insecure-registry=harbor.192.168.1.7.nip.io" # certificate. The CA below is what makes dockerd accept it.
env: env:
- name: DOCKER_TLS_CERTDIR - name: DOCKER_TLS_CERTDIR
value: "" value: ""
volumeMounts: volumeMounts:
- name: docker-graph-storage - name: docker-graph-storage
mountPath: /var/lib/docker mountPath: /var/lib/docker
- name: registry-ca
mountPath: /etc/docker/certs.d/harbor.35.238.248.203.nip.io
readOnly: true
- name: docker-cli - name: docker-cli
image: docker:27-cli image: docker:27-cli
command: ["cat"] command: ["cat"]
@@ -100,26 +129,26 @@ spec:
configMap: configMap:
name: base-images-mirror name: base-images-mirror
defaultMode: 0755 defaultMode: 0755
- name: registry-ca
configMap:
name: registry-ca
EOF EOF
``` ```
**No `harbor-robot-dockerconfig` mount here on purpose** — that robot **Deliberately no `harbor-robot-dockerconfig` mount.** That robot is scoped
account is scoped only to the `homelab` project (push+pull there to the `homelab` project only, and making `base-images` public grants
specifically); it has no grant on `base-images` at all, and making anonymous *pull*, never push — push always needs credentials scoped to the
`base-images` public only grants anonymous *pull*, never push. Push project. For something run this rarely, logging in with the Harbor admin
always needs real credentials scoped to that project regardless. For a account inside the pod beats provisioning another robot:
one-off manual task like this, simplest is just logging in with your
Harbor admin account directly inside the pod, rather than provisioning
a whole new robot account for something run this rarely:
``` ```
kubectl wait --for=condition=Ready pod/base-images-mirror -n jenkins --timeout=120s kubectl wait --for=condition=Ready pod/base-images-mirror -n jenkins --timeout=180s
kubectl exec -it -n jenkins base-images-mirror -c docker-cli -- sh -c ' kubectl exec -it -n jenkins base-images-mirror -c docker-cli -- sh -c '
for i in $(seq 1 30); do docker info >/dev/null 2>&1 && break; sleep 2; done for i in $(seq 1 30); do docker info >/dev/null 2>&1 && break; sleep 2; done
docker login harbor.192.168.1.7.nip.io docker login harbor.35.238.248.203.nip.io
' '
``` ```
(enter your Harbor admin username/password when prompted) (enter the Harbor admin username and password when prompted)
``` ```
kubectl exec -n jenkins base-images-mirror -c docker-cli -- sh -c ' kubectl exec -n jenkins base-images-mirror -c docker-cli -- sh -c '
@@ -129,16 +158,26 @@ kubectl delete pod base-images-mirror -n jenkins
kubectl delete configmap base-images-mirror -n jenkins kubectl delete configmap base-images-mirror -n jenkins
``` ```
**3. Verify:** Pulls come from Docker Hub, which the nodes reach through Cloud NAT — the
``` nodes have no public IPs of their own.
curl -s http://harbor.192.168.1.7.nip.io/v2/_catalog
```
Should list `base-images/golang`, `base-images/alpine`,
`base-images/node`, `base-images/python`,
`base-images/maven`, `base-images/eclipse-temurin`, `base-images/php`.
## Adding a new image / updating a version **4. Verify:**
Add a line to `images.txt`, commit, push, then re-run step 2 above (the ```
ConfigMap's `images.txt` needs updating to match — copy the current file curl --cacert <(terraform -chdir=$TF output -raw registry_ca_cert_pem) \
content in, same as the initial setup). -s https://harbor.35.238.248.203.nip.io/v2/_catalog
```
Should list `base-images/golang`, `base-images/alpine`, `base-images/node`,
`base-images/python`, `base-images/maven`, `base-images/eclipse-temurin` and
`base-images/php`.
If the push fails with a certificate error, the mount path is the usual
cause: it must be `/etc/docker/certs.d/harbor.35.238.248.203.nip.io/ca.crt`
exactly, hostname included, in the dind container.
## Adding a new image or updating a version
Add a line to `images.txt`, commit, push, then re-run step 3 — the
ConfigMap's copy of `images.txt` has to be updated to match, since the pod
reads that rather than this repo.
mirror.sh
+14 -7
View File
@@ -1,15 +1,22 @@
#!/bin/sh #!/bin/sh
# Reads images.txt and pulls/retags/pushes each entry into Harbor's # Reads images.txt and pulls/retags/pushes each entry into Harbor's
# base-images project. Meant to run inside a docker-capable container # base-images project. Meant to run inside a docker-capable container
# with push credentials already available (docker-cli + dind sidecar, # with push credentials already available. On this cluster that means a
# harbor-robot-dockerconfig mounted at /root/.docker/config.json) — same # docker-cli + dind pod where you have run `docker login` as the Harbor
# pattern devops-lib's build-tools.Dockerfile bootstrap already uses, see # admin: the jenkins robot account is scoped to the homelab project and has
# README.md's one-off kubectl pod commands. Not run through any Jenkins # no grant on base-images, and making base-images public grants anonymous
# pipeline — genuinely one-time/occasional, re-run by hand when # pull, never push.
# images.txt changes. #
# dind must also trust the registry CA (mounted at
# /etc/docker/certs.d/<registry host>/ca.crt) — Harbor here serves real TLS,
# unlike the homelab, where the equivalent pod passes --insecure-registry.
# See README.md for the exact pod manifest.
#
# Not run through any Jenkins pipeline — genuinely occasional, re-run by
# hand when images.txt changes.
set -eu set -eu
REGISTRY="${REGISTRY:-harbor.192.168.1.7.nip.io}" REGISTRY="${REGISTRY:-harbor.35.238.248.203.nip.io}"
PROJECT="${PROJECT:-base-images}" PROJECT="${PROJECT:-base-images}"
while read -r src target; do while read -r src target; do