gitadmin/devops-base-images-gcp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

GKE: mirror base images into this cluster's Harbor over TLS

Browse Source 
Same manifest and same script as the homelab repo; what differs is the
registry host and, substantially, TLS.

The homelab's Harbor speaks plain HTTP and its mirror pod passes
--insecure-registry. Harbor here serves a real certificate issued from the
private CA Terraform created, so the pod instead mounts that CA into the
dind container at /etc/docker/certs.d/<registry host>/ca.crt. It goes in
the dind container specifically: dockerd performs the push, while
docker-cli only talks to it over TCP.

The README's project-creation and verification curls now use https and
--cacert, since a laptop has no reason to trust this CA either.

Still no robot credentials in the pod, for the same reason as the
homelab's: the jenkins robot is scoped to the homelab project, and making
base-images public grants anonymous pull but never push. A one-off admin
docker login beats provisioning another robot for something run this
rarely.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
This commit is contained in:
Mukul Sharma
2026-09-13 00:58:55 +05:30
co-authored by Claude Opus 5
parent 8d875aa704
commit b38af45c31
2 changed files with 98 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+84 -45
View File
@@ -1,35 +1,60 @@
# devops-base-images
# devops-base-images (GKE)
One-time-setup repo for mirroring minimized base images into Harbor's
`base-images` project, so language builds (via `devops-lib`'s
`buildDocker.groovy` fallback templates) never depend on Docker Hub at
build time, and the images that actually ship are the leanest official
variant available for each language.
Mirrors minimized base images into Harbor's `base-images` project, so
language builds — `devops-lib`'s `buildDocker.groovy` fallback templates —
never depend on Docker Hub at build time, and the images that ship are the
leanest official variant for each language.
Not wired into any Jenkins pipeline — this is deliberately a manual,
occasional operation (re-run when `images.txt` changes: a new language,
a version bump, or picking up an upstream base-image update), using the
same one-off `kubectl`-based DinD pod pattern `devops-lib`'s
`build-tools.Dockerfile` bootstrap already uses, since the VM itself may
not have Docker installed directly.
GKE counterpart of the homelab repo of the same name. The manifest and the
script are identical; what differs is the registry host and, more
substantially, **TLS**. The homelab's Harbor speaks plain HTTP and its dind
pod passes `--insecure-registry`. Here Harbor has a real certificate issued
from a private CA, so instead the CA must be installed into the Docker
daemon's trust store.
Not wired into any pipeline. This is deliberately manual and occasional —
re-run when `images.txt` changes (a new language, a version bump, or picking
up an upstream base-image update), using a throwaway `kubectl` DinD pod,
since neither the nodes nor your laptop necessarily has Docker.
## One-time setup
**1. Create the Harbor project** (public — these are just re-hosted
public images, no confidentiality concern, and public avoids needing any
pull credentials wired into every build):
**1. Create the Harbor project.** Public: these are re-hosted public images,
and public means no pull credentials need wiring into any build.
Harbor UI → New Project → name `base-images` → check **Public** → Create.
Or via API:
Or via the API. `--cacert` is needed because the certificate is signed by
the private CA, which your laptop has no reason to trust:
```
curl -X POST -u admin:'<harbor admin password>' -H "Content-Type: application/json" \
TF=~/Documents/localgit/gcp/toolshed-gke-infra/envs/prod/10-infra
curl --cacert <(terraform -chdir=$TF output -raw registry_ca_cert_pem) \
-X POST -u admin:'<harbor admin password>' -H "Content-Type: application/json" \
-d '{"project_name":"base-images","metadata":{"public":"true"}}' \
"http://harbor.192.168.1.7.nip.io/api/v2.0/projects"
"https://harbor.35.238.248.203.nip.io/api/v2.0/projects"
```
**2. Run the mirror**, via a throwaway DinD pod (same shape as the
`build-tools` image bootstrap):
**2. Make sure the CA is available in the cluster.** The `registry-ca`
ConfigMap in the `jenkins` namespace holds the CA's public certificate and
is managed by `devops-infra-argo-config-gcp` (`extra-manifests/`). Check:
```
kubectl -n jenkins get configmap registry-ca
```
If `app-of-extra-manifests` has not synced yet, create it directly — it is
a public certificate, not a secret:
```
kubectl -n jenkins create configmap registry-ca \
--from-file=ca.crt=<(terraform -chdir=$TF output -raw registry_ca_cert_pem)
```
**3. Run the mirror.** Note where the CA is mounted: dockerd looks for
`/etc/docker/certs.d/<registry host>/ca.crt`, and the path must contain the
registry hostname exactly. It goes in the **dind** container, not
`docker-cli` — dockerd performs the push, the CLI only talks to it over TCP.
```
cat <<'EOF' | kubectl apply -f -
@@ -41,6 +66,7 @@ metadata:
data:
images.txt: |
golang:1.22-alpine golang:1.22-alpine
golang:1.24-alpine golang:1.24-alpine
alpine:3.20 alpine:3.20
node:20-alpine node:20-alpine
python:3.12-alpine python:3.12-alpine
@@ -50,7 +76,7 @@ data:
mirror.sh: |
#!/bin/sh
set -eu
REGISTRY="harbor.192.168.1.7.nip.io"
REGISTRY="harbor.35.238.248.203.nip.io"
PROJECT="base-images"
while read -r src target; do
[ -z "$src" ] && continue
@@ -75,14 +101,17 @@ spec:
image: docker:27-dind
securityContext:
privileged: true
args:
- "--insecure-registry=harbor.192.168.1.7.nip.io"
# No --insecure-registry, unlike the homelab: Harbor here has a real
# certificate. The CA below is what makes dockerd accept it.
env:
- name: DOCKER_TLS_CERTDIR
value: ""
volumeMounts:
- name: docker-graph-storage
mountPath: /var/lib/docker
- name: registry-ca
mountPath: /etc/docker/certs.d/harbor.35.238.248.203.nip.io
readOnly: true
- name: docker-cli
image: docker:27-cli
command: ["cat"]
@@ -100,26 +129,26 @@ spec:
configMap:
name: base-images-mirror
defaultMode: 0755
- name: registry-ca
configMap:
name: registry-ca
EOF
```
**No `harbor-robot-dockerconfig` mount here on purpose** — that robot
account is scoped only to the `homelab` project (push+pull there
specifically); it has no grant on `base-images` at all, and making
`base-images` public only grants anonymous *pull*, never push. Push
always needs real credentials scoped to that project regardless. For a
one-off manual task like this, simplest is just logging in with your
Harbor admin account directly inside the pod, rather than provisioning
a whole new robot account for something run this rarely:
**Deliberately no `harbor-robot-dockerconfig` mount.** That robot is scoped
to the `homelab` project only, and making `base-images` public grants
anonymous *pull*, never push — push always needs credentials scoped to the
project. For something run this rarely, logging in with the Harbor admin
account inside the pod beats provisioning another robot:
```
kubectl wait --for=condition=Ready pod/base-images-mirror -n jenkins --timeout=120s
kubectl wait --for=condition=Ready pod/base-images-mirror -n jenkins --timeout=180s
kubectl exec -it -n jenkins base-images-mirror -c docker-cli -- sh -c '
for i in $(seq 1 30); do docker info >/dev/null 2>&1 && break; sleep 2; done
docker login harbor.192.168.1.7.nip.io
docker login harbor.35.238.248.203.nip.io
'
```
(enter your Harbor admin username/password when prompted)
(enter the Harbor admin username and password when prompted)
```
kubectl exec -n jenkins base-images-mirror -c docker-cli -- sh -c '
@@ -129,16 +158,26 @@ kubectl delete pod base-images-mirror -n jenkins
kubectl delete configmap base-images-mirror -n jenkins
```
**3. Verify:**
```
curl -s http://harbor.192.168.1.7.nip.io/v2/_catalog
```
Should list `base-images/golang`, `base-images/alpine`,
`base-images/node`, `base-images/python`,
`base-images/maven`, `base-images/eclipse-temurin`, `base-images/php`.
Pulls come from Docker Hub, which the nodes reach through Cloud NAT — the
nodes have no public IPs of their own.
## Adding a new image / updating a version
**4. Verify:**
Add a line to `images.txt`, commit, push, then re-run step 2 above (the
ConfigMap's `images.txt` needs updating to match — copy the current file
content in, same as the initial setup).
```
curl --cacert <(terraform -chdir=$TF output -raw registry_ca_cert_pem) \
-s https://harbor.35.238.248.203.nip.io/v2/_catalog
```
Should list `base-images/golang`, `base-images/alpine`, `base-images/node`,
`base-images/python`, `base-images/maven`, `base-images/eclipse-temurin` and
`base-images/php`.
If the push fails with a certificate error, the mount path is the usual
cause: it must be `/etc/docker/certs.d/harbor.35.238.248.203.nip.io/ca.crt`
exactly, hostname included, in the dind container.
## Adding a new image or updating a version
Add a line to `images.txt`, commit, push, then re-run step 3 — the
ConfigMap's copy of `images.txt` has to be updated to match, since the pod
reads that rather than this repo.
mirror.sh
+14 -7
View File
@@ -1,15 +1,22 @@
#!/bin/sh
# Reads images.txt and pulls/retags/pushes each entry into Harbor's
# base-images project. Meant to run inside a docker-capable container
# with push credentials already available (docker-cli + dind sidecar,
# harbor-robot-dockerconfig mounted at /root/.docker/config.json) — same
# pattern devops-lib's build-tools.Dockerfile bootstrap already uses, see
# README.md's one-off kubectl pod commands. Not run through any Jenkins
# pipeline — genuinely one-time/occasional, re-run by hand when
# images.txt changes.
# with push credentials already available. On this cluster that means a
# docker-cli + dind pod where you have run `docker login` as the Harbor
# admin: the jenkins robot account is scoped to the homelab project and has
# no grant on base-images, and making base-images public grants anonymous
# pull, never push.
#
# dind must also trust the registry CA (mounted at
# /etc/docker/certs.d/<registry host>/ca.crt) — Harbor here serves real TLS,
# unlike the homelab, where the equivalent pod passes --insecure-registry.
# See README.md for the exact pod manifest.
#
# Not run through any Jenkins pipeline — genuinely occasional, re-run by
# hand when images.txt changes.
set -eu
REGISTRY="${REGISTRY:-harbor.192.168.1.7.nip.io}"
REGISTRY="${REGISTRY:-harbor.35.238.248.203.nip.io}"
PROJECT="${PROJECT:-base-images}"
while read -r src target; do