GKE: point the pipeline at this cluster's registry, over TLS

The GCP counterpart of devops-lib. Registered in Jenkins under the same
name, so consuming repos need no change: the two-line Jenkinsfile is
identical on both clusters, and which library it resolves to is a property
of the Jenkins running it.

Substantive changes, all consequences of GKE being a real cloud:

- Harbor speaks TLS here, so dind no longer passes --insecure-registry.
  It mounts the private CA at
  /etc/docker/certs.d/harbor.35.238.248.203.nip.io/ca.crt instead, from the
  registry-ca ConfigMap. This is not redundant with the node pool's trust:
  that covers pulls, performed by containerd on the node, while the push
  comes from dockerd in the build pod with its own trust store. Without it,
  pushes fail TLS verification while pulls of the same image succeed —
  which reads like a broken registry rather than a missing trust anchor.

- The registry hostname changes in the push target and all five fallback
  Dockerfiles. It still must be spelled identically everywhere, because
  Docker matches credentials and trust by exact hostname.

- helm_repo_url moves to cluster DNS. That clone runs in a build pod, so
  sending it out through the ingress and back would make the pipeline
  depend on Contour for pod-to-pod traffic. syncArgoApp already addressed
  ArgoCD this way and needed no change.

- build-tools.Dockerfile is removed: devops-base-images-gcp owns it now,
  next to the mirrored base images, and the pod references the result by
  tag at base-images/build-tools:1. The base-images project is public, so
  the pod can pull it before it has any credentials.

Verified: no homelab addresses remain; the pod template parses with the CA
mount, the new image and no insecure-registry flag; and every fallback
template's base image, with the default version buildDocker would pick, is
present in the mirror manifest — an unmirrored tag now fails the build
rather than silently falling back to Docker Hub.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N

README.md

@@ -1,11 +1,32 @@
-# devops-lib
+# devops-lib (GKE)
 
-Jenkins Shared Library for this homelab's CI/CD pipeline. Adapted from a
+Jenkins Shared Library for the CI/CD pipeline on the GKE cluster. The GCP
-much larger, company-wide library — everything not needed for a
+counterpart of the homelab repo of the same name, and a copy rather than a
-single-node homelab (GKE/EKS, JFrog, S3, Sonar, org-specific BU/team
+shared repo because several values here are cluster-specific in ways that
-validation, and ~65 other files covering languages/deploy-targets this
+would break the other cluster if crossed over.
-setup never uses) has been removed rather than carried along unused; see
+
-git history if any of that is ever worth reviving.
+Register it in Jenkins under the name `devops-lib`, exactly as on the
+homelab. Consuming repos then need no change at all — the same two-line
+Jenkinsfile works on either cluster, and which library it resolves to is a
+property of the Jenkins it runs on.
+
+**What differs from the homelab copy**, all of it a consequence of GKE
+being a real cloud rather than one VM:
+
+- **The registry hostname** is `harbor.35.238.248.203.nip.io`, in the push
+  target and in all five fallback Dockerfiles.
+- **Harbor speaks TLS.** The homelab's dind passes `--insecure-registry`;
+  here the pod mounts the private CA into dockerd's trust store instead.
+  Node trust covers pulls only — a push is a separate client.
+- **`helm_repo_url` uses cluster DNS**, since the clone happens inside a
+  build pod. The homelab points it at an ingress hostname.
+- **`build-tools` is not in this repo.** It lives in
+  `devops-base-images-gcp` alongside the mirrored base images, and is
+  referenced here only by tag.
+
+Everything else — the stage flow, the fallback templates, the hooks
+contract — is unchanged. The library was already adapted from a much
+larger, company-wide one; see git history for what was removed.
 
 ## Using it in a service repo
 
@@ -26,7 +47,7 @@ in after checkout; repo-committed values win over the Jenkinsfile call).
 | `service_name` | `repo_name` | Second path segment under `devops-helm-charts/values/` |
 | `argo_app_name` | `repo_name` | Must match the ArgoCD Application's `metadata.name` |
 | `harbor_project` | `homelab` | Must be an existing, public Harbor project |
-| `helm_repo_url` | `devops-helm-charts` on this Gitea | — |
+| `helm_repo_url` | `devops-helm-charts-gcp`, over cluster DNS | `http://gitea-http.gitea.svc.cluster.local:3000/gitadmin/…` — pod-to-pod, so it never leaves the cluster and comes back through the ingress |
 | `image_tag_yq_path` | `.deployment.image.tag` | **Override this if the app's chart isn't `1.0.0`** — e.g. `sts-2.0.0` uses `.podtemplate.image.tag` instead. Getting this wrong doesn't fail loudly: `yq -i` creates the path if missing rather than erroring, silently leaving the real field un-bumped. |
 | `dockerBuildVersion` | none | Only read when the repo has **no Dockerfile of its own** — picks a fallback template (see below). No default; either ship a Dockerfile or set this. |
 
@@ -49,7 +70,9 @@ a `podTemplate` (`resources/org/homelab/dind-pod.yaml`) via
   based on `dockerBuildVersion` (e.g. `go-1.22`, `node-20`,
   `python-3.12`, `java-21`, `php-8.3`). All fallback templates pull base
   images from Harbor's `base-images` project (mirrored via the separate
-  `devops-base-images` repo), not Docker Hub directly.
+  `devops-base-images-gcp` repo), not Docker Hub directly. Only versions
+  actually mirrored there resolve — an unmirrored tag fails the build
+  rather than silently falling back to Docker Hub.
 - **`updateHelmTag`** — clones `devops-helm-charts`, bumps the image tag
   via `yq` at `image_tag_yq_path`, commits, pushes to `main`.
 - **`syncArgoApp`** — calls the ArgoCD REST API to sync `argo_app_name`.
@@ -65,8 +88,29 @@ a `podTemplate` (`resources/org/homelab/dind-pod.yaml`) via
 
 ## Build-tools image
 
-`resources/org/homelab/build-tools.Dockerfile` bakes git/yq/bash/
+The `docker-cli` container runs
-python3+pip/venv/curl into the `docker-cli` container's image, so
+`harbor.35.238.248.203.nip.io/base-images/build-tools:1`, which bakes in
-nothing gets installed on demand on every single build. Built and pushed
+git, yq, bash, python3 with pip and venv, and curl, so nothing is installed
-manually (not through any Jenkins job) — see that file's own header
+on demand on every build.
-comment.
+
+**It is not built here.** The Dockerfile lives in `devops-base-images-gcp`,
+next to the mirrored base images, because it is the same kind of artefact:
+built by hand, occasionally, and pushed to Harbor. A Jenkins job could not
+build it anyway — it is the image Jenkins builds *in*.
+
+`dind-pod.yaml` pins the tag, so rebuilding the image rolls nothing out
+until that pin is bumped. Bump the tag rather than overwriting one.
+
+## Registry trust
+
+`dind-pod.yaml` mounts the `registry-ca` ConfigMap (published by
+`devops-infra-argo-config-gcp`) into the dind container at
+`/etc/docker/certs.d/harbor.35.238.248.203.nip.io/ca.crt`.
+
+Without it, pushes fail TLS verification while pulls of the same image
+succeed, which reads like a broken registry. The reason is that the two are
+different clients: pulls are performed by containerd on the node, which was
+told to trust this CA when the node pool was created, whereas the push comes
+from dockerd inside the build pod, which has its own trust store. The
+directory name must be the registry hostname exactly — dockerd looks the
+path up by host and silently ignores a mismatch.

resources/com/homelab/go-Dockerfile

@@ -13,14 +13,14 @@
 # a dockerBuildVersion whose tag isn't in devops-base-images/images.txt
 # yet needs that added and re-mirrored first, unlike pulling straight
 # from Docker Hub where any tag "just worked".
-FROM harbor.192.168.1.7.nip.io/base-images/golang:${version}-alpine AS build
+FROM harbor.35.238.248.203.nip.io/base-images/golang:${version}-alpine AS build
 WORKDIR /src
 COPY go.mod go.sum* ./
 RUN go mod download 2>/dev/null || true
 COPY . .
 RUN CGO_ENABLED=0 go build -o /app .
 
-FROM harbor.192.168.1.7.nip.io/base-images/alpine:3.20
+FROM harbor.35.238.248.203.nip.io/base-images/alpine:3.20
 COPY --from=build /app /app
 EXPOSE 8080
 ENTRYPOINT ["/app"]

resources/com/homelab/java-Dockerfile

@@ -8,14 +8,14 @@
 # to Maven Central for plugins/dependencies during the build regardless
 # of base image — this only removes the Docker Hub dependency for the
 # base image layer, not package-registry traffic during the build.
-FROM harbor.192.168.1.7.nip.io/base-images/maven:3-eclipse-temurin-${version}-alpine AS build
+FROM harbor.35.238.248.203.nip.io/base-images/maven:3-eclipse-temurin-${version}-alpine AS build
 WORKDIR /src
 COPY pom.xml .
 RUN mvn -B dependency:go-offline
 COPY . .
 RUN mvn -B package -DskipTests
 
-FROM harbor.192.168.1.7.nip.io/base-images/eclipse-temurin:${version}-jre-alpine
+FROM harbor.35.238.248.203.nip.io/base-images/eclipse-temurin:${version}-jre-alpine
 WORKDIR /app
 COPY --from=build /src/target/*.jar app.jar
 EXPOSE 8080

resources/com/homelab/node-Dockerfile

@@ -5,14 +5,14 @@
 # Assumes a standard `npm run build` + `npm start` repo. Switched from
 # node:*-slim (Debian) to node:*-alpine for both stages — smaller, still
 # keeps a shell for kubectl exec debugging (not distroless).
-FROM harbor.192.168.1.7.nip.io/base-images/node:${version}-alpine AS build
+FROM harbor.35.238.248.203.nip.io/base-images/node:${version}-alpine AS build
 WORKDIR /app
 COPY package*.json ./
 RUN npm ci
 COPY . .
 RUN npm run build --if-present
 
-FROM harbor.192.168.1.7.nip.io/base-images/node:${version}-alpine
+FROM harbor.35.238.248.203.nip.io/base-images/node:${version}-alpine
 WORKDIR /app
 COPY --from=build /app .
 ENV NODE_ENV=production

resources/com/homelab/php-Dockerfile

@@ -36,7 +36,7 @@
 # very comment did so), new edits to this header should avoid typing
 # the character at all — write "dollar sign" in words instead of using
 # the glyph.
-FROM harbor.192.168.1.7.nip.io/base-images/php:${version}-cli-alpine
+FROM harbor.35.238.248.203.nip.io/base-images/php:${version}-cli-alpine
 WORKDIR /var/www/html
 RUN apk add --no-cache --virtual .build-deps \$PHPIZE_DEPS \
     && docker-php-ext-install pdo pdo_mysql \

resources/com/homelab/python-Dockerfile

@@ -9,7 +9,7 @@
 # extensions that only ship glibc wheels may need musl-dev/gcc added
 # here to build from source on Alpine — fine for this repo's pure-Python
 # deps, worth knowing if a future repo's requirements.txt needs more.
-FROM harbor.192.168.1.7.nip.io/base-images/python:${version}-alpine
+FROM harbor.35.238.248.203.nip.io/base-images/python:${version}-alpine
 WORKDIR /app
 COPY requirements.txt .
 RUN pip install --no-cache-dir -r requirements.txt

resources/org/homelab/build-tools.Dockerfile

@@ -1,13 +0,0 @@
-# Custom docker-cli image for dind-pod.yaml's docker-cli container —
-# bakes in everything the pipeline stages need at runtime (git, yq,
-# bash, python3 + pip/venv for runHooks' python hooks, curl) so nothing
-# gets apk-installed or curl-downloaded on every single build. Rebuild
-# and push this (see the one-off build commands in the commit that
-# added this file) whenever this list changes; dind-pod.yaml pins the
-# resulting image tag explicitly, so a rebuild doesn't silently roll
-# out until that pin is also bumped.
-FROM docker:27-cli
-
-RUN apk add --no-cache git bash python3 py3-pip py3-virtualenv curl \
-    && curl -sL -o /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 \
-    && chmod +x /usr/local/bin/yq

resources/org/homelab/dind-pod.yaml

@@ -15,32 +15,47 @@ spec:
       image: docker:27-dind
       securityContext:
         privileged: true
-      # Harbor's ingress serves plain HTTP too (TLS disabled cluster-wide
+      # No --insecure-registry, unlike the homelab: Harbor here serves a real
-      # by design — see claude.md's "everything is plain HTTP" note).
+      # certificate, issued by cert-manager from the private CA that the node
-      # Docker defaults to attempting HTTPS against any bare registry
+      # pool was told to trust when it was created.
-      # hostname regardless of network path, so this is needed
+      #
-      # regardless of which hostname is used — was previously
+      # That node trust covers image PULLS, which containerd performs on the
-      # harbor-core.harbor.svc.cluster.local (cluster-internal Service
+      # node. This push is a different client — dockerd, inside this pod,
-      # DNS, works from this pod but not from the node's own containerd
+      # with its own trust store and no knowledge of what the node trusts —
-      # when pulling for a real Deployment); switched to the Contour
+      # so it needs the CA mounted itself. dockerd looks it up at
-      # ingress hostname so push and pull can share one reference.
+      # /etc/docker/certs.d/<registry host>/ca.crt, and the directory name
-      args:
+      # must be the registry hostname exactly; anything else is silently
-        - "--insecure-registry=harbor.192.168.1.7.nip.io"
+      # ignored, and the push then fails TLS verification while a pull of the
+      # very same image works.
+      #
+      # The registry hostname (rather than harbor-core.harbor.svc.cluster.local)
+      # carries over unchanged from the homelab, for a reason that still
+      # holds: cluster DNS resolves from this pod but not from the node's
+      # containerd doing the real Deployment pull, and Docker matches both
+      # stored credentials and trust by exact hostname — so push and pull
+      # have to name the registry identically.
       env:
         - name: DOCKER_TLS_CERTDIR
           value: ""
       volumeMounts:
         - name: docker-graph-storage
           mountPath: /var/lib/docker
+        - name: registry-ca
+          mountPath: /etc/docker/certs.d/harbor.35.238.248.203.nip.io
+          readOnly: true
     - name: docker-cli
-      # Custom image (see build-tools.Dockerfile in this same directory)
+      # Custom image, built and pushed by hand from devops-base-images-gcp
-      # — bakes in git/yq/bash/python3+pip/venv/curl so nothing needs
+      # (build-tools.Dockerfile there) — bakes in git/yq/bash/python3 with
-      # apk-installing or curl-downloading on every single build (was
+      # pip and venv/curl so nothing needs installing on every single build,
-      # slow and, per its own point, defeats the purpose of a
+      # which was slow and quietly undermined reproducibility.
-      # reproducible pipeline to be quietly downloading a tool binary
+      #
-      # fresh on every run). Versioned tag, not :latest — a rebuild of
+      # It lives in the base-images project rather than homelab because that
-      # the tools image doesn't roll out until this pin is bumped too.
+      # project is public: this pod pulls the image before any credential is
-      image: harbor.192.168.1.7.nip.io/homelab/build-tools:1
+      # available to it.
+      #
+      # Versioned tag, never :latest — rebuilding the tools image must not
+      # roll out until this pin is bumped deliberately.
+      image: harbor.35.238.248.203.nip.io/base-images/build-tools:1
       command: ["cat"]
       tty: true
       env:
@@ -77,6 +92,13 @@ spec:
   volumes:
     - name: docker-graph-storage
       emptyDir: {}
+    - name: registry-ca
+      configMap:
+        # Published by devops-infra-argo-config-gcp (extra-manifests). The
+        # CA's public certificate only — its private key never leaves
+        # Terraform state and cert-manager, which is why this is a ConfigMap
+        # rather than a Secret.
+        name: registry-ca
     - name: docker-config
       secret:
         secretName: harbor-robot-dockerconfig

src/com/homelab/stages/buildDocker.groovy

@@ -20,15 +20,19 @@ import com.homelab.utilities.constructTemplate
 def run(Map config) {
     def tag = "${env.BUILD_NUMBER}-${env.GIT_COMMIT?.take(7) ?: 'dev'}"
     env.TAG = tag
-    // Was harbor-core.harbor.svc.cluster.local (cluster-internal Service
+    // The registry ingress hostname, not harbor-core.harbor.svc.cluster.local.
-    // DNS) — worked for this push, since it runs inside a pod with
+    // Cluster DNS would work for this push, which runs inside a pod, but the
-    // pod-network DNS (CoreDNS). But the actual Deployment's image PULL
+    // Deployment's image PULL is performed by containerd on the node, using
-    // happens via containerd running on the node itself, using the
+    // the node's own resolver, which has no route to *.svc.cluster.local at
-    // node's host-level DNS resolver, which has no route to
+    // all — that namespace exists only in CoreDNS. This is the one place the
-    // *.svc.cluster.local at all. Using the Contour ingress hostname
+    // project's "always use cluster DNS between services" rule cannot apply:
-    // instead makes the same reference resolvable from both contexts
+    // the puller is not a service.
-    // (nip.io resolves via normal public DNS, reachable from the host).
+    //
-    def image = "harbor.192.168.1.7.nip.io/${config.harbor_project}/${config.repo_name}:${tag}"
+    // It also has to be spelled identically everywhere, because Docker
+    // matches stored credentials and TLS trust by exact hostname: here, the
+    // dockerconfigjson auths key, Harbor's externalURL, and the node pool's
+    // CA trust config.
+    def image = "harbor.35.238.248.203.nip.io/${config.harbor_project}/${config.repo_name}:${tag}"
     try {
         stage(stageName('Build & push image')) {
             container('docker-cli') {

src/com/homelab/stages/updateHelmTag.groovy

@@ -16,7 +16,9 @@ package com.homelab.stages
 //   service_name      second path segment; same as repo_name for a
 //                     single-service repo, different for a monorepo with
 //                     several services sharing one repo
-//   helm_repo_url     e.g. http://gitea.192.168.1.7.nip.io/mukul/devops-helm-charts.git
+//   helm_repo_url     e.g. http://gitea-http.gitea.svc.cluster.local:3000/gitadmin/devops-helm-charts-gcp.git
+//                     (cluster DNS — this clone runs in a build pod, so it
+//                     never goes out through the ingress and back)
 //   image_tag_yq_path  yq path to the tag field, e.g. .deployment.image.tag
 //   gitea_cred        Jenkins credential ID for a Gitea push-capable token (default: gitea-ci-credentials)
 //

vars/homelabPipeline.groovy

@@ -24,7 +24,10 @@ def call(Map config) {
     config.service_name      = config.service_name      ?: config.repo_name
     config.argo_app_name     = config.argo_app_name      ?: config.repo_name
     config.harbor_project    = config.harbor_project     ?: 'homelab'
-    config.helm_repo_url     = config.helm_repo_url      ?: 'http://gitea.192.168.1.7.nip.io/mukul/devops-helm-charts.git'
+    // Cluster DNS, not the ingress hostname: this clone happens from a build
+    // pod, so it is pod-to-pod traffic and has no business leaving the
+    // cluster and coming back in through Contour.
+    config.helm_repo_url     = config.helm_repo_url      ?: 'http://gitea-http.gitea.svc.cluster.local:3000/gitadmin/devops-helm-charts-gcp.git'
     config.image_tag_yq_path = config.image_tag_yq_path  ?: '.deployment.image.tag'
 
     // Every stage file (both the ones adapted for this homelab and the