From 6d743cbe48e3d6589af768678488f76e01686839 Mon Sep 17 00:00:00 2001 From: Mukul Sharma Date: Thu, 17 Sep 2026 09:31:32 +0530 Subject: [PATCH] Make the registry hostname configurable and drop the CA mount MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit buildDocker.groovy hardcoded harbor.35.238.248.203.nip.io as the push target, so the registry could not move without editing this shared library and every consumer moving in the same commit. It now reads config.harbor_registry, whose default lives in homelabPipeline.groovy beside harbor_project and every other key. The stage errors rather than defaulting when the value is missing. Carrying a second copy of the literal would leave two defaults free to disagree, and an unset value would otherwise build an image named "null//" — which docker accepts as a hostname and then fails to resolve, pointing nowhere near the cause. The dind pod no longer mounts the registry CA. That mount existed because the registry was a nip.io name, which no public CA will issue for, so cert-manager signed Harbor from a private CA; the node pool was told to trust it for pulls, but a push comes from dockerd inside the build pod, which has its own trust store. harbor.infra.deployshed.com carries a Let's Encrypt certificate that both already trust, so the mount, its volume and the whole arrangement go away rather than being repointed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N --- README.md | 39 ++++++++++-------- resources/com/homelab/go-Dockerfile | 4 +- resources/com/homelab/java-Dockerfile | 4 +- resources/com/homelab/node-Dockerfile | 4 +- resources/com/homelab/php-Dockerfile | 2 +- resources/com/homelab/python-Dockerfile | 2 +- resources/org/homelab/dind-pod.yaml | 48 ++++++++++------------- src/com/homelab/stages/buildDocker.groovy | 22 +++++++++-- vars/homelabPipeline.groovy | 11 ++++++ 9 files changed, 81 insertions(+), 55 deletions(-) diff --git a/README.md b/README.md index 9dd8316..57bbbee 100644 --- a/README.md +++ b/README.md @@ -13,11 +13,12 @@ property of the Jenkins it runs on. **What differs from the homelab copy**, all of it a consequence of GKE being a real cloud rather than one VM: -- **The registry hostname** is `harbor.35.238.248.203.nip.io`, in the push - target and in all five fallback Dockerfiles. -- **Harbor speaks TLS.** The homelab's dind passes `--insecure-registry`; - here the pod mounts the private CA into dockerd's trust store instead. - Node trust covers pulls only — a push is a separate client. +- **The registry hostname** is `harbor.infra.deployshed.com`, in the push + target (`harbor_registry`, see the table below) and in all five fallback + Dockerfiles. +- **Harbor speaks TLS, with a public certificate.** The homelab's dind + passes `--insecure-registry`; here nothing is needed at all, because + Let's Encrypt issues for the real domain and dockerd already trusts it. - **`helm_repo_url` uses cluster DNS**, since the clone happens inside a build pod. The homelab points it at an ingress hostname. - **`build-tools` is not in this repo.** It lives in @@ -46,6 +47,7 @@ in after checkout; repo-committed values win over the Jenkinsfile call). |---|---|---| | `service_name` | `repo_name` | Second path segment under `devops-helm-charts/values/` | | `argo_app_name` | `repo_name` | Must match the ArgoCD Application's `metadata.name` | +| `harbor_registry` | `harbor.infra.deployshed.com` | The registry hostname images are pushed to and pulled from. Must be spelled identically here, in the `dockerconfigjson` `auths` key, and in Harbor's `externalURL` — docker matches both stored credentials and TLS trust by exact hostname, so a mismatch fails as `unauthorized` rather than as a name problem | | `harbor_project` | `apps-registry` | Must already exist in Harbor, and be public unless you also wire an `imagePullSecret` — the app values assume anonymous pull. A push to a missing project fails as `unauthorized: project not found` | | `helm_repo_url` | `devops-helm-charts-gcp`, over cluster DNS | `http://gitea-http.gitea.svc.cluster.local:3000/gitadmin/…` — pod-to-pod, so it never leaves the cluster and comes back through the ingress | | `image_tag_yq_path` | `.deployment.image.tag` | **Override this if the app's chart isn't `1.0.0`** — e.g. `sts-2.0.0` uses `.podtemplate.image.tag` instead. Getting this wrong doesn't fail loudly: `yq -i` creates the path if missing rather than erroring, silently leaving the real field un-bumped. | @@ -89,7 +91,7 @@ a `podTemplate` (`resources/org/homelab/dind-pod.yaml`) via ## Build-tools image The `docker-cli` container runs -`harbor.35.238.248.203.nip.io/base-images/build-tools:1`, which bakes in +`harbor.infra.deployshed.com/base-images/build-tools:1`, which bakes in git, yq, bash, python3 with pip and venv, and curl, so nothing is installed on demand on every build. @@ -103,14 +105,19 @@ until that pin is bumped. Bump the tag rather than overwriting one. ## Registry trust -`dind-pod.yaml` mounts the `registry-ca` ConfigMap (published by -`devops-infra-argo-config-gcp`) into the dind container at -`/etc/docker/certs.d/harbor.35.238.248.203.nip.io/ca.crt`. +Nothing to configure. Harbor is reached at `harbor.infra.deployshed.com`, +which carries a publicly trusted Let's Encrypt certificate, so both +containerd on the node (pulls) and dockerd in the build pod (pushes) accept +it out of the box. -Without it, pushes fail TLS verification while pulls of the same image -succeed, which reads like a broken registry. The reason is that the two are -different clients: pulls are performed by containerd on the node, which was -told to trust this CA when the node pool was created, whereas the push comes -from dockerd inside the build pod, which has its own trust store. The -directory name must be the registry hostname exactly — dockerd looks the -path up by host and silently ignores a mismatch. +This used to be a real piece of setup, and the history is worth keeping +because reintroducing a nip.io registry name would bring it all back. A +nip.io address cannot have a public certificate — it is not on the public +suffix list, and every `*.nip.io` certificate on the internet shares one +rate limit — so cert-manager signed Harbor from a private CA instead. The +node pool was told to trust that CA at creation, which covered pulls; the +push came from dockerd inside the build pod, a separate client with its own +trust store, so `dind-pod.yaml` had to mount the CA at +`/etc/docker/certs.d//ca.crt` as well. The failure when that +mount was missing was thoroughly confusing: pushes failed TLS verification +while pulls of the very same image succeeded. diff --git a/resources/com/homelab/go-Dockerfile b/resources/com/homelab/go-Dockerfile index 40a72ca..00ab099 100644 --- a/resources/com/homelab/go-Dockerfile +++ b/resources/com/homelab/go-Dockerfile @@ -13,14 +13,14 @@ # a dockerBuildVersion whose tag isn't in devops-base-images/images.txt # yet needs that added and re-mirrored first, unlike pulling straight # from Docker Hub where any tag "just worked". -FROM harbor.35.238.248.203.nip.io/base-images/golang:${version}-alpine AS build +FROM harbor.infra.deployshed.com/base-images/golang:${version}-alpine AS build WORKDIR /src COPY go.mod go.sum* ./ RUN go mod download 2>/dev/null || true COPY . . RUN CGO_ENABLED=0 go build -o /app . -FROM harbor.35.238.248.203.nip.io/base-images/alpine:3.20 +FROM harbor.infra.deployshed.com/base-images/alpine:3.20 COPY --from=build /app /app EXPOSE 8080 ENTRYPOINT ["/app"] diff --git a/resources/com/homelab/java-Dockerfile b/resources/com/homelab/java-Dockerfile index b00b84f..c6b6c7f 100644 --- a/resources/com/homelab/java-Dockerfile +++ b/resources/com/homelab/java-Dockerfile @@ -8,14 +8,14 @@ # to Maven Central for plugins/dependencies during the build regardless # of base image — this only removes the Docker Hub dependency for the # base image layer, not package-registry traffic during the build. -FROM harbor.35.238.248.203.nip.io/base-images/maven:3-eclipse-temurin-${version}-alpine AS build +FROM harbor.infra.deployshed.com/base-images/maven:3-eclipse-temurin-${version}-alpine AS build WORKDIR /src COPY pom.xml . RUN mvn -B dependency:go-offline COPY . . RUN mvn -B package -DskipTests -FROM harbor.35.238.248.203.nip.io/base-images/eclipse-temurin:${version}-jre-alpine +FROM harbor.infra.deployshed.com/base-images/eclipse-temurin:${version}-jre-alpine WORKDIR /app COPY --from=build /src/target/*.jar app.jar EXPOSE 8080 diff --git a/resources/com/homelab/node-Dockerfile b/resources/com/homelab/node-Dockerfile index 87cbdae..3f80637 100644 --- a/resources/com/homelab/node-Dockerfile +++ b/resources/com/homelab/node-Dockerfile @@ -5,14 +5,14 @@ # Assumes a standard `npm run build` + `npm start` repo. Switched from # node:*-slim (Debian) to node:*-alpine for both stages — smaller, still # keeps a shell for kubectl exec debugging (not distroless). -FROM harbor.35.238.248.203.nip.io/base-images/node:${version}-alpine AS build +FROM harbor.infra.deployshed.com/base-images/node:${version}-alpine AS build WORKDIR /app COPY package*.json ./ RUN npm ci COPY . . RUN npm run build --if-present -FROM harbor.35.238.248.203.nip.io/base-images/node:${version}-alpine +FROM harbor.infra.deployshed.com/base-images/node:${version}-alpine WORKDIR /app COPY --from=build /app . ENV NODE_ENV=production diff --git a/resources/com/homelab/php-Dockerfile b/resources/com/homelab/php-Dockerfile index ea02e99..cafded4 100644 --- a/resources/com/homelab/php-Dockerfile +++ b/resources/com/homelab/php-Dockerfile @@ -36,7 +36,7 @@ # very comment did so), new edits to this header should avoid typing # the character at all — write "dollar sign" in words instead of using # the glyph. -FROM harbor.35.238.248.203.nip.io/base-images/php:${version}-cli-alpine +FROM harbor.infra.deployshed.com/base-images/php:${version}-cli-alpine WORKDIR /var/www/html RUN apk add --no-cache --virtual .build-deps \$PHPIZE_DEPS \ && docker-php-ext-install pdo pdo_mysql \ diff --git a/resources/com/homelab/python-Dockerfile b/resources/com/homelab/python-Dockerfile index 0d6c362..99e918a 100644 --- a/resources/com/homelab/python-Dockerfile +++ b/resources/com/homelab/python-Dockerfile @@ -9,7 +9,7 @@ # extensions that only ship glibc wheels may need musl-dev/gcc added # here to build from source on Alpine — fine for this repo's pure-Python # deps, worth knowing if a future repo's requirements.txt needs more. -FROM harbor.35.238.248.203.nip.io/base-images/python:${version}-alpine +FROM harbor.infra.deployshed.com/base-images/python:${version}-alpine WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt diff --git a/resources/org/homelab/dind-pod.yaml b/resources/org/homelab/dind-pod.yaml index 867e291..a7c3058 100644 --- a/resources/org/homelab/dind-pod.yaml +++ b/resources/org/homelab/dind-pod.yaml @@ -15,34 +15,33 @@ spec: image: docker:27-dind securityContext: privileged: true - # No --insecure-registry, unlike the homelab: Harbor here serves a real - # certificate, issued by cert-manager from the private CA that the node - # pool was told to trust when it was created. + # No --insecure-registry and no mounted CA: Harbor is reached at + # harbor.infra.deployshed.com, which carries a publicly trusted Let's + # Encrypt certificate, so dockerd's own trust store already accepts it + # with nothing configured. # - # That node trust covers image PULLS, which containerd performs on the - # node. This push is a different client — dockerd, inside this pod, - # with its own trust store and no knowledge of what the node trusts — - # so it needs the CA mounted itself. dockerd looks it up at - # /etc/docker/certs.d//ca.crt, and the directory name - # must be the registry hostname exactly; anything else is silently - # ignored, and the push then fails TLS verification while a pull of the - # very same image works. + # This previously mounted the private registry CA here, because the + # registry was a nip.io name that no public CA can issue for (nip.io is + # not on the public suffix list, and every *.nip.io certificate shares + # one rate limit), so cert-manager signed it from a private CA instead. + # A pull was fine — the node pool was told to trust that CA and + # containerd performs pulls — but a push is a different client with its + # own trust store, which is why dockerd needed the CA at + # /etc/docker/certs.d//ca.crt. Moving to a real domain + # removes the whole arrangement rather than repointing it. # - # The registry hostname (rather than harbor-core.harbor.svc.cluster.local) - # carries over unchanged from the homelab, for a reason that still - # holds: cluster DNS resolves from this pod but not from the node's - # containerd doing the real Deployment pull, and Docker matches both - # stored credentials and trust by exact hostname — so push and pull - # have to name the registry identically. + # The registry is still named by its ingress hostname rather than + # harbor-core.harbor.svc.cluster.local, for a reason that still holds: + # cluster DNS resolves from this pod but not from the node's containerd + # doing the real Deployment pull, and Docker matches both stored + # credentials and trust by exact hostname — so push and pull have to + # name the registry identically. env: - name: DOCKER_TLS_CERTDIR value: "" volumeMounts: - name: docker-graph-storage mountPath: /var/lib/docker - - name: registry-ca - mountPath: /etc/docker/certs.d/harbor.35.238.248.203.nip.io - readOnly: true - name: docker-cli # Custom image, built and pushed by hand from devops-base-images-gcp # (build-tools.Dockerfile there) — bakes in git/yq/bash/python3 with @@ -55,7 +54,7 @@ spec: # # Versioned tag, never :latest — rebuilding the tools image must not # roll out until this pin is bumped deliberately. - image: harbor.35.238.248.203.nip.io/base-images/build-tools:1 + image: harbor.infra.deployshed.com/base-images/build-tools:1 command: ["cat"] tty: true env: @@ -92,13 +91,6 @@ spec: volumes: - name: docker-graph-storage emptyDir: {} - - name: registry-ca - configMap: - # Published by devops-infra-argo-config-gcp (extra-manifests). The - # CA's public certificate only — its private key never leaves - # Terraform state and cert-manager, which is why this is a ConfigMap - # rather than a Secret. - name: registry-ca - name: docker-config secret: secretName: harbor-robot-dockerconfig diff --git a/src/com/homelab/stages/buildDocker.groovy b/src/com/homelab/stages/buildDocker.groovy index 0c8b570..00b31b1 100644 --- a/src/com/homelab/stages/buildDocker.groovy +++ b/src/com/homelab/stages/buildDocker.groovy @@ -30,9 +30,25 @@ def run(Map config) { // // It also has to be spelled identically everywhere, because Docker // matches stored credentials and TLS trust by exact hostname: here, the - // dockerconfigjson auths key, Harbor's externalURL, and the node pool's - // CA trust config. - def image = "harbor.35.238.248.203.nip.io/${config.harbor_project}/${config.repo_name}:${tag}" + // dockerconfigjson auths key and Harbor's externalURL. + // + // Configurable rather than hardcoded, which is what this used to be. A + // literal here meant the registry could not move without editing the + // shared library itself, and every consumer moving in the same commit — + // the failure being a push to a hostname nothing serves, several minutes + // into a build. + // + // The DEFAULT lives in homelabPipeline.groovy beside harbor_project and + // every other key, not here. Repeating the literal in both places would + // leave two defaults free to disagree, and the one that lost would only + // show up as a push to the wrong registry. Failing loudly beats + // defaulting quietly: an unset value would otherwise build an image + // named "null//", which docker accepts as a hostname and + // then fails to resolve, pointing nowhere near the cause. + if (!config.harbor_registry) { + error("buildDocker: config.harbor_registry is not set. homelabPipeline normally defaults it; if this stage is being called directly, pass harbor_registry (e.g. 'harbor.infra.deployshed.com').") + } + def image = "${config.harbor_registry}/${config.harbor_project}/${config.repo_name}:${tag}" try { stage(stageName('Build & push image')) { container('docker-cli') { diff --git a/vars/homelabPipeline.groovy b/vars/homelabPipeline.groovy index 84f553b..7d83ee3 100644 --- a/vars/homelabPipeline.groovy +++ b/vars/homelabPipeline.groovy @@ -29,6 +29,17 @@ def call(Map config) { // "unauthorized: project not found", which reads like a // credentials problem rather than a missing project. config.harbor_project = config.harbor_project ?: 'apps-registry' + // The registry hostname images are pushed to and pulled from. This is + // the ONLY default for it — buildDocker.groovy deliberately errors + // rather than carrying a second copy, since two defaults for one value + // are free to disagree and the loser only shows up as a push to the + // wrong registry. + // + // It must be spelled identically here, in the dockerconfigjson auths + // key, and in Harbor's own externalURL: docker matches both stored + // credentials and TLS trust by exact hostname, so a mismatch fails as + // "unauthorized" rather than as anything resembling a name problem. + config.harbor_registry = config.harbor_registry ?: 'harbor.infra.deployshed.com' // Cluster DNS, not the ingress hostname: this clone happens from a build // pod, so it is pod-to-pod traffic and has no business leaving the // cluster and coming back in through Contour.