diff --git a/README.md b/README.md index 0b035be..07bc0bb 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,32 @@ -# devops-lib +# devops-lib (GKE) -Jenkins Shared Library for this homelab's CI/CD pipeline. Adapted from a -much larger, company-wide library — everything not needed for a -single-node homelab (GKE/EKS, JFrog, S3, Sonar, org-specific BU/team -validation, and ~65 other files covering languages/deploy-targets this -setup never uses) has been removed rather than carried along unused; see -git history if any of that is ever worth reviving. +Jenkins Shared Library for the CI/CD pipeline on the GKE cluster. The GCP +counterpart of the homelab repo of the same name, and a copy rather than a +shared repo because several values here are cluster-specific in ways that +would break the other cluster if crossed over. + +Register it in Jenkins under the name `devops-lib`, exactly as on the +homelab. Consuming repos then need no change at all — the same two-line +Jenkinsfile works on either cluster, and which library it resolves to is a +property of the Jenkins it runs on. + +**What differs from the homelab copy**, all of it a consequence of GKE +being a real cloud rather than one VM: + +- **The registry hostname** is `harbor.35.238.248.203.nip.io`, in the push + target and in all five fallback Dockerfiles. +- **Harbor speaks TLS.** The homelab's dind passes `--insecure-registry`; + here the pod mounts the private CA into dockerd's trust store instead. + Node trust covers pulls only — a push is a separate client. +- **`helm_repo_url` uses cluster DNS**, since the clone happens inside a + build pod. The homelab points it at an ingress hostname. +- **`build-tools` is not in this repo.** It lives in + `devops-base-images-gcp` alongside the mirrored base images, and is + referenced here only by tag. + +Everything else — the stage flow, the fallback templates, the hooks +contract — is unchanged. The library was already adapted from a much +larger, company-wide one; see git history for what was removed. ## Using it in a service repo @@ -26,7 +47,7 @@ in after checkout; repo-committed values win over the Jenkinsfile call). | `service_name` | `repo_name` | Second path segment under `devops-helm-charts/values/` | | `argo_app_name` | `repo_name` | Must match the ArgoCD Application's `metadata.name` | | `harbor_project` | `homelab` | Must be an existing, public Harbor project | -| `helm_repo_url` | `devops-helm-charts` on this Gitea | — | +| `helm_repo_url` | `devops-helm-charts-gcp`, over cluster DNS | `http://gitea-http.gitea.svc.cluster.local:3000/gitadmin/…` — pod-to-pod, so it never leaves the cluster and comes back through the ingress | | `image_tag_yq_path` | `.deployment.image.tag` | **Override this if the app's chart isn't `1.0.0`** — e.g. `sts-2.0.0` uses `.podtemplate.image.tag` instead. Getting this wrong doesn't fail loudly: `yq -i` creates the path if missing rather than erroring, silently leaving the real field un-bumped. | | `dockerBuildVersion` | none | Only read when the repo has **no Dockerfile of its own** — picks a fallback template (see below). No default; either ship a Dockerfile or set this. | @@ -49,7 +70,9 @@ a `podTemplate` (`resources/org/homelab/dind-pod.yaml`) via based on `dockerBuildVersion` (e.g. `go-1.22`, `node-20`, `python-3.12`, `java-21`, `php-8.3`). All fallback templates pull base images from Harbor's `base-images` project (mirrored via the separate - `devops-base-images` repo), not Docker Hub directly. + `devops-base-images-gcp` repo), not Docker Hub directly. Only versions + actually mirrored there resolve — an unmirrored tag fails the build + rather than silently falling back to Docker Hub. - **`updateHelmTag`** — clones `devops-helm-charts`, bumps the image tag via `yq` at `image_tag_yq_path`, commits, pushes to `main`. - **`syncArgoApp`** — calls the ArgoCD REST API to sync `argo_app_name`. @@ -65,8 +88,29 @@ a `podTemplate` (`resources/org/homelab/dind-pod.yaml`) via ## Build-tools image -`resources/org/homelab/build-tools.Dockerfile` bakes git/yq/bash/ -python3+pip/venv/curl into the `docker-cli` container's image, so -nothing gets installed on demand on every single build. Built and pushed -manually (not through any Jenkins job) — see that file's own header -comment. +The `docker-cli` container runs +`harbor.35.238.248.203.nip.io/base-images/build-tools:1`, which bakes in +git, yq, bash, python3 with pip and venv, and curl, so nothing is installed +on demand on every build. + +**It is not built here.** The Dockerfile lives in `devops-base-images-gcp`, +next to the mirrored base images, because it is the same kind of artefact: +built by hand, occasionally, and pushed to Harbor. A Jenkins job could not +build it anyway — it is the image Jenkins builds *in*. + +`dind-pod.yaml` pins the tag, so rebuilding the image rolls nothing out +until that pin is bumped. Bump the tag rather than overwriting one. + +## Registry trust + +`dind-pod.yaml` mounts the `registry-ca` ConfigMap (published by +`devops-infra-argo-config-gcp`) into the dind container at +`/etc/docker/certs.d/harbor.35.238.248.203.nip.io/ca.crt`. + +Without it, pushes fail TLS verification while pulls of the same image +succeed, which reads like a broken registry. The reason is that the two are +different clients: pulls are performed by containerd on the node, which was +told to trust this CA when the node pool was created, whereas the push comes +from dockerd inside the build pod, which has its own trust store. The +directory name must be the registry hostname exactly — dockerd looks the +path up by host and silently ignores a mismatch. diff --git a/resources/com/homelab/go-Dockerfile b/resources/com/homelab/go-Dockerfile index 91ee2d8..40a72ca 100644 --- a/resources/com/homelab/go-Dockerfile +++ b/resources/com/homelab/go-Dockerfile @@ -13,14 +13,14 @@ # a dockerBuildVersion whose tag isn't in devops-base-images/images.txt # yet needs that added and re-mirrored first, unlike pulling straight # from Docker Hub where any tag "just worked". -FROM harbor.192.168.1.7.nip.io/base-images/golang:${version}-alpine AS build +FROM harbor.35.238.248.203.nip.io/base-images/golang:${version}-alpine AS build WORKDIR /src COPY go.mod go.sum* ./ RUN go mod download 2>/dev/null || true COPY . . RUN CGO_ENABLED=0 go build -o /app . -FROM harbor.192.168.1.7.nip.io/base-images/alpine:3.20 +FROM harbor.35.238.248.203.nip.io/base-images/alpine:3.20 COPY --from=build /app /app EXPOSE 8080 ENTRYPOINT ["/app"] diff --git a/resources/com/homelab/java-Dockerfile b/resources/com/homelab/java-Dockerfile index 4641e11..b00b84f 100644 --- a/resources/com/homelab/java-Dockerfile +++ b/resources/com/homelab/java-Dockerfile @@ -8,14 +8,14 @@ # to Maven Central for plugins/dependencies during the build regardless # of base image — this only removes the Docker Hub dependency for the # base image layer, not package-registry traffic during the build. -FROM harbor.192.168.1.7.nip.io/base-images/maven:3-eclipse-temurin-${version}-alpine AS build +FROM harbor.35.238.248.203.nip.io/base-images/maven:3-eclipse-temurin-${version}-alpine AS build WORKDIR /src COPY pom.xml . RUN mvn -B dependency:go-offline COPY . . RUN mvn -B package -DskipTests -FROM harbor.192.168.1.7.nip.io/base-images/eclipse-temurin:${version}-jre-alpine +FROM harbor.35.238.248.203.nip.io/base-images/eclipse-temurin:${version}-jre-alpine WORKDIR /app COPY --from=build /src/target/*.jar app.jar EXPOSE 8080 diff --git a/resources/com/homelab/node-Dockerfile b/resources/com/homelab/node-Dockerfile index be2fd4b..87cbdae 100644 --- a/resources/com/homelab/node-Dockerfile +++ b/resources/com/homelab/node-Dockerfile @@ -5,14 +5,14 @@ # Assumes a standard `npm run build` + `npm start` repo. Switched from # node:*-slim (Debian) to node:*-alpine for both stages — smaller, still # keeps a shell for kubectl exec debugging (not distroless). -FROM harbor.192.168.1.7.nip.io/base-images/node:${version}-alpine AS build +FROM harbor.35.238.248.203.nip.io/base-images/node:${version}-alpine AS build WORKDIR /app COPY package*.json ./ RUN npm ci COPY . . RUN npm run build --if-present -FROM harbor.192.168.1.7.nip.io/base-images/node:${version}-alpine +FROM harbor.35.238.248.203.nip.io/base-images/node:${version}-alpine WORKDIR /app COPY --from=build /app . ENV NODE_ENV=production diff --git a/resources/com/homelab/php-Dockerfile b/resources/com/homelab/php-Dockerfile index 12b4221..ea02e99 100644 --- a/resources/com/homelab/php-Dockerfile +++ b/resources/com/homelab/php-Dockerfile @@ -36,7 +36,7 @@ # very comment did so), new edits to this header should avoid typing # the character at all — write "dollar sign" in words instead of using # the glyph. -FROM harbor.192.168.1.7.nip.io/base-images/php:${version}-cli-alpine +FROM harbor.35.238.248.203.nip.io/base-images/php:${version}-cli-alpine WORKDIR /var/www/html RUN apk add --no-cache --virtual .build-deps \$PHPIZE_DEPS \ && docker-php-ext-install pdo pdo_mysql \ diff --git a/resources/com/homelab/python-Dockerfile b/resources/com/homelab/python-Dockerfile index 9c6ad8c..0d6c362 100644 --- a/resources/com/homelab/python-Dockerfile +++ b/resources/com/homelab/python-Dockerfile @@ -9,7 +9,7 @@ # extensions that only ship glibc wheels may need musl-dev/gcc added # here to build from source on Alpine — fine for this repo's pure-Python # deps, worth knowing if a future repo's requirements.txt needs more. -FROM harbor.192.168.1.7.nip.io/base-images/python:${version}-alpine +FROM harbor.35.238.248.203.nip.io/base-images/python:${version}-alpine WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt diff --git a/resources/org/homelab/build-tools.Dockerfile b/resources/org/homelab/build-tools.Dockerfile deleted file mode 100644 index bbcb5bb..0000000 --- a/resources/org/homelab/build-tools.Dockerfile +++ /dev/null @@ -1,13 +0,0 @@ -# Custom docker-cli image for dind-pod.yaml's docker-cli container — -# bakes in everything the pipeline stages need at runtime (git, yq, -# bash, python3 + pip/venv for runHooks' python hooks, curl) so nothing -# gets apk-installed or curl-downloaded on every single build. Rebuild -# and push this (see the one-off build commands in the commit that -# added this file) whenever this list changes; dind-pod.yaml pins the -# resulting image tag explicitly, so a rebuild doesn't silently roll -# out until that pin is also bumped. -FROM docker:27-cli - -RUN apk add --no-cache git bash python3 py3-pip py3-virtualenv curl \ - && curl -sL -o /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 \ - && chmod +x /usr/local/bin/yq diff --git a/resources/org/homelab/dind-pod.yaml b/resources/org/homelab/dind-pod.yaml index fab5a60..867e291 100644 --- a/resources/org/homelab/dind-pod.yaml +++ b/resources/org/homelab/dind-pod.yaml @@ -15,32 +15,47 @@ spec: image: docker:27-dind securityContext: privileged: true - # Harbor's ingress serves plain HTTP too (TLS disabled cluster-wide - # by design — see claude.md's "everything is plain HTTP" note). - # Docker defaults to attempting HTTPS against any bare registry - # hostname regardless of network path, so this is needed - # regardless of which hostname is used — was previously - # harbor-core.harbor.svc.cluster.local (cluster-internal Service - # DNS, works from this pod but not from the node's own containerd - # when pulling for a real Deployment); switched to the Contour - # ingress hostname so push and pull can share one reference. - args: - - "--insecure-registry=harbor.192.168.1.7.nip.io" + # No --insecure-registry, unlike the homelab: Harbor here serves a real + # certificate, issued by cert-manager from the private CA that the node + # pool was told to trust when it was created. + # + # That node trust covers image PULLS, which containerd performs on the + # node. This push is a different client — dockerd, inside this pod, + # with its own trust store and no knowledge of what the node trusts — + # so it needs the CA mounted itself. dockerd looks it up at + # /etc/docker/certs.d//ca.crt, and the directory name + # must be the registry hostname exactly; anything else is silently + # ignored, and the push then fails TLS verification while a pull of the + # very same image works. + # + # The registry hostname (rather than harbor-core.harbor.svc.cluster.local) + # carries over unchanged from the homelab, for a reason that still + # holds: cluster DNS resolves from this pod but not from the node's + # containerd doing the real Deployment pull, and Docker matches both + # stored credentials and trust by exact hostname — so push and pull + # have to name the registry identically. env: - name: DOCKER_TLS_CERTDIR value: "" volumeMounts: - name: docker-graph-storage mountPath: /var/lib/docker + - name: registry-ca + mountPath: /etc/docker/certs.d/harbor.35.238.248.203.nip.io + readOnly: true - name: docker-cli - # Custom image (see build-tools.Dockerfile in this same directory) - # — bakes in git/yq/bash/python3+pip/venv/curl so nothing needs - # apk-installing or curl-downloading on every single build (was - # slow and, per its own point, defeats the purpose of a - # reproducible pipeline to be quietly downloading a tool binary - # fresh on every run). Versioned tag, not :latest — a rebuild of - # the tools image doesn't roll out until this pin is bumped too. - image: harbor.192.168.1.7.nip.io/homelab/build-tools:1 + # Custom image, built and pushed by hand from devops-base-images-gcp + # (build-tools.Dockerfile there) — bakes in git/yq/bash/python3 with + # pip and venv/curl so nothing needs installing on every single build, + # which was slow and quietly undermined reproducibility. + # + # It lives in the base-images project rather than homelab because that + # project is public: this pod pulls the image before any credential is + # available to it. + # + # Versioned tag, never :latest — rebuilding the tools image must not + # roll out until this pin is bumped deliberately. + image: harbor.35.238.248.203.nip.io/base-images/build-tools:1 command: ["cat"] tty: true env: @@ -77,6 +92,13 @@ spec: volumes: - name: docker-graph-storage emptyDir: {} + - name: registry-ca + configMap: + # Published by devops-infra-argo-config-gcp (extra-manifests). The + # CA's public certificate only — its private key never leaves + # Terraform state and cert-manager, which is why this is a ConfigMap + # rather than a Secret. + name: registry-ca - name: docker-config secret: secretName: harbor-robot-dockerconfig diff --git a/src/com/homelab/stages/buildDocker.groovy b/src/com/homelab/stages/buildDocker.groovy index 57cf964..0c8b570 100644 --- a/src/com/homelab/stages/buildDocker.groovy +++ b/src/com/homelab/stages/buildDocker.groovy @@ -20,15 +20,19 @@ import com.homelab.utilities.constructTemplate def run(Map config) { def tag = "${env.BUILD_NUMBER}-${env.GIT_COMMIT?.take(7) ?: 'dev'}" env.TAG = tag - // Was harbor-core.harbor.svc.cluster.local (cluster-internal Service - // DNS) — worked for this push, since it runs inside a pod with - // pod-network DNS (CoreDNS). But the actual Deployment's image PULL - // happens via containerd running on the node itself, using the - // node's host-level DNS resolver, which has no route to - // *.svc.cluster.local at all. Using the Contour ingress hostname - // instead makes the same reference resolvable from both contexts - // (nip.io resolves via normal public DNS, reachable from the host). - def image = "harbor.192.168.1.7.nip.io/${config.harbor_project}/${config.repo_name}:${tag}" + // The registry ingress hostname, not harbor-core.harbor.svc.cluster.local. + // Cluster DNS would work for this push, which runs inside a pod, but the + // Deployment's image PULL is performed by containerd on the node, using + // the node's own resolver, which has no route to *.svc.cluster.local at + // all — that namespace exists only in CoreDNS. This is the one place the + // project's "always use cluster DNS between services" rule cannot apply: + // the puller is not a service. + // + // It also has to be spelled identically everywhere, because Docker + // matches stored credentials and TLS trust by exact hostname: here, the + // dockerconfigjson auths key, Harbor's externalURL, and the node pool's + // CA trust config. + def image = "harbor.35.238.248.203.nip.io/${config.harbor_project}/${config.repo_name}:${tag}" try { stage(stageName('Build & push image')) { container('docker-cli') { diff --git a/src/com/homelab/stages/updateHelmTag.groovy b/src/com/homelab/stages/updateHelmTag.groovy index 75ace30..262362d 100644 --- a/src/com/homelab/stages/updateHelmTag.groovy +++ b/src/com/homelab/stages/updateHelmTag.groovy @@ -16,7 +16,9 @@ package com.homelab.stages // service_name second path segment; same as repo_name for a // single-service repo, different for a monorepo with // several services sharing one repo -// helm_repo_url e.g. http://gitea.192.168.1.7.nip.io/mukul/devops-helm-charts.git +// helm_repo_url e.g. http://gitea-http.gitea.svc.cluster.local:3000/gitadmin/devops-helm-charts-gcp.git +// (cluster DNS — this clone runs in a build pod, so it +// never goes out through the ingress and back) // image_tag_yq_path yq path to the tag field, e.g. .deployment.image.tag // gitea_cred Jenkins credential ID for a Gitea push-capable token (default: gitea-ci-credentials) // diff --git a/vars/homelabPipeline.groovy b/vars/homelabPipeline.groovy index b0a5708..d72c8d3 100644 --- a/vars/homelabPipeline.groovy +++ b/vars/homelabPipeline.groovy @@ -24,7 +24,10 @@ def call(Map config) { config.service_name = config.service_name ?: config.repo_name config.argo_app_name = config.argo_app_name ?: config.repo_name config.harbor_project = config.harbor_project ?: 'homelab' - config.helm_repo_url = config.helm_repo_url ?: 'http://gitea.192.168.1.7.nip.io/mukul/devops-helm-charts.git' + // Cluster DNS, not the ingress hostname: this clone happens from a build + // pod, so it is pod-to-pod traffic and has no business leaving the + // cluster and coming back in through Contour. + config.helm_repo_url = config.helm_repo_url ?: 'http://gitea-http.gitea.svc.cluster.local:3000/gitadmin/devops-helm-charts-gcp.git' config.image_tag_yq_path = config.image_tag_yq_path ?: '.deployment.image.tag' // Every stage file (both the ones adapted for this homelab and the