gitadmin/devops-lib-gcp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

GKE: point the pipeline at this cluster's registry, over TLS

Browse Source 
The GCP counterpart of devops-lib. Registered in Jenkins under the same
name, so consuming repos need no change: the two-line Jenkinsfile is
identical on both clusters, and which library it resolves to is a property
of the Jenkins running it.

Substantive changes, all consequences of GKE being a real cloud:

- Harbor speaks TLS here, so dind no longer passes --insecure-registry.
  It mounts the private CA at
  /etc/docker/certs.d/harbor.35.238.248.203.nip.io/ca.crt instead, from the
  registry-ca ConfigMap. This is not redundant with the node pool's trust:
  that covers pulls, performed by containerd on the node, while the push
  comes from dockerd in the build pod with its own trust store. Without it,
  pushes fail TLS verification while pulls of the same image succeed —
  which reads like a broken registry rather than a missing trust anchor.

- The registry hostname changes in the push target and all five fallback
  Dockerfiles. It still must be spelled identically everywhere, because
  Docker matches credentials and trust by exact hostname.

- helm_repo_url moves to cluster DNS. That clone runs in a build pod, so
  sending it out through the ingress and back would make the pipeline
  depend on Contour for pod-to-pod traffic. syncArgoApp already addressed
  ArgoCD this way and needed no change.

- build-tools.Dockerfile is removed: devops-base-images-gcp owns it now,
  next to the mirrored base images, and the pod references the result by
  tag at base-images/build-tools:1. The base-images project is public, so
  the pod can pull it before it has any credentials.

Verified: no homelab addresses remain; the pod template parses with the CA
mount, the new image and no insecure-registry flag; and every fallback
template's base image, with the default version buildDocker would pick, is
present in the mirror manifest — an unmirrored tag now fails the build
rather than silently falling back to Docker Hub.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
This commit is contained in:
Mukul Sharma
2026-09-13 01:27:05 +05:30
co-authored by Claude Opus 5
parent 43ca78e83e
commit 3e09eecbfe
11 changed files with 127 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+58 -14
View File
@@ -1,11 +1,32 @@
# devops-lib
# devops-lib (GKE)
Jenkins Shared Library for this homelab's CI/CD pipeline. Adapted from a
much larger, company-wide library — everything not needed for a
single-node homelab (GKE/EKS, JFrog, S3, Sonar, org-specific BU/team
validation, and ~65 other files covering languages/deploy-targets this
setup never uses) has been removed rather than carried along unused; see
git history if any of that is ever worth reviving.
Jenkins Shared Library for the CI/CD pipeline on the GKE cluster. The GCP
counterpart of the homelab repo of the same name, and a copy rather than a
shared repo because several values here are cluster-specific in ways that
would break the other cluster if crossed over.
Register it in Jenkins under the name `devops-lib`, exactly as on the
homelab. Consuming repos then need no change at all — the same two-line
Jenkinsfile works on either cluster, and which library it resolves to is a
property of the Jenkins it runs on.
**What differs from the homelab copy**, all of it a consequence of GKE
being a real cloud rather than one VM:
- **The registry hostname** is `harbor.35.238.248.203.nip.io`, in the push
target and in all five fallback Dockerfiles.
- **Harbor speaks TLS.** The homelab's dind passes `--insecure-registry`;
here the pod mounts the private CA into dockerd's trust store instead.
Node trust covers pulls only — a push is a separate client.
- **`helm_repo_url` uses cluster DNS**, since the clone happens inside a
build pod. The homelab points it at an ingress hostname.
- **`build-tools` is not in this repo.** It lives in
`devops-base-images-gcp` alongside the mirrored base images, and is
referenced here only by tag.
Everything else — the stage flow, the fallback templates, the hooks
contract — is unchanged. The library was already adapted from a much
larger, company-wide one; see git history for what was removed.
## Using it in a service repo
@@ -26,7 +47,7 @@ in after checkout; repo-committed values win over the Jenkinsfile call).
| `service_name` | `repo_name` | Second path segment under `devops-helm-charts/values/` |
| `argo_app_name` | `repo_name` | Must match the ArgoCD Application's `metadata.name` |
| `harbor_project` | `homelab` | Must be an existing, public Harbor project |
| `helm_repo_url` | `devops-helm-charts` on this Gitea | — |
| `helm_repo_url` | `devops-helm-charts-gcp`, over cluster DNS | `http://gitea-http.gitea.svc.cluster.local:3000/gitadmin/…` — pod-to-pod, so it never leaves the cluster and comes back through the ingress |
| `image_tag_yq_path` | `.deployment.image.tag` | **Override this if the app's chart isn't `1.0.0`** — e.g. `sts-2.0.0` uses `.podtemplate.image.tag` instead. Getting this wrong doesn't fail loudly: `yq -i` creates the path if missing rather than erroring, silently leaving the real field un-bumped. |
| `dockerBuildVersion` | none | Only read when the repo has **no Dockerfile of its own** — picks a fallback template (see below). No default; either ship a Dockerfile or set this. |
@@ -49,7 +70,9 @@ a `podTemplate` (`resources/org/homelab/dind-pod.yaml`) via
based on `dockerBuildVersion` (e.g. `go-1.22`, `node-20`,
`python-3.12`, `java-21`, `php-8.3`). All fallback templates pull base
images from Harbor's `base-images` project (mirrored via the separate
`devops-base-images` repo), not Docker Hub directly.
`devops-base-images-gcp` repo), not Docker Hub directly. Only versions
actually mirrored there resolve — an unmirrored tag fails the build
rather than silently falling back to Docker Hub.
- **`updateHelmTag`** — clones `devops-helm-charts`, bumps the image tag
via `yq` at `image_tag_yq_path`, commits, pushes to `main`.
- **`syncArgoApp`** — calls the ArgoCD REST API to sync `argo_app_name`.
@@ -65,8 +88,29 @@ a `podTemplate` (`resources/org/homelab/dind-pod.yaml`) via
## Build-tools image
`resources/org/homelab/build-tools.Dockerfile` bakes git/yq/bash/
python3+pip/venv/curl into the `docker-cli` container's image, so
nothing gets installed on demand on every single build. Built and pushed
manually (not through any Jenkins job) — see that file's own header
comment.
The `docker-cli` container runs
`harbor.35.238.248.203.nip.io/base-images/build-tools:1`, which bakes in
git, yq, bash, python3 with pip and venv, and curl, so nothing is installed
on demand on every build.
**It is not built here.** The Dockerfile lives in `devops-base-images-gcp`,
next to the mirrored base images, because it is the same kind of artefact:
built by hand, occasionally, and pushed to Harbor. A Jenkins job could not
build it anyway — it is the image Jenkins builds *in*.
`dind-pod.yaml` pins the tag, so rebuilding the image rolls nothing out
until that pin is bumped. Bump the tag rather than overwriting one.
## Registry trust
`dind-pod.yaml` mounts the `registry-ca` ConfigMap (published by
`devops-infra-argo-config-gcp`) into the dind container at
`/etc/docker/certs.d/harbor.35.238.248.203.nip.io/ca.crt`.
Without it, pushes fail TLS verification while pulls of the same image
succeed, which reads like a broken registry. The reason is that the two are
different clients: pulls are performed by containerd on the node, which was
told to trust this CA when the node pool was created, whereas the push comes
from dockerd inside the build pod, which has its own trust store. The
directory name must be the registry hostname exactly — dockerd looks the
path up by host and silently ignores a mismatch.
resources/com/homelab/go-Dockerfile
+2 -2
View File
@@ -13,14 +13,14 @@
# a dockerBuildVersion whose tag isn't in devops-base-images/images.txt
# yet needs that added and re-mirrored first, unlike pulling straight
# from Docker Hub where any tag "just worked".
FROM harbor.192.168.1.7.nip.io/base-images/golang:${version}-alpine AS build
FROM harbor.35.238.248.203.nip.io/base-images/golang:${version}-alpine AS build
WORKDIR /src
COPY go.mod go.sum* ./
RUN go mod download 2>/dev/null || true
COPY . .
RUN CGO_ENABLED=0 go build -o /app .
FROM harbor.192.168.1.7.nip.io/base-images/alpine:3.20
FROM harbor.35.238.248.203.nip.io/base-images/alpine:3.20
COPY --from=build /app /app
EXPOSE 8080
ENTRYPOINT ["/app"]
resources/com/homelab/java-Dockerfile
+2 -2
View File
@@ -8,14 +8,14 @@
# to Maven Central for plugins/dependencies during the build regardless
# of base image — this only removes the Docker Hub dependency for the
# base image layer, not package-registry traffic during the build.
FROM harbor.192.168.1.7.nip.io/base-images/maven:3-eclipse-temurin-${version}-alpine AS build
FROM harbor.35.238.248.203.nip.io/base-images/maven:3-eclipse-temurin-${version}-alpine AS build
WORKDIR /src
COPY pom.xml .
RUN mvn -B dependency:go-offline
COPY . .
RUN mvn -B package -DskipTests
FROM harbor.192.168.1.7.nip.io/base-images/eclipse-temurin:${version}-jre-alpine
FROM harbor.35.238.248.203.nip.io/base-images/eclipse-temurin:${version}-jre-alpine
WORKDIR /app
COPY --from=build /src/target/*.jar app.jar
EXPOSE 8080
resources/com/homelab/node-Dockerfile
+2 -2
View File
@@ -5,14 +5,14 @@
# Assumes a standard `npm run build` + `npm start` repo. Switched from
# node:*-slim (Debian) to node:*-alpine for both stages — smaller, still
# keeps a shell for kubectl exec debugging (not distroless).
FROM harbor.192.168.1.7.nip.io/base-images/node:${version}-alpine AS build
FROM harbor.35.238.248.203.nip.io/base-images/node:${version}-alpine AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build --if-present
FROM harbor.192.168.1.7.nip.io/base-images/node:${version}-alpine
FROM harbor.35.238.248.203.nip.io/base-images/node:${version}-alpine
WORKDIR /app
COPY --from=build /app .
ENV NODE_ENV=production
resources/com/homelab/php-Dockerfile
+1 -1
View File
@@ -36,7 +36,7 @@
# very comment did so), new edits to this header should avoid typing
# the character at all — write "dollar sign" in words instead of using
# the glyph.
FROM harbor.192.168.1.7.nip.io/base-images/php:${version}-cli-alpine
FROM harbor.35.238.248.203.nip.io/base-images/php:${version}-cli-alpine
WORKDIR /var/www/html
RUN apk add --no-cache --virtual .build-deps \$PHPIZE_DEPS \
&& docker-php-ext-install pdo pdo_mysql \
resources/com/homelab/python-Dockerfile
+1 -1
View File
@@ -9,7 +9,7 @@
# extensions that only ship glibc wheels may need musl-dev/gcc added
# here to build from source on Alpine — fine for this repo's pure-Python
# deps, worth knowing if a future repo's requirements.txt needs more.
FROM harbor.192.168.1.7.nip.io/base-images/python:${version}-alpine
FROM harbor.35.238.248.203.nip.io/base-images/python:${version}-alpine
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
resources/org/homelab/build-tools.Dockerfile
-13
View File
@@ -1,13 +0,0 @@
# Custom docker-cli image for dind-pod.yaml's docker-cli container —
# bakes in everything the pipeline stages need at runtime (git, yq,
# bash, python3 + pip/venv for runHooks' python hooks, curl) so nothing
# gets apk-installed or curl-downloaded on every single build. Rebuild
# and push this (see the one-off build commands in the commit that
# added this file) whenever this list changes; dind-pod.yaml pins the
# resulting image tag explicitly, so a rebuild doesn't silently roll
# out until that pin is also bumped.
FROM docker:27-cli
RUN apk add --no-cache git bash python3 py3-pip py3-virtualenv curl \
&& curl -sL -o /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 \
&& chmod +x /usr/local/bin/yq
resources/org/homelab/dind-pod.yaml
+41 -19
View File
@@ -15,32 +15,47 @@ spec:
image: docker:27-dind
securityContext:
privileged: true
# Harbor's ingress serves plain HTTP too (TLS disabled cluster-wide
# by design — see claude.md's "everything is plain HTTP" note).
# Docker defaults to attempting HTTPS against any bare registry
# hostname regardless of network path, so this is needed
# regardless of which hostname is used — was previously
# harbor-core.harbor.svc.cluster.local (cluster-internal Service
# DNS, works from this pod but not from the node's own containerd
# when pulling for a real Deployment); switched to the Contour
# ingress hostname so push and pull can share one reference.
args:
- "--insecure-registry=harbor.192.168.1.7.nip.io"
# No --insecure-registry, unlike the homelab: Harbor here serves a real
# certificate, issued by cert-manager from the private CA that the node
# pool was told to trust when it was created.
#
# That node trust covers image PULLS, which containerd performs on the
# node. This push is a different client — dockerd, inside this pod,
# with its own trust store and no knowledge of what the node trusts —
# so it needs the CA mounted itself. dockerd looks it up at
# /etc/docker/certs.d/<registry host>/ca.crt, and the directory name
# must be the registry hostname exactly; anything else is silently
# ignored, and the push then fails TLS verification while a pull of the
# very same image works.
#
# The registry hostname (rather than harbor-core.harbor.svc.cluster.local)
# carries over unchanged from the homelab, for a reason that still
# holds: cluster DNS resolves from this pod but not from the node's
# containerd doing the real Deployment pull, and Docker matches both
# stored credentials and trust by exact hostname — so push and pull
# have to name the registry identically.
env:
- name: DOCKER_TLS_CERTDIR
value: ""
volumeMounts:
- name: docker-graph-storage
mountPath: /var/lib/docker
- name: registry-ca
mountPath: /etc/docker/certs.d/harbor.35.238.248.203.nip.io
readOnly: true
- name: docker-cli
# Custom image (see build-tools.Dockerfile in this same directory)
# — bakes in git/yq/bash/python3+pip/venv/curl so nothing needs
# apk-installing or curl-downloading on every single build (was
# slow and, per its own point, defeats the purpose of a
# reproducible pipeline to be quietly downloading a tool binary
# fresh on every run). Versioned tag, not :latest — a rebuild of
# the tools image doesn't roll out until this pin is bumped too.
image: harbor.192.168.1.7.nip.io/homelab/build-tools:1
# Custom image, built and pushed by hand from devops-base-images-gcp
# (build-tools.Dockerfile there) — bakes in git/yq/bash/python3 with
# pip and venv/curl so nothing needs installing on every single build,
# which was slow and quietly undermined reproducibility.
#
# It lives in the base-images project rather than homelab because that
# project is public: this pod pulls the image before any credential is
# available to it.
#
# Versioned tag, never :latest — rebuilding the tools image must not
# roll out until this pin is bumped deliberately.
image: harbor.35.238.248.203.nip.io/base-images/build-tools:1
command: ["cat"]
tty: true
env:
@@ -77,6 +92,13 @@ spec:
volumes:
- name: docker-graph-storage
emptyDir: {}
- name: registry-ca
configMap:
# Published by devops-infra-argo-config-gcp (extra-manifests). The
# CA's public certificate only — its private key never leaves
# Terraform state and cert-manager, which is why this is a ConfigMap
# rather than a Secret.
name: registry-ca
- name: docker-config
secret:
secretName: harbor-robot-dockerconfig
src/com/homelab/stages/buildDocker.groovy
+13 -9
View File
@@ -20,15 +20,19 @@ import com.homelab.utilities.constructTemplate
def run(Map config) {
def tag = "${env.BUILD_NUMBER}-${env.GIT_COMMIT?.take(7) ?: 'dev'}"
env.TAG = tag
// Was harbor-core.harbor.svc.cluster.local (cluster-internal Service
// DNS) — worked for this push, since it runs inside a pod with
// pod-network DNS (CoreDNS). But the actual Deployment's image PULL
// happens via containerd running on the node itself, using the
// node's host-level DNS resolver, which has no route to
// *.svc.cluster.local at all. Using the Contour ingress hostname
// instead makes the same reference resolvable from both contexts
// (nip.io resolves via normal public DNS, reachable from the host).
def image = "harbor.192.168.1.7.nip.io/${config.harbor_project}/${config.repo_name}:${tag}"
// The registry ingress hostname, not harbor-core.harbor.svc.cluster.local.
// Cluster DNS would work for this push, which runs inside a pod, but the
// Deployment's image PULL is performed by containerd on the node, using
// the node's own resolver, which has no route to *.svc.cluster.local at
// all — that namespace exists only in CoreDNS. This is the one place the
// project's "always use cluster DNS between services" rule cannot apply:
// the puller is not a service.
//
// It also has to be spelled identically everywhere, because Docker
// matches stored credentials and TLS trust by exact hostname: here, the
// dockerconfigjson auths key, Harbor's externalURL, and the node pool's
// CA trust config.
def image = "harbor.35.238.248.203.nip.io/${config.harbor_project}/${config.repo_name}:${tag}"
try {
stage(stageName('Build & push image')) {
container('docker-cli') {
src/com/homelab/stages/updateHelmTag.groovy
+3 -1
View File
@@ -16,7 +16,9 @@ package com.homelab.stages
// service_name second path segment; same as repo_name for a
// single-service repo, different for a monorepo with
// several services sharing one repo
// helm_repo_url e.g. http://gitea.192.168.1.7.nip.io/mukul/devops-helm-charts.git
// helm_repo_url e.g. http://gitea-http.gitea.svc.cluster.local:3000/gitadmin/devops-helm-charts-gcp.git
// (cluster DNS — this clone runs in a build pod, so it
// never goes out through the ingress and back)
// image_tag_yq_path yq path to the tag field, e.g. .deployment.image.tag
// gitea_cred Jenkins credential ID for a Gitea push-capable token (default: gitea-ci-credentials)
//
vars/homelabPipeline.groovy
+4 -1
View File
@@ -24,7 +24,10 @@ def call(Map config) {
config.service_name = config.service_name ?: config.repo_name
config.argo_app_name = config.argo_app_name ?: config.repo_name
config.harbor_project = config.harbor_project ?: 'homelab'
config.helm_repo_url = config.helm_repo_url ?: 'http://gitea.192.168.1.7.nip.io/mukul/devops-helm-charts.git'
// Cluster DNS, not the ingress hostname: this clone happens from a build
// pod, so it is pod-to-pod traffic and has no business leaving the
// cluster and coming back in through Contour.
config.helm_repo_url = config.helm_repo_url ?: 'http://gitea-http.gitea.svc.cluster.local:3000/gitadmin/devops-helm-charts-gcp.git'
config.image_tag_yq_path = config.image_tag_yq_path ?: '.deployment.image.tag'
// Every stage file (both the ones adapted for this homelab and the