The GCP counterpart of devops-lib. Registered in Jenkins under the same
name, so consuming repos need no change: the two-line Jenkinsfile is
identical on both clusters, and which library it resolves to is a property
of the Jenkins running it.
Substantive changes, all consequences of GKE being a real cloud:
- Harbor speaks TLS here, so dind no longer passes --insecure-registry.
It mounts the private CA at
/etc/docker/certs.d/harbor.35.238.248.203.nip.io/ca.crt instead, from the
registry-ca ConfigMap. This is not redundant with the node pool's trust:
that covers pulls, performed by containerd on the node, while the push
comes from dockerd in the build pod with its own trust store. Without it,
pushes fail TLS verification while pulls of the same image succeed —
which reads like a broken registry rather than a missing trust anchor.
- The registry hostname changes in the push target and all five fallback
Dockerfiles. It still must be spelled identically everywhere, because
Docker matches credentials and trust by exact hostname.
- helm_repo_url moves to cluster DNS. That clone runs in a build pod, so
sending it out through the ingress and back would make the pipeline
depend on Contour for pod-to-pod traffic. syncArgoApp already addressed
ArgoCD this way and needed no change.
- build-tools.Dockerfile is removed: devops-base-images-gcp owns it now,
next to the mirrored base images, and the pod references the result by
tag at base-images/build-tools:1. The base-images project is public, so
the pod can pull it before it has any credentials.
Verified: no homelab addresses remain; the pod template parses with the CA
mount, the new image and no insecure-registry flag; and every fallback
template's base image, with the default version buildDocker would pick, is
present in the mirror manifest — an unmirrored tag now fails the build
rather than silently falling back to Docker Hub.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
Every fallback template (go/node/python/java) assumes one fixed repo
layout — a root go.mod, a root package.json, requirements.txt, a root
pom.xml. Without this, a repo shaped any other way failed inside
`docker build`'s COPY step with a raw "file not found" error that never
said which file the fallback expected or why, several minutes into a
build someone was told needs no Dockerfile at all (testing1 hit exactly
this: no go.mod, "COPY failed: file not found in build context: go.mod").
validateRepoStructure checks for the one file each language's template
actually requires before rendering it, and errors with the file name,
what it's for, and the escape hatch (bring your own Dockerfile, which
always wins over this fallback regardless of layout).
This does not make the fallback dynamic — it is still one fixed layout
per language. It only turns a silent wrong assumption into a message
that says what the assumption was. Real detection across arbitrary repo
layouts is Cloud Native Buildpacks' job, tracked as its own step in
toolshed's docs/PRODUCT-ARCHITECTURE.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
The previous fix (47a60b7) escaped the functional \$PHPIZE_DEPS
reference but added an explanatory comment that itself used several
bare/example $ characters (${PHPIZE_DEPS}, $PHPIZE_DEPS, $IDENTIFIER,
${IDENTIFIER}, and a bare $ before a comma). SimpleTemplateEngine
parses the ENTIRE file as template source, comments included, so any
of those threw either MissingPropertyException or a harder
'illegal string body character after dollar sign' parse error
depending on what followed the $.
Rewrote the comment to explain the rule in prose without using the $
glyph as an illustrative example at all, so future edits to the
comment can't reintroduce this. Verified via grep across all five
fallback Dockerfiles that the only remaining $ occurrences are the
intentional ${version} placeholders plus the one properly-escaped
\$PHPIZE_DEPS.
Build failed: groovy.lang.MissingPropertyException: No such property:
PHPIZE_DEPS. This whole file is rendered through
constructTemplate.groovy's SimpleTemplateEngine before it ever becomes
a real Dockerfile — bare $IDENTIFIER is Groovy GString interpolation
syntax, same as ${IDENTIFIER}, so it tried resolving PHPIZE_DEPS
against the render binding (which only has `version`) instead of
leaving it as literal shell syntax for docker-php-ext-install to
expand at RUN time. \$ is the correct escape for a literal dollar sign
in SimpleTemplateEngine output.
~65 files carried over unmodified from the original org-wide library
(GKE/EKS-specific pipeline entry points, real Maven/Gradle/Node/Python/
PHP/Rust build stages entangled with AWS ECR/S3, GCP GAR/GCS, JFrog,
Sonar, a hardcoded internal security-scan endpoint, GitHub-specific
multi-branch scripting, BU/team org validation, and their supporting
pod templates/Helm value templates) — none directly usable without a
full rewrite regardless (same effort as what's already built for the
5 supported languages), so removed rather than kept as unused
reference. Also drops the Gradle wrapper (build.gradle, gradlew) — its
test suite was already removed in an earlier cleanup this session, so
nothing was left to build/test with it.
README.md rewrote entirely — it was still describing the old system
(eksCICD entry point, GKE/EKS, JFrog, S3, buTeamMapping) and even
referenced buildObjHelper.groovy, a file this same commit removes.
Only the live path remains: 7 stage files, 1 utility, 3 vars files,
the 5 language fallback Dockerfiles + Jenkinsfile template, and the
dind pod template + build-tools Dockerfile.
Every language's Dockerfile fallback template now pulls from
harbor.192.168.1.7.nip.io/base-images/... (mirrored from Docker Hub via
the new devops-base-images repo) instead of pulling live from Docker
Hub on every build. Eliminates that external dependency at build time,
and lets the mirrored tags be deliberately the leanest official
variant rather than whatever a public tag happens to default to.
- go: unchanged tags (golang:1.22-alpine, alpine:3.20 — already
minimal), just re-hosted.
- node/python/java: moved from their Debian-slim defaults to the
-alpine equivalent (node:20-alpine, python:3.12-alpine,
maven:3-eclipse-temurin-21-alpine, eclipse-temurin:21-jre-alpine).
- php: bigger change — dropped php:*-apache (Debian, full Apache
httpd) entirely for php:*-cli-alpine + PHP's own built-in dev server
(`php -S`), moving to port 8080 like every other language instead of
PHP's special-cased 80. Not production-grade PHP serving (PHP's own
docs call the built-in server not designed for that), but genuinely
minimal and fine for a homelab/demo app — would need php-fpm+nginx
for anything serving real traffic.
Only versions actually mirrored into Harbor resolve now — a
dockerBuildVersion whose tag isn't in devops-base-images/images.txt
needs that added and re-mirrored first, unlike pulling straight from
Docker Hub where any tag "just worked". Chose keeping a shell (Alpine)
over full distroless — homelab kubectl-exec debuggability weighed more
than the last bit of attack-surface reduction.
Build failed: "Cloning into '.'..." with no further output, then
git config later failing "fatal: not in a git directory" despite the
cloned file being present and editable. Likely cause: this stage just
started running in container('docker-cli') (root — docker:27-cli's
base has no non-root USER) right when it broke, while deleteDir()
just before it runs via the Jenkins agent's own JNLP process (a
different, non-root UID) — git refuses to trust a repo directory
owned by a different UID than the current process, and that refusal
can surface as an unrelated-looking error on a later command rather
than a clear ownership error on the clone itself. Adds
`git config --global --add safe.directory '*'` (safe here — this
workspace is a throwaway, container-local checkout for one build) and
set -e so a genuinely failed clone stops the script immediately
instead of running later commands against partial state.
Replaces per-build on-demand installs (apk add bash/python3/curl,
curl-downloading yq to /tmp) in runHooks.groovy, syncArgoApp.groovy,
and updateHelmTag.groovy with a single custom image
(build-tools.Dockerfile) that has all of it baked in once, at
image-build time — not repeated on every single pipeline run.
updateHelmTag.groovy also now runs inside container('docker-cli')
(previously unwrapped, defaulting to the auto-injected jnlp agent
container, which is why it needed the curl-downloaded yq fallback in
the first place — that container has git but not yq).
dind-pod.yaml's docker-cli container now points at
harbor.192.168.1.7.nip.io/homelab/build-tools:1 instead of the stock
docker:27-cli — this image needs building and pushing once before any
build using this pod template will work; see build-tools.Dockerfile's
header comment.
demo-go-app's Deployment failed to pull: "dial tcp: lookup
harbor-core.harbor.svc.cluster.local on 127.0.0.53:53: server
misbehaving". docker push worked from the Jenkins build pod because
it has pod-network DNS (CoreDNS); pulling for a real Deployment
happens via containerd on the node itself, using the node's host-level
resolver, which has no route to *.svc.cluster.local at all. Switches
buildDocker.groovy's push target and dind-pod.yaml's
--insecure-registry flag to the Contour ingress hostname instead,
which resolves via normal public DNS (nip.io) from both pods and the
host.
Build #14/#15 kept failing "ARGOCD_TOKEN is empty" even after
confirming the actual Kubernetes Secret has a real token value. Root
cause: the check used Groovy's env.ARGOCD_TOKEN, which is Jenkins'
own pipeline-level environment map — populated from build parameters,
environment{} blocks, withEnv, etc. — not the container's actual OS
environment. A container-scoped env: entry in a podTemplate YAML
(dind-pod.yaml's secretKeyRef) never populates that Groovy map; it
only sets the real process environment inside that container, which
sh steps correctly inherit. So this check was always going to see
null regardless of how correctly Vault/ESO/the Secret were wired —
every fix to that chain was chasing the wrong problem. Moves the
emptiness check into the shell script itself, where $ARGOCD_TOKEN
genuinely resolves.
Build #13: "apk: not found". This sh step isn't wrapped in
container('docker-cli') — unlike runHooks/buildDocker, it never
specifies a container, so it runs in the auto-injected jnlp agent
container (Debian-based jenkins/inbound-agent), not the Alpine
docker-cli one. git clone worked fine in the same step (Debian image
bundles git), just no apk. Fetches the static mikefarah/yq binary via
curl instead of relying on any particular package manager being
present, to /tmp rather than /usr/local/bin since the agent likely
runs as a non-root user.
Build #11 failed: "yq: not found". docker:27-cli's Alpine base
doesn't ship yq by default, same gap as the bash/python3 on-demand
installs already in runHooks.groovy. Alpine renamed the mikefarah/yq
package from `yq` to `yq-go` at v3.20 (and an unrelated Python-based
tool is also sometimes packaged as plain `yq` on other distros, with
incompatible syntax) — tries yq-go first, falls back to yq, rather
than assume which Alpine version docker:27-cli currently ships.
Build #10 failed before its sh step even ran:
"IllegalArgumentException: named capturing group is missing trailing
'}'". Root cause: replaceFirst('http://', "http://\${GIT_USER}:...")
— replaceFirst's *replacement* argument is parsed with Java
regex-replacement syntax, where ${name} means "substitute named
capture group", not literal text. The pattern 'http://' has no named
groups, so Java's regex engine choked trying to resolve
${GIT_USER}/${GIT_PASS} as group references. Replaced with a plain
string split + concatenation, which has no regex-replacement
semantics to collide with, while still keeping \${GIT_USER}/
\${GIT_PASS} literal in the Groovy string so the shell (not Groovy)
expands them from the credential-bound env vars at sh-step time.
Confirmed via a direct token request to Harbor's own token endpoint
(same username/password from harbor-robot-dockerconfig) that the
robot account genuinely has push access to homelab/demo-go-app —
Harbor returned a valid token with actions:[pull,push]. So the actual
`docker push` failure ("unauthorized... action: push") wasn't a
permissions problem at all.
Root cause: kubernetes.io/dockerconfigjson secrets are required to
store their data under the fixed key `.dockerconfigjson`. Mounting
the secret without remapping that key meant the file that actually
landed at /root/.docker was named `.dockerconfigjson`, not
`config.json` — the only filename docker's CLI reads for stored
credentials. Docker found nothing there and pushed unauthenticated,
which Harbor correctly rejected. Adds an items: remap so the mounted
file is named config.json.
Build #7 failed with "Cannot connect to the Docker daemon at
tcp://localhost:2375" right at the first sh step. Verified the dind
entrypoint script directly (docker-library/docker's
dockerd-entrypoint.sh) — with DOCKER_TLS_CERTDIR="" and a
dash-prefixed arg it correctly builds
`dockerd --host=tcp://0.0.0.0:2375 --insecure-registry=...`, so the
--insecure-registry flag added last commit isn't logically wrong.
This is a container-start race instead: Kubernetes doesn't guarantee
ordering between containers in the same pod, so the docker-cli
container's first sh can fire before dockerd in the sibling container
has finished its startup checks (iptables detection etc. run on every
start). Polls `docker info` for up to 60s before the actual build
instead of assuming instant availability.
docker push was hanging: "Client.Timeout exceeded while awaiting
headers" doing a TLS handshake against harbor-core.harbor.svc.cluster.local,
which only ever speaks plain HTTP (TLS disabled cluster-wide by
design). Docker defaults to HTTPS for any bare registry hostname
regardless of network path — that default has nothing to do with
whether traffic routes through Contour/Ingress, contrary to what an
earlier pending-items note assumed. Adds --insecure-registry to the
dind container's dockerd startup args.
Build #5 got through checkout, loadConfig, and the pre_build hook,
then hit two real bugs at the actual docker build step:
1. `docker build` failed with "mkdir /root/.docker/buildx: read-only
file system" — dind-pod.yaml mounts the Harbor push-auth secret
read-only at /root/.docker, but modern docker defaults to
BuildKit/buildx, which wants to write its own state there. Forces
the classic builder via DOCKER_BUILDKIT=0 instead.
2. The subsequent error-handling itself then threw a
NullPointerException — every stage file (including untouched
legacy ones from the real devops-lib) reads env.FAILURE when
setting currentBuild.result, but nothing in this repo ever defined
it, so it was null. Defined once in homelabPipeline.groovy rather
than touching 40+ individual occurrences across every stage file.
Drops AGENTS.md, BUGS_AND_IMPROVEMENTS_REPORT.md, CLAUDE.md,
ai-blitz/*, and docs/{SECURITY.md,acronyms.md,adr/*} — leftover
documentation and task scaffolding from the original org-wide devops-
lib that don't describe this homelab's simplified single-service
pipeline. These were already missing from the working tree from
earlier cleanup; this commit just records that state.
Renames src/com/meesho -> src/com/homelab, resources/com/meesho ->
resources/com/homelab, resources/org/meesho -> resources/org/homelab
(via git mv, preserving history), and sweeps every remaining
occurrence of "meesho" (any casing) out of package declarations,
imports, libraryResource() paths, and comments across the whole repo.
Also drops the per-user allowlist in vars/eksCICD.groovy, which
hardcoded real former-colleagues' emails and doesn't apply to a
single-person homelab — that branch is now permanently skipped rather
than deleted outright, to avoid hand-editing the escape-sequence-heavy
echo blocks it guards (eksCICD.groovy itself is unused legacy code,
not called by homelabPipeline.groovy).
Does not touch the ~114 files that were already missing from the
working tree but still tracked in the prior commit — that's unrelated
pre-existing state, left as-is.
