9.9 KiB
Tribal Knowledge
Non-obvious conventions, design decisions, and operational patterns in devops-lib that aren't captured in comments or standard docs. Authored manually from codebase review.
1. Five whitelists = five git clones per build
constructParam.groovy:getWhitelistedRepos does a fresh git clone of Meesho/whitelists for every whitelist check — there is no caching between calls. A build that hits all five gates (skip-sonar, app-config-disabled, multizone-enabled, allowedNonDevelopPrDeployment, ValidateCacConfig) clones the repo five times. This is deliberate: each clone captures the latest whitelist state so a DevOps change takes effect on the very next build without any library release. The cost is ~5× clone latency under GitHub rate-limiting.
Takeaway: Never refactor getWhitelistedRepos to cache the clone across calls without confirming the freshness guarantee is no longer required.
2. ringmaster-bot is the only signal that distinguishes Ringmaster from Turbo-Turtle
deployRingmaster.groovy:run switches the callback target based solely on getCause(UserIdCause).getUserId() == "ringmaster-bot". There is no explicit flag or env var. If Ringmaster ever renames its bot user, callbacks silently fall through to the Turbo-Turtle endpoint, which will reject them.
Takeaway: The string "ringmaster-bot" is a load-bearing constant. Don't change it without coordinating with the Ringmaster team.
3. buildObjHelper falls through to defaultBuild silently
buildObjHelper.groovy:run matches the toolchain field against a series of conditions (maven, gradle, go, node, python, php, docker). If none match, it instantiates defaultBuild without logging a warning. A mis-spelled toolchain value (e.g. golang instead of go) produces a silent no-op build that succeeds with no artifact.
Takeaway: If a build produces no Docker image but reports success, check toolchain spelling in deployment.yaml first.
4. JVM memory flags are auto-calculated — don't set them manually
deployArgoCD.groovy:update_helm_repo computes xms and xmx from the pod's memory_limit using the formula xms = xmx = memory_limit * 0.5. If a service hard-codes -Xmx in JAVA_OPTS, the auto-calculated value in the Helm values will collide, with the last one winning depending on JVM arg order.
Takeaway: Leave xms/xmx unset in service configs; let the pipeline compute them. If you must override, set jvm_memory_override: true in deployment.yaml to suppress auto-calc.
5. Canary is mandatory for sp0 and up0 — no override
deployArgoCD.groovy:run checks priority_v2 and blocks a non-canary prd deploy if the priority is sp0 or up0. There is no whitelist or flag to bypass this. The check happens before any Helm update, so the build fails fast.
Takeaway: Any service with priority_v2: sp0 or up0 must have canary configured in deployment.yaml. Attempting a direct prd deploy will always fail at the ArgoCD stage.
6. Turbo-Turtle callback uses a temp file to avoid shell escaping
deployRingmaster.groovy:run passes the Turbo-Turtle JSON payload inline via curl -d '$newCICD_JSON'.
Takeaway: If you add a new field to the Turbo-Turtle payload, add it to the temp-file write — never inline it in the curl command.
7. env.CHANGE_ID is the canonical PR-build detector
Every builder and stage that needs to distinguish a PR build from a branch build checks env.CHANGE_ID (set by the GitHub Branch Source plugin). constructParam.groovy:run remaps cicd_environment from prd to int for main/master-targeting PRs and from stg to ftr for develop-targeting PRs based on this. Don't check env.BRANCH_NAME =~ /PR-/; that pattern breaks on non-GitHub SCMs and on re-triggered builds.
Takeaway: Use env.CHANGE_ID to detect PR context, not branch name patterns.
8. Non-prd node pools are BU-scoped, not service-scoped
nodePoolSelection.groovy:run assigns stg/ftr/dev pods to {BU}-shared pools. All services in the same BU share one node pool in non-prd environments. A memory leak or noisy-neighbour in one supply service degrades all other supply services on staging.
Takeaway: Non-prd performance issues may be caused by a neighbour in the same BU pool, not the service under test.
9. The constructTemplate._construct() method is @NonCPS
constructTemplate.groovy:_construct() is annotated @NonCPS because it uses Java regex and string interpolation that is not serialisable by the Jenkins CPS engine. Any caller that invokes it inside a parallel block or closure must ensure the closure itself is either also @NonCPS or does not cross a serialisation boundary.
Takeaway: Don't move _construct into a CPS context (e.g. by inlining its logic into a stage body). Keep the @NonCPS annotation and call it from a CPS-safe wrapper.
11. Node build changes always require updating both buildNode.groovy and node-Dockerfile
src/com/meesho/stages/buildNode.groovy and resources/com/meesho/node-Dockerfile are paired files — the Dockerfile's else branch (the default install command, currently npm ci) must stay in sync with the detection/selection logic in buildNode.groovy. When npm_install_arg is not explicitly set in config.yaml, buildNode.groovy detects the package manager at runtime and passes the install command to the Dockerfile via the npm_install_arg template variable. The Dockerfile's default branch handles the fallback when no explicit or detected command is provided.
Takeaway: Any change to Node install logic (npm ci, npm install, pnpm detection) in buildNode.groovy must be paired with the same intent reflected in node-Dockerfile. Changing only one file leaves the two out of sync and silently breaks either the runtime path or the fallback.
12. The build-tools pod image version encodes the Go and sonar-scanner toolchain
resources/org/meesho/prd-pod.yaml and stg-pod.yaml reference a build-tools image from the internal GAR registry (e.g. asia-southeast1-docker.pkg.dev/meesho-devops-admin-0622/admin/devops/build-tools:lunar-v2.0.21). This image bundles the Go compiler, sonar-scanner-cli, and other language toolchain binaries. When a new toolchain binary is needed (e.g. a newer Go version or sonar-scanner), the fix is to bump the image tag in both pod YAMLs — not to download the tool at build time from go.dev or sonarsource.com. External downloads are a reliability risk behind a corporate network and are the wrong pattern for this repo.
Takeaway: If a build stage needs a new CLI tool or toolchain binary, bump the build-tools image tag in prd-pod.yaml + stg-pod.yaml. Never curl / wget a tool from the internet inside a pipeline stage.
13. Go sonar skip logic lives in constructParam.groovy::skipSonarCheckForGo()
Quality gate and sonar skip decisions for Go builds live in constructParam.groovy, not inline in buildGo.groovy. The pattern mirrors the existing skipSonarCheckForbidden() method: check getWhitelistedRepos("skip-sonar-whitelist") and env.BRANCH_NAME.contains("hotfix"), return true if either matches. This keeps all skip-sonar policy in one place. When adding Go sonar support, add skipSonarCheckForGo(Map config) to constructParam.groovy and call it from buildGo.groovy::buildDckr().
Takeaway: Sonar skip logic belongs in constructParam.groovy. Don't embed whitelist checks or hotfix branch checks inline in language build stages.
14. DinD sidecar DOCKER_HOST values must match the existing infrastructure endpoints
When creating a Docker-in-Docker sidecar pod, set DOCKER_HOST in the pod YAML to the pre-provisioned TCP service endpoints — do not use tcp://localhost:2375 (socket sharing). The correct values are:
| Environment | DOCKER_HOST |
|---|---|
| prd | dind-prd-svc |
| stg / ftr | dind-dev-new-svc.jenkins-new.svc.cluster.local |
These are exactly the same values constructParam.groovy sets via accountDetails[env.cicd_environment]['dockerHost']. Because the pod YAML and constructParam agree on the value, no change to constructParam.groovy is needed when adding a sidecar pod. Using localhost creates a conflict that requires an env.SIDECAR_ENABLED guard — unnecessary complexity that the TCP endpoint approach eliminates.
Also: the DinD container image must come from the internal GAR registry: asia-southeast1-docker.pkg.dev/meesho-devops-admin-0622/admin/devops/docker:28-dind. Never use docker:N-dind from Docker Hub — it will be blocked by network policy and is not security-scanned.
Takeaway: DinD sidecar DOCKER_HOST = dind-prd-svc (prd) or dind-dev-new-svc.jenkins-new.svc.cluster.local (stg). DinD image = internal GAR docker:28-dind. No constructParam changes needed.
15. scm is unavailable inside Jenkins shared library code
The scm variable (branch, remote URL, credentials) is injected by the GitHub Branch Source plugin into consumer Jenkinsfiles only. It is not available inside vars/ or src/ of the shared library. Referencing scm.branches or scm.userRemoteConfigs in library code causes a MissingPropertyException at runtime.
Takeaway: Never access scm in vars/*.groovy or src/com/meesho/**/*.groovy. Use env.BRANCH_NAME, env.GIT_URL, or env.CHANGE_* variables instead — these are set by the plugin before library code runs.
10. ArgoCD app-of-apps must be refreshed before per-service sync
deployArgoCD.groovy:refresh_app_of_apps triggers a sync of the ArgoCD app-of-apps application before syncing the individual service app. Skipping this step means a newly created service (first deploy) won't have its ArgoCD Application object created yet, and the subsequent refresh_and_sync will target a non-existent app.
Takeaway: The four-step deploy order (update_argo_repo → refresh_app_of_apps → update_helm_repo → refresh_and_sync) is load-bearing. Steps 2 and 4 are not interchangeable.