134 lines
9.9 KiB
Markdown
134 lines
9.9 KiB
Markdown
# Tribal Knowledge
|
||
|
||
> Non-obvious conventions, design decisions, and operational patterns in devops-lib
|
||
> that aren't captured in comments or standard docs. Authored manually from codebase review.
|
||
|
||
---
|
||
|
||
## 1. Five whitelists = five git clones per build
|
||
|
||
`constructParam.groovy:getWhitelistedRepos` does a fresh `git clone` of `Meesho/whitelists` for every whitelist check — there is no caching between calls. A build that hits all five gates (`skip-sonar`, `app-config-disabled`, `multizone-enabled`, `allowedNonDevelopPrDeployment`, `ValidateCacConfig`) clones the repo five times. This is deliberate: each clone captures the latest whitelist state so a DevOps change takes effect on the very next build without any library release. The cost is ~5× clone latency under GitHub rate-limiting.
|
||
|
||
**Takeaway:** Never refactor `getWhitelistedRepos` to cache the clone across calls without confirming the freshness guarantee is no longer required.
|
||
|
||
---
|
||
|
||
## 2. `ringmaster-bot` is the only signal that distinguishes Ringmaster from Turbo-Turtle
|
||
|
||
`deployRingmaster.groovy:run` switches the callback target based solely on `getCause(UserIdCause).getUserId() == "ringmaster-bot"`. There is no explicit flag or env var. If Ringmaster ever renames its bot user, callbacks silently fall through to the Turbo-Turtle endpoint, which will reject them.
|
||
|
||
**Takeaway:** The string `"ringmaster-bot"` is a load-bearing constant. Don't change it without coordinating with the Ringmaster team.
|
||
|
||
---
|
||
|
||
## 3. buildObjHelper falls through to `defaultBuild` silently
|
||
|
||
`buildObjHelper.groovy:run` matches the `toolchain` field against a series of conditions (maven, gradle, go, node, python, php, docker). If none match, it instantiates `defaultBuild` without logging a warning. A mis-spelled toolchain value (e.g. `golang` instead of `go`) produces a silent no-op build that succeeds with no artifact.
|
||
|
||
**Takeaway:** If a build produces no Docker image but reports success, check `toolchain` spelling in `deployment.yaml` first.
|
||
|
||
---
|
||
|
||
## 4. JVM memory flags are auto-calculated — don't set them manually
|
||
|
||
`deployArgoCD.groovy:update_helm_repo` computes `xms` and `xmx` from the pod's `memory_limit` using the formula `xms = xmx = memory_limit * 0.5`. If a service hard-codes `-Xmx` in `JAVA_OPTS`, the auto-calculated value in the Helm values will collide, with the last one winning depending on JVM arg order.
|
||
|
||
**Takeaway:** Leave `xms`/`xmx` unset in service configs; let the pipeline compute them. If you must override, set `jvm_memory_override: true` in `deployment.yaml` to suppress auto-calc.
|
||
|
||
---
|
||
|
||
## 5. Canary is mandatory for sp0 and up0 — no override
|
||
|
||
`deployArgoCD.groovy:run` checks `priority_v2` and blocks a non-canary `prd` deploy if the priority is `sp0` or `up0`. There is no whitelist or flag to bypass this. The check happens before any Helm update, so the build fails fast.
|
||
|
||
**Takeaway:** Any service with `priority_v2: sp0` or `up0` must have canary configured in `deployment.yaml`. Attempting a direct prd deploy will always fail at the ArgoCD stage.
|
||
|
||
---
|
||
|
||
## 6. Turbo-Turtle callback uses a temp file to avoid shell escaping
|
||
|
||
`deployRingmaster.groovy:run` passes the Turbo-Turtle JSON payload inline via `curl -d '$newCICD_JSON'`.
|
||
|
||
**Takeaway:** If you add a new field to the Turbo-Turtle payload, add it to the temp-file write — never inline it in the curl command.
|
||
|
||
---
|
||
|
||
## 7. `env.CHANGE_ID` is the canonical PR-build detector
|
||
|
||
Every builder and stage that needs to distinguish a PR build from a branch build checks `env.CHANGE_ID` (set by the GitHub Branch Source plugin). `constructParam.groovy:run` remaps `cicd_environment` from `prd` to `int` for main/master-targeting PRs and from `stg` to `ftr` for develop-targeting PRs based on this. Don't check `env.BRANCH_NAME =~ /PR-/`; that pattern breaks on non-GitHub SCMs and on re-triggered builds.
|
||
|
||
**Takeaway:** Use `env.CHANGE_ID` to detect PR context, not branch name patterns.
|
||
|
||
---
|
||
|
||
## 8. Non-prd node pools are BU-scoped, not service-scoped
|
||
|
||
`nodePoolSelection.groovy:run` assigns `stg`/`ftr`/`dev` pods to `{BU}-shared` pools. All services in the same BU share one node pool in non-prd environments. A memory leak or noisy-neighbour in one `supply` service degrades all other `supply` services on staging.
|
||
|
||
**Takeaway:** Non-prd performance issues may be caused by a neighbour in the same BU pool, not the service under test.
|
||
|
||
---
|
||
|
||
## 9. The `constructTemplate._construct()` method is `@NonCPS`
|
||
|
||
`constructTemplate.groovy:_construct()` is annotated `@NonCPS` because it uses Java regex and string interpolation that is not serialisable by the Jenkins CPS engine. Any caller that invokes it inside a `parallel` block or closure must ensure the closure itself is either also `@NonCPS` or does not cross a serialisation boundary.
|
||
|
||
**Takeaway:** Don't move `_construct` into a CPS context (e.g. by inlining its logic into a `stage` body). Keep the `@NonCPS` annotation and call it from a CPS-safe wrapper.
|
||
|
||
---
|
||
|
||
## 11. Node build changes always require updating both buildNode.groovy and node-Dockerfile
|
||
|
||
`src/com/meesho/stages/buildNode.groovy` and `resources/com/meesho/node-Dockerfile` are paired files — the Dockerfile's `else` branch (the default install command, currently `npm ci`) must stay in sync with the detection/selection logic in `buildNode.groovy`. When `npm_install_arg` is not explicitly set in `config.yaml`, `buildNode.groovy` detects the package manager at runtime and passes the install command to the Dockerfile via the `npm_install_arg` template variable. The Dockerfile's default branch handles the fallback when no explicit or detected command is provided.
|
||
|
||
**Takeaway:** Any change to Node install logic (`npm ci`, `npm install`, pnpm detection) in `buildNode.groovy` must be paired with the same intent reflected in `node-Dockerfile`. Changing only one file leaves the two out of sync and silently breaks either the runtime path or the fallback.
|
||
|
||
---
|
||
|
||
## 12. The build-tools pod image version encodes the Go and sonar-scanner toolchain
|
||
|
||
`resources/org/meesho/prd-pod.yaml` and `stg-pod.yaml` reference a `build-tools` image from the internal GAR registry (e.g. `asia-southeast1-docker.pkg.dev/meesho-devops-admin-0622/admin/devops/build-tools:lunar-v2.0.21`). This image bundles the Go compiler, `sonar-scanner-cli`, and other language toolchain binaries. When a new toolchain binary is needed (e.g. a newer Go version or sonar-scanner), the fix is to bump the image tag in both pod YAMLs — not to download the tool at build time from `go.dev` or `sonarsource.com`. External downloads are a reliability risk behind a corporate network and are the wrong pattern for this repo.
|
||
|
||
**Takeaway:** If a build stage needs a new CLI tool or toolchain binary, bump the `build-tools` image tag in `prd-pod.yaml` + `stg-pod.yaml`. Never `curl` / `wget` a tool from the internet inside a pipeline stage.
|
||
|
||
---
|
||
|
||
## 13. Go sonar skip logic lives in `constructParam.groovy::skipSonarCheckForGo()`
|
||
|
||
Quality gate and sonar skip decisions for Go builds live in `constructParam.groovy`, not inline in `buildGo.groovy`. The pattern mirrors the existing `skipSonarCheckForbidden()` method: check `getWhitelistedRepos("skip-sonar-whitelist")` and `env.BRANCH_NAME.contains("hotfix")`, return `true` if either matches. This keeps all skip-sonar policy in one place. When adding Go sonar support, add `skipSonarCheckForGo(Map config)` to `constructParam.groovy` and call it from `buildGo.groovy::buildDckr()`.
|
||
|
||
**Takeaway:** Sonar skip logic belongs in `constructParam.groovy`. Don't embed whitelist checks or hotfix branch checks inline in language build stages.
|
||
|
||
---
|
||
|
||
## 14. DinD sidecar DOCKER_HOST values must match the existing infrastructure endpoints
|
||
|
||
When creating a Docker-in-Docker sidecar pod, set `DOCKER_HOST` in the pod YAML to the pre-provisioned TCP service endpoints — **do not use `tcp://localhost:2375`** (socket sharing). The correct values are:
|
||
|
||
| Environment | DOCKER_HOST |
|
||
|---|---|
|
||
| prd | `dind-prd-svc` |
|
||
| stg / ftr | `dind-dev-new-svc.jenkins-new.svc.cluster.local` |
|
||
|
||
These are exactly the same values `constructParam.groovy` sets via `accountDetails[env.cicd_environment]['dockerHost']`. Because the pod YAML and `constructParam` agree on the value, **no change to `constructParam.groovy` is needed** when adding a sidecar pod. Using `localhost` creates a conflict that requires an `env.SIDECAR_ENABLED` guard — unnecessary complexity that the TCP endpoint approach eliminates.
|
||
|
||
Also: the DinD container image must come from the internal GAR registry: `asia-southeast1-docker.pkg.dev/meesho-devops-admin-0622/admin/devops/docker:28-dind`. Never use `docker:N-dind` from Docker Hub — it will be blocked by network policy and is not security-scanned.
|
||
|
||
**Takeaway:** DinD sidecar DOCKER_HOST = `dind-prd-svc` (prd) or `dind-dev-new-svc.jenkins-new.svc.cluster.local` (stg). DinD image = internal GAR `docker:28-dind`. No constructParam changes needed.
|
||
|
||
---
|
||
|
||
## 15. `scm` is unavailable inside Jenkins shared library code
|
||
|
||
The `scm` variable (branch, remote URL, credentials) is injected by the GitHub Branch Source plugin into **consumer Jenkinsfiles** only. It is not available inside `vars/` or `src/` of the shared library. Referencing `scm.branches` or `scm.userRemoteConfigs` in library code causes a `MissingPropertyException` at runtime.
|
||
|
||
**Takeaway:** Never access `scm` in `vars/*.groovy` or `src/com/meesho/**/*.groovy`. Use `env.BRANCH_NAME`, `env.GIT_URL`, or `env.CHANGE_*` variables instead — these are set by the plugin before library code runs.
|
||
|
||
---
|
||
|
||
## 10. ArgoCD app-of-apps must be refreshed before per-service sync
|
||
|
||
`deployArgoCD.groovy:refresh_app_of_apps` triggers a sync of the ArgoCD `app-of-apps` application before syncing the individual service app. Skipping this step means a newly created service (first deploy) won't have its ArgoCD `Application` object created yet, and the subsequent `refresh_and_sync` will target a non-existent app.
|
||
|
||
**Takeaway:** The four-step deploy order (`update_argo_repo → refresh_app_of_apps → update_helm_repo → refresh_and_sync`) is load-bearing. Steps 2 and 4 are not interchangeable.
|