gitadmin/devops-lib-gcp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added files

Browse Source
This commit is contained in:
Your Name
2026-08-26 02:02:24 +05:30
parent 58ee8a276a
commit 3419cfba0c
200 changed files with 22132 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/wiki/pages/concepts/observability.md
+55
View File
@@ -0,0 +1,55 @@
<!-- m-wiki: type=concept slug=observability topic=concepts base-sha=5399a5ddc36b generated-at=2026-05-21 sources=[code:vars/log.groovy] -->
> Generated 2026-05-21 at base-sha 5399a5ddc36b. Type: concept. 1 source.
# Logging via `log.groovy`
[`vars/log.groovy`](../../../../vars/log.groovy) is the closest thing this codebase has to a logger. It's a 24-line Groovy file that wraps `echo` with ANSI colour codes.
## The three calls
```groovy
log.info(msg) // → echo "${GREEN}INFO: ${msg}${BLACK}"
log.warning(msg) // → echo "${RED}WARNING: ${msg}${BLACK}"
log.error(msg) // → echo "${RED}ERROR: ${msg}${BLACK}"
```
That's the whole API. There is no severity level configuration, no structured fields (no JSON), no destination other than the Jenkins console.
## What `log.info` gives you over bare `echo`
- **ANSI colour** — pipeline output in the Jenkins UI is easier to scan when INFO is green and ERROR is red.
- **Consistent prefix** (`INFO: ` / `WARNING: ` / `ERROR: `) — a `grep ERROR pipeline.log` pattern works across every service.
- **Nothing else.** No timestamps (Jenkins adds those via `timestamps` wrapper), no caller tracing, no correlation ID.
## When `bare echo` is acceptable
Lots of existing code uses bare `echo`. The mix is mostly historical. New code should prefer `log.info` for normal messages and `log.error` for failures, but you'll see bare `echo` in:
- One-off banners (`echo "=========="`).
- Single-line status (`echo "Building module: ${m}"` inside a loop).
- ANSI-coloured ad-hoc messages where the code path needs a one-off colour (some `vars/eksCICD.groovy` and `vars/buildPipeline.groovy` lines do this).
There is no policy that bans bare `echo`. The reconcile run's `BUGS_AND_IMPROVEMENTS_REPORT.md` does list "inconsistent logging" as a P1 item but no rule has been graduated into `CLAUDE.md` NEVER DO yet.
## `env.msg` as failure state
A separate convention: most stages set `env.msg` to a human-readable failure reason before re-throwing, e.g. [`buildNode.groovy:21`](../../../../src/com/meesho/stages/buildNode.groovy):
```groovy
env.msg = 'Error in building node packages...'
log.error(env.msg)
currentBuild.result = env.FAILURE
throw e
```
`notify.groovy` then reads `env.msg` (and the related `env.error_msg_to_db`) when composing the Slack message and the deployment-tracker payload. **Don't rename `env.msg`** — too many callers read it. [`deployRingmaster.groovy`](../../../../src/com/meesho/stages/deployRingmaster.groovy) catch blocks are inconsistent here: some set `env.msg`, others don't — flagged in `BUGS_AND_IMPROVEMENTS_REPORT.md` as standardisation work.
## What this codebase doesn't have
- No structured logging (no JSON, no key-value pairs, no trace IDs).
- No log level filtering — every `log.info` always prints.
- No log forwarding to anything other than the Jenkins console.
- No "audit log" stream separate from the pipeline log.
If you need to instrument a pipeline run for external observability, the current convention is to POST to a downstream system directly (see how `notify.groovy` calls the Deployment Tracker at lines 108-152). There is no shared metric/event emitter.
docs/wiki/pages/concepts/secrets-and-auth.md
+51
View File
@@ -0,0 +1,51 @@
<!-- m-wiki: type=concept slug=secrets-and-auth topic=concepts base-sha=5399a5ddc36b generated-at=2026-05-21 sources=[code:src/com/meesho/utilities/addSSHKey.groovy, code:src/com/meesho/stages/buildNode.groovy, code:src/com/meesho/stages/buildGo.groovy, code:src/com/meesho/stages/deployArgoCD.groovy, code:src/com/meesho/stages/securityScan.groovy, code:src/com/meesho/stages/notify.groovy] -->
> Generated 2026-05-21 at base-sha 5399a5ddc36b. Type: concept. 6 sources.
# Secrets, SSH keys, and downstream auth
How the pipeline authenticates to every external system, in one place. Every entry below comes from a real `withCredentials { ... }` block or env var read in the current code.
## Downstreams + auth mechanisms
| Downstream | Credential / mechanism | Code site |
|---|---|---|
| GitHub clone/push (HTTPS) | `env.GITHUB_CRED = 'svc-devops-meesho'` (`gitUsernamePassword`) | [`gitActions.groovy:13`](../../../../src/com/meesho/utilities/gitActions.groovy), [`buildGo.groovy:300`](../../../../src/com/meesho/stages/buildGo.groovy), [`buildNode.groovy:413`](../../../../src/com/meesho/stages/buildNode.groovy) |
| GitHub clone (SSH, private repos) | `credentialsId: 'ssh-private-key'` written to `./id_github_jenkins` (0600) | [`addSSHKey.groovy:4-5`](../../../../src/com/meesho/utilities/addSSHKey.groovy) |
| Vault (GCP secrets) | `env.vaultToken` string | [`buildNode.groovy:555`](../../../../src/com/meesho/stages/buildNode.groovy), [`constructParam.groovy:170,175`](../../../../src/com/meesho/utilities/constructParam.groovy) (`vault-prd.meeshogcp.in`, `vault-dev.meeshogcp.in`) |
| JFrog (Maven deploy) | `-DuseProdRepo=true` / `-DuseTestRepo=true` Maven profile | [`onlyPushtoJfrog.groovy:43-47`](../../../../vars/onlyPushtoJfrog.groovy) |
| GCP Docker registry | `gcloud auth configure-docker` (SDK ambient auth) | [`buildGo.groovy:126`](../../../../src/com/meesho/stages/buildGo.groovy), [`buildNode.groovy:374`](../../../../src/com/meesho/stages/buildNode.groovy) |
| AWS ECR | `aws ecr get-login-password ... \| docker login --password-stdin` | [`buildNode.groovy:371`](../../../../src/com/meesho/stages/buildNode.groovy), [`buildPython.groovy:93`](../../../../src/com/meesho/stages/buildPython.groovy), [`buildMaven.groovy:566`](../../../../src/com/meesho/stages/buildMaven.groovy), [`buildGradle.groovy:520`](../../../../src/com/meesho/stages/buildGradle.groovy), [`buildPhp.groovy:59`](../../../../src/com/meesho/stages/buildPhp.groovy) |
| ArgoCD | `env.argoCreds` (`usernamePassword`); `argocd login ${env.argoURL}:443` | [`deployArgoCD.groovy:490, 522`](../../../../src/com/meesho/stages/deployArgoCD.groovy) |
| npm registry | `.npmrc` from AWS Secrets Manager or Vault → written to workspace | [`buildNode.groovy:63-68, 337`](../../../../src/com/meesho/stages/buildNode.groovy) |
| SonarQube | `env.sonarToken` string; `withSonarQubeEnv { ... }` against `sonarqube-prd` | [`buildMaven.groovy:245-251`](../../../../src/com/meesho/stages/buildMaven.groovy), [`buildNode.groovy:406`](../../../../src/com/meesho/stages/buildNode.groovy) |
| Ringmaster | `credentialsId: 'ringmaster-token'` (`usernamePassword`) | [`deployRingmaster.groovy:115`](../../../../src/com/meesho/stages/deployRingmaster.groovy), [`notify.groovy:117`](../../../../src/com/meesho/stages/notify.groovy) |
## The SSH key write path
[`addSSHKey.groovy:3-7`](../../../../src/com/meesho/utilities/addSSHKey.groovy) writes the credential file inside `withCredentials { ... }`:
```groovy
withCredentials([sshUserPrivateKey(credentialsId: 'ssh-private-key', keyFileVariable: 'FILE')]) {
sh "cat ${FILE} > ./id_github_jenkins; chmod 600 ./id_github_jenkins; ..."
}
```
There is a **race window** between the `cat` write and the `chmod` — a co-resident process could read the file with default umask permissions for that brief interval. There is also **no cleanup** of `./id_github_jenkins` after use. Both are flagged in `BUGS_AND_IMPROVEMENTS_REPORT.md`.
The key file is **NOT cat'd to stdout / logs** — earlier PR-review concerns (PR #634) about that pattern have been remediated; the current `cat ${FILE} > ./id_github_jenkins` is a file write, not a print. See [`review-learnings.md`](../../../../review-learnings.md) for the historical trail.
## NEVER DO
- **Never print or `cat` an SSH private key to stdout / logs.** Always go through `withCredentials` + a 0600 file. (Graduated rule — see [`CLAUDE.md`](../../../../CLAUDE.md) NEVER DO.)
- **Never hard-code bare IPs as curl/HTTP targets.** Use DNS hostnames. Known existing violation: [`securityScan.groovy:11`](../../../../src/com/meesho/stages/securityScan.groovy) — `final String url = '172.31.5.29:63232/scans'`. Flagged for remediation; do not add new violations.
- **Never pass passwords on the command line** where they'll appear in `ps`. ArgoCD's login at [`deployArgoCD.groovy:494, 525`](../../../../src/com/meesho/stages/deployArgoCD.groovy) does pass `--password ${ARGO_PASSWORD}` on argv — also flagged.
## Where the secrets actually live
| System | Where the credential is provisioned |
|---|---|
| Jenkins credential store | `svc-devops-meesho`, `ssh-private-key`, `argoCreds`, `ringmaster-token`, `sonarToken`, `vaultToken` |
| Vault (`vault-prd.meeshogcp.in` / `vault-dev.meeshogcp.in`) | runtime service secrets, `MEESHO_NPMRC_SECRET` |
| AWS Secrets Manager | `MEESHO_NPMRC_SECRET` (alternate fetch path) |
| GCP IAM service accounts | Docker registry, GKE access — via ambient `gcloud auth` |
docs/wiki/pages/concepts/whitelists.md
+31
View File
@@ -0,0 +1,31 @@
<!-- m-wiki: type=concept slug=whitelists topic=concepts base-sha=5399a5ddc36b generated-at=2026-05-21 sources=[code:src/com/meesho/utilities/constructParam.groovy] -->
> Generated 2026-05-21 at base-sha 5399a5ddc36b. Type: concept. 1 source.
# The five whitelist gates
[`constructParam.groovy`](../../../../src/com/meesho/utilities/constructParam.groovy) runs five independent whitelist checks against `Meesho/whitelists`. Each one does a **fresh `git clone`** — there is no caching. A build that hits all five clones the whitelist repo five times.
## The five gates
| Gate | What it controls | Function | Read at |
|---|---|---|---|
| `skip-sonar-whitelist` | Blocks `skip_sonar=true` for Maven on prd unless repo is allowlisted | `skipSonarCheckForbidden` | [`constructParam.groovy:40-57`](../../../../src/com/meesho/utilities/constructParam.groovy) |
| `app-config-disabled` | Blocks `appConfig=false` on stg for Maven/Gradle unless allowlisted | `appConfigDisabledForbidden` | [`constructParam.groovy:62-72`](../../../../src/com/meesho/utilities/constructParam.groovy) |
| `multizone-enabled-repos` | Gates the multi-zone deploy path | `isMultizoneEnabled` | [`constructParam.groovy:29-35`](../../../../src/com/meesho/utilities/constructParam.groovy) |
| `allowedNonDevelopPrDeploymentToInt` | Allows non-`develop` PRs to deploy to `int` | `allowedNonDevelopPrDeploymentToIntRepos` | [`constructParam.groovy:77-83`](../../../../src/com/meesho/utilities/constructParam.groovy) |
| `ValidateCacConfig` | Gates CAC validation on PR build | `ValidateCacConfigForRepo` | [`constructParam.groovy:88-95`](../../../../src/com/meesho/utilities/constructParam.groovy) |
## Why fresh-clone every time
This is **intentional**. The whitelist is the live, authoritative source of which repos opt out of which check. By re-cloning on every call, a DevOps change to the whitelist takes effect on the **next** build in the org without needing a devops-lib release. The cost is ~5× clone latency under GitHub rate-limiting; the benefit is zero release coordination.
## Do NOT add caching
The single most tempting refactor in this code is to cache the clone across the five calls in a single build. Don't — the freshness guarantee is the load-bearing property. See [`docs/tribal-knowledge.md`](../../../tribal-knowledge.md) §1. If you must improve clone performance, do it inside the clone itself (shallow clone, single-branch fetch) without touching the per-call invocation pattern.
## Where the whitelist lives
`https://github.com/Meesho/whitelists.git` (cloned via [`gitActions.groovy`](../../../../src/com/meesho/utilities/gitActions.groovy) helpers). The repo contains one YAML per whitelist name, e.g. `skip-sonar-whitelist.yaml`, `multizone-enabled-repos.yaml`. Each is a flat list of repo names.
To add a repo to a whitelist: open a PR on `Meesho/whitelists`, get a DevOps reviewer to approve, merge. The next pipeline run picks up the change automatically.