130 lines
6.5 KiB
Markdown
130 lines
6.5 KiB
Markdown
# Skill: build-failure-debugger
|
|
|
|
**One-line description:** Given a Jenkins job log from a failed devops-lib build, identify exactly which stage failed, trace the code path through devops-lib that produced the failure, and output a concrete fix with the exact file and line driving the error.
|
|
|
|
**Owner:** DevOps Platform team
|
|
|
|
---
|
|
|
|
## Why this skill exists
|
|
|
|
Jenkins job logs are long, noisy, and mixed-language (Groovy stack traces, Maven output, Docker build output, ArgoCD sync output). When a build fails, an engineer must:
|
|
|
|
1. Scroll through 2000+ lines to find the failure point
|
|
2. Know that "Build" maps to `buildMaven.groovy`, "Deploying to ArgoCD" maps to `deployArgoCD.groovy`, etc.
|
|
3. Know which code path was taken based on `build_tool` and `cicd_environment`
|
|
4. Know which external system (GAR, SonarQube, ArgoCD, Helm repo) caused the failure vs which devops-lib logic caused it
|
|
|
|
The mapping from Jenkins stage names to devops-lib source files is non-obvious and not documented anywhere. A "SonarQube Quality Gate" failure could mean: the threshold was breached, the project doesn't exist in Sonar yet, or a whitelist entry is missing. Three different fixes, identical log output.
|
|
|
|
This skill knows the stage-to-file mapping, the code paths for each failure mode, and the set of conditions that trigger each error.
|
|
|
|
---
|
|
|
|
## Trigger
|
|
|
|
- User pastes a Jenkins job log (or the relevant failure excerpt)
|
|
- `"this Jenkins build failed, help me debug it"` with log attached
|
|
- `"sonar quality gate failing for auth-service on develop"`
|
|
- `"build stuck at ArgoCD sync for catalog-service in prd"`
|
|
- Proactively: when a user shares a build URL, fetch the log and diagnose
|
|
|
|
---
|
|
|
|
## Input
|
|
|
|
- Jenkins job log (pasted text or URL to the Jenkins job)
|
|
- Optionally: the service `config.yaml` (to know `build_tool`, `skip_sonar`, `deployArgo`, etc.)
|
|
|
|
The skill can infer build_tool and environment from the log itself if config.yaml is not provided (Jenkins prints `build_tool` and `cicd_environment` at pipeline start via `constructParam.run()`).
|
|
|
|
---
|
|
|
|
## Expected output
|
|
|
|
Given: Jenkins log showing failure in "Deploying to ArgoCD" stage for `catalog-service` on `develop`
|
|
|
|
```
|
|
Build Failure Debug — catalog-service | branch: develop | env: stg
|
|
|
|
Stage failed: Deploying to ArgoCD
|
|
devops-lib: src/com/meesho/utilities/deployArgoCD.groovy
|
|
|
|
Step that failed: update_helm_repo (step 3 of 4)
|
|
Error in log: ERROR: Helm push failed — chart version 0.1.142 already exists
|
|
|
|
Root cause: The Helm chart version was not bumped between this push and the previous
|
|
build. deployArgoCD.groovy uses the image SHA as the chart version suffix, but the
|
|
SHA collision occurred because the source didn't change (config-only build — latest
|
|
tag was reused from GAR without a new image build).
|
|
|
|
Fix: Force a source change to generate a new image SHA, OR bump the chart version
|
|
manually in devops-argo-config.
|
|
|
|
deployment_order check: 'catalog-service-stg' — verify this matches the app name
|
|
in devops-argo-config (wrong name → silent sync failure at step 4, not step 3).
|
|
|
|
Relevant code:
|
|
deployArgoCD.groovy:update_helm_repo() — pushes chart with SHA-derived version
|
|
constructParam.groovy:getConfigOnlyChange() — sets SKIP_BUILD flag when only YAMLs changed
|
|
```
|
|
|
|
---
|
|
|
|
## Stage-to-file mapping (built-in knowledge)
|
|
|
|
| Jenkins stage name | devops-lib file |
|
|
|---|---|
|
|
| `Build` | `buildMaven.groovy` / `buildGo.groovy` / `buildNode.groovy` / etc. (by `build_tool`) |
|
|
| `Docker Build & Push` | `buildMaven.groovy:dockerBuildAndPush()` / equivalent in each build stage |
|
|
| `SonarQube Analysis` | `buildMaven.groovy:sonar_scan()` / `buildGo.groovy:sonar_scan()` |
|
|
| `SonarQube Quality Gate` | `constructParam.groovy:waitForQualityGate()` |
|
|
| `CAC Validation` | `buildMaven.groovy:cac_validation()` / `buildGo.groovy:cac_validation()` |
|
|
| `Deploying to ArgoCD` | `deployArgoCD.groovy` (4-step: update_argo_repo → refresh_app_of_apps → update_helm_repo → refresh_and_sync) |
|
|
| `Notify` | `notify.groovy` / `notifySlack.groovy` |
|
|
| `AppConfig Validation` | `constructParam.groovy:validateAppConfig()` |
|
|
|
|
---
|
|
|
|
## Failure mode taxonomy
|
|
|
|
The skill classifies every failure into one of these categories before diagnosis:
|
|
|
|
| Category | Signal in log | devops-lib location |
|
|
|---|---|---|
|
|
| Build compilation | `BUILD FAILURE` / `go build failed` | build<Lang>.groovy |
|
|
| Docker push to GAR | `denied` / `UNAUTHORIZED` on push | build<Lang>.groovy:dockerBuildAndPush() |
|
|
| Sonar threshold | `Quality Gate status: FAILED` | constructParam.groovy:waitForQualityGate() |
|
|
| Sonar project missing | `Project not found` | buildGo.groovy:sonar_scan() auto-create logic |
|
|
| ArgoCD step 1 (argo_repo update) | `git push failed` in argo-config repo | deployArgoCD.groovy:update_argo_repo() |
|
|
| ArgoCD step 2 (app-of-apps refresh) | `app not found` / `no Application object` | deployArgoCD.groovy:refresh_app_of_apps() — new service, ArgoCD app not yet created |
|
|
| ArgoCD step 3 (helm push) | `chart version already exists` | deployArgoCD.groovy:update_helm_repo() |
|
|
| ArgoCD step 4 (sync) | `OutOfSync` / `Helm values error` | deployArgoCD.groovy:refresh_and_sync() |
|
|
| Wrong deployment_order | Silent sync on wrong app name | config.yaml:deployment_order vs devops-argo-config |
|
|
| Multizone gate | `must be deployed via Ringmaster` | constructParam.groovy:isMultizoneEnabled() |
|
|
|
|
---
|
|
|
|
## Dependencies
|
|
|
|
- Full read access to devops-lib source (stage-to-file mapping, failure message strings)
|
|
- Jenkins log (pasted by engineer or fetched via Jenkins API)
|
|
- Optionally: `Meesho/whitelists` (to check if skip_sonar or multizone entries explain the failure)
|
|
- Optionally: `Meesho/devops-argo-config` (to validate `deployment_order` app names for ArgoCD failures)
|
|
|
|
---
|
|
|
|
## Design notes
|
|
|
|
- The skill must handle truncated logs (Jenkins UI often shows the last N lines). It should ask for the full log if the failure point is not visible.
|
|
- For hotfix builds: sonar and quality gate failures are expected to be suppressed; if they appear, the hotfix path was not taken — the skill should check the branch name pattern.
|
|
- The skill should distinguish between a devops-lib bug (code path is wrong) vs a configuration error (wrong value in config.yaml or whitelist) vs an external system error (GAR down, Sonar unreachable).
|
|
|
|
---
|
|
|
|
## Open questions
|
|
|
|
- Should it auto-fetch the Jenkins log via the Jenkins API if given a job URL?
|
|
- For ArgoCD step 4 failures (Helm values schema errors): should it parse the exact Helm error and cross-reference with the deployment.yaml schema?
|
|
- Should it suggest a rerun command or ArgoCD force-sync as a recovery action?
|