Skill: build-failure-debugger

One-line description: Given a Jenkins job log from a failed devops-lib build, identify exactly which stage failed, trace the code path through devops-lib that produced the failure, and output a concrete fix with the exact file and line driving the error.

Owner: DevOps Platform team

Why this skill exists

Jenkins job logs are long, noisy, and mixed-language (Groovy stack traces, Maven output, Docker build output, ArgoCD sync output). When a build fails, an engineer must:

Scroll through 2000+ lines to find the failure point
Know that "Build" maps to buildMaven.groovy, "Deploying to ArgoCD" maps to deployArgoCD.groovy, etc.
Know which code path was taken based on build_tool and cicd_environment
Know which external system (GAR, SonarQube, ArgoCD, Helm repo) caused the failure vs which devops-lib logic caused it

The mapping from Jenkins stage names to devops-lib source files is non-obvious and not documented anywhere. A "SonarQube Quality Gate" failure could mean: the threshold was breached, the project doesn't exist in Sonar yet, or a whitelist entry is missing. Three different fixes, identical log output.

This skill knows the stage-to-file mapping, the code paths for each failure mode, and the set of conditions that trigger each error.

Trigger

User pastes a Jenkins job log (or the relevant failure excerpt)
"this Jenkins build failed, help me debug it" with log attached
"sonar quality gate failing for auth-service on develop"
"build stuck at ArgoCD sync for catalog-service in prd"
Proactively: when a user shares a build URL, fetch the log and diagnose

Input

Jenkins job log (pasted text or URL to the Jenkins job)
Optionally: the service config.yaml (to know build_tool, skip_sonar, deployArgo, etc.)

The skill can infer build_tool and environment from the log itself if config.yaml is not provided (Jenkins prints build_tool and cicd_environment at pipeline start via constructParam.run()).

Expected output

Given: Jenkins log showing failure in "Deploying to ArgoCD" stage for catalog-service on develop

Build Failure Debug — catalog-service | branch: develop | env: stg

Stage failed:  Deploying to ArgoCD
devops-lib:    src/com/meesho/utilities/deployArgoCD.groovy

Step that failed: update_helm_repo (step 3 of 4)
Error in log:  ERROR: Helm push failed — chart version 0.1.142 already exists

Root cause: The Helm chart version was not bumped between this push and the previous
  build. deployArgoCD.groovy uses the image SHA as the chart version suffix, but the
  SHA collision occurred because the source didn't change (config-only build — latest
  tag was reused from GAR without a new image build).

Fix: Force a source change to generate a new image SHA, OR bump the chart version
  manually in devops-argo-config.

deployment_order check: 'catalog-service-stg' — verify this matches the app name
  in devops-argo-config (wrong name → silent sync failure at step 4, not step 3).

Relevant code:
  deployArgoCD.groovy:update_helm_repo() — pushes chart with SHA-derived version
  constructParam.groovy:getConfigOnlyChange() — sets SKIP_BUILD flag when only YAMLs changed

Stage-to-file mapping (built-in knowledge)

Jenkins stage name	devops-lib file
`Build`	`buildMaven.groovy` / `buildGo.groovy` / `buildNode.groovy` / etc. (by `build_tool`)
`Docker Build & Push`	`buildMaven.groovy:dockerBuildAndPush()` / equivalent in each build stage
`SonarQube Analysis`	`buildMaven.groovy:sonar_scan()` / `buildGo.groovy:sonar_scan()`
`SonarQube Quality Gate`	`constructParam.groovy:waitForQualityGate()`
`CAC Validation`	`buildMaven.groovy:cac_validation()` / `buildGo.groovy:cac_validation()`
`Deploying to ArgoCD`	`deployArgoCD.groovy` (4-step: update_argo_repo → refresh_app_of_apps → update_helm_repo → refresh_and_sync)
`Notify`	`notify.groovy` / `notifySlack.groovy`
`AppConfig Validation`	`constructParam.groovy:validateAppConfig()`

Failure mode taxonomy

The skill classifies every failure into one of these categories before diagnosis:

Category	Signal in log	devops-lib location
Build compilation	`BUILD FAILURE` / `go build failed`	build.groovy
Docker push to GAR	`denied` / `UNAUTHORIZED` on push	build.groovy:dockerBuildAndPush()
Sonar threshold	`Quality Gate status: FAILED`	constructParam.groovy:waitForQualityGate()
Sonar project missing	`Project not found`	buildGo.groovy:sonar_scan() auto-create logic
ArgoCD step 1 (argo_repo update)	`git push failed` in argo-config repo	deployArgoCD.groovy:update_argo_repo()
ArgoCD step 2 (app-of-apps refresh)	`app not found` / `no Application object`	deployArgoCD.groovy:refresh_app_of_apps() — new service, ArgoCD app not yet created
ArgoCD step 3 (helm push)	`chart version already exists`	deployArgoCD.groovy:update_helm_repo()
ArgoCD step 4 (sync)	`OutOfSync` / `Helm values error`	deployArgoCD.groovy:refresh_and_sync()
Wrong deployment_order	Silent sync on wrong app name	config.yaml:deployment_order vs devops-argo-config
Multizone gate	`must be deployed via Ringmaster`	constructParam.groovy:isMultizoneEnabled()

Dependencies

Full read access to devops-lib source (stage-to-file mapping, failure message strings)
Jenkins log (pasted by engineer or fetched via Jenkins API)
Optionally: Meesho/whitelists (to check if skip_sonar or multizone entries explain the failure)
Optionally: Meesho/devops-argo-config (to validate deployment_order app names for ArgoCD failures)

Design notes

The skill must handle truncated logs (Jenkins UI often shows the last N lines). It should ask for the full log if the failure point is not visible.
For hotfix builds: sonar and quality gate failures are expected to be suppressed; if they appear, the hotfix path was not taken — the skill should check the branch name pattern.
The skill should distinguish between a devops-lib bug (code path is wrong) vs a configuration error (wrong value in config.yaml or whitelist) vs an external system error (GAR down, Sonar unreachable).

Open questions

Should it auto-fetch the Jenkins log via the Jenkins API if given a job URL?
For ArgoCD step 4 failures (Helm values schema errors): should it parse the exact Helm error and cross-reference with the deployment.yaml schema?
Should it suggest a rerun command or ArgoCD force-sync as a recovery action?

6.5 KiB Raw Blame History