fix argo health check

2026-08-31 06:43:23 +05:30
parent 69c2807fd7
commit 775c2ef79e
4 changed files with 149 additions and 0 deletions
@@ -0,0 +1,131 @@
+# argo-cd — bootstrap runbook
+
+Thin wrapper chart (see `Chart.yaml` — dependency on `argo/argo-cd`). This
+README covers a full reset and clean bootstrap of ArgoCD itself on the
+homelab VM (cluster `k8s-admin-prd-ase1`, namespace `argocd`), end to end.
+ArgoCD is the one thing in this repo that can't bootstrap itself via
+GitOps — it has to exist before it can manage anything, including itself —
+so every step here is a manual, imperative command run once.
+
+## 0. Full uninstall (only if resetting)
+
+```bash
+helm uninstall argocd-admin-prd -n argocd
+kubectl delete namespace argocd
+```
+
+`helm uninstall` removes everything the release owns, including the
+`Application`/`AppProject`/`ApplicationSet` CRDs (this chart installs them
+as plain templates, not as a separate hook that survives uninstall).
+Deleting the namespace on top cleans up anything left behind (PVCs for the
+bundled Redis, stray ConfigMaps/Secrets created outside the Helm release,
+etc.). Confirm it's actually clean before moving on:
+
+```bash
+kubectl get all -n argocd            # should error "not found" once the ns is gone
+kubectl get crd | grep argoproj.io   # should return nothing
+```
+
+## 1. Pull the real chart
+
+This wrapper's `Chart.yaml` only declares a dependency — the actual
+`argo-cd` chart isn't vendored, so pull it first:
+
+```bash
+helm repo add argo https://argoproj.github.io/argo-helm
+helm repo update
+helm dependency update helm-templates/argo-cd   # run from the repo root
+```
+
+## 2. Install
+
+```bash
+kubectl create namespace argocd --dry-run=client -o yaml | kubectl apply -f -
+
+helm upgrade --install argocd-admin-prd helm-templates/argo-cd \
+  --namespace argocd \
+  -f helm-overrides/k8s-admin-prd-ase1/argocd-admin-prd/custom-values.yaml
+
+kubectl -n argocd rollout status deploy -l app.kubernetes.io/component=server --timeout=180s
+```
+
+`argocd-admin-prd` is the release name — it has to match the `nameOverride`
+in `devops-infra-argo-config`'s
+`values/incubator-infra-k8s-admin-prd-ase1-values.yaml` `appSpec` entry for
+`argocd`, since once GitOps sync takes over, Argo renders using that same
+name.
+
+## 3. Get in
+
+```bash
+kubectl -n argocd get secret argocd-initial-admin-secret \
+  -o jsonpath='{.data.password}' | base64 -d; echo
+```
+
+Username is always `admin`. This secret only gets created by a
+**first-install** Helm hook — it won't reappear on a later `helm upgrade`
+against an existing release, only right after step 2 on a genuinely fresh
+install (which is why step 0's full uninstall matters if you're trying to
+get back to a known state rather than just patching forward).
+
+If it's missing later and you're locked out:
+```bash
+kubectl -n argocd exec -it deploy/argocd-server -- argocd admin initial-password -n argocd
+# or, as a last resort, set it directly:
+kubectl -n argocd exec -it deploy/argocd-server -- argocd account bcrypt --password '<new-password>'
+kubectl -n argocd patch secret argocd-secret \
+  -p '{"stringData": {"admin.password": "<hash-from-above>", "admin.passwordMtime": "'$(date +%FT%T%Z)'"}}'
+```
+
+Reach the UI at `argocd.192.168.1.7.nip.io` (Contour ingress, plain HTTP —
+`--insecure` is set in `custom-values.yaml` on purpose, see that file).
+
+## 4. Give it repos to sync from
+
+Push `devops-infra-argo-config` and `devops-infra-helm-charts` to Gitea. If
+either repo isn't anonymous-read, ArgoCD needs a credential for **both** —
+the incubator Application sources `devops-infra-argo-config` directly, and
+every child Application's `$values` ref sources `devops-infra-helm-charts`.
+See `devops-infra-argo-config/gitea-repo-credentials.example.yaml` for the
+Secret template (copy it, fill in a real Gitea access token, `kubectl
+apply`, never commit the filled-in version).
+
+## 5. Hand control to GitOps
+
+From `devops-infra-argo-config`:
+
+```bash
+kubectl apply -f app-of-projects.yaml                              # creates the "devops" AppProject
+kubectl apply -f incubator/incubator-infra-k8s-admin-prd-ase1.yaml  # renders argocd + gitea Applications
+```
+
+Check it landed:
+
+```bash
+kubectl -n argocd get applications
+```
+
+Expect to see `app-of-projects`, `incubator-infra-k8s-admin-prd-ase1`,
+`argocd-admin-prd`, and `gitea`. Everything in this repo is **manual-sync
+only** on purpose (no app has an `automated:` syncPolicy) — click Sync in
+the UI (or `argocd app sync <name>` via CLI) rather than expecting anything
+to reconcile on its own. `argocd-admin-prd` going green means ArgoCD is now
+managing its own Helm release via Git instead of the imperative commands
+above — from here on, values changes go through
+`helm-overrides/k8s-admin-prd-ase1/argocd-admin-prd/custom-values.yaml`,
+not a manual `helm upgrade`.
+
+## Gotchas hit getting here (don't relitigate these)
+
+- **`spec.project: devops` on child Applications requires the `devops`
+  AppProject to exist first** — that's what step 5's `app-of-projects.yaml`
+  apply is for. Skipping it means `gitea`'s (and any future app's)
+  Application object exists in Git/gets rendered but fails to validate.
+- **Deleting a ConfigMap this chart owns doesn't self-heal** — same
+  manual-sync-only reasoning as above. Re-running step 2's `helm upgrade
+  --install` recreates anything missing from the release; it just won't
+  regenerate `argocd-initial-admin-secret` (see step 3).
+- Gitea-specific adoption issues (the `value`→`valueFrom` env var
+  migration, the single-node `RollingUpdate` LevelDB lock collision) are
+  documented in `helm-overrides/k8s-admin-prd-ase1/gitea/custom-values.yaml`
+  instead — not an ArgoCD problem, don't duplicate the notes here.