Read-only Kubernetes RCA (root cause analysis) MCP server. A diagnosis-only tool surface for an RCA bot / AI client (Bifrost) — it can never mutate cluster state. The image entrypoint is k8s-rca-mcp, configured entirely via env vars (no CLI args), and exposes only an allowlisted read-only toolset over streamable-http.

The chart/release keeps the name kubectl-mcp-server for continuity with the existing ArgoCD apps and overrides. The underlying server is the in-house env-driven RCA build (prd/.../kubectl-mcp-server:v2).

Two modes (`mcp.mode`)

	`single`	`multi` (hub)
Scope	one cluster, in-cluster SA	federation router
`cluster` tool param	none	required on every tool
Deploy where	every cluster	one hub
Credentials	own ServiceAccount	none for remotes — forwards MCP calls with a per-backend bearer token

In multi mode every tool gains a required cluster argument. cluster == self is served natively via the hub's in-cluster SA; any other registered name is resolved to a DNS endpoint and the identical tool call is forwarded over streamable-http to that cluster's single-mode instance. The hub holds no remote kubeconfigs — only endpoints and bearer tokens.

Authentication

Inbound: the MCP endpoint requires a bearer token. On http transport the server refuses to start unless MCP_AUTH_TOKEN is present (or auth.allowAnonymous: true). Clients send Authorization: Bearer <token>.

All bearer tokens (the inbound MCP_AUTH_TOKEN and, on a hub, every backend token) are stored in Vault and synced via an ExternalSecret — never in values or git. One ExternalSecret extracts a Vault path into the Secret kubectl-mcp-server-creds, which the deployment injects with envFrom, so each Vault key becomes a container env var verbatim.

externalSecrets:
  enabled: true
  refreshInterval: "150s"
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  dataFrom:
    secretKey: "meesho/stg/cntr/devop/kubectl-mcp-server"   # Vault path (configurable)

Store these keys at that Vault path (key names map 1:1 to env vars, so they must be valid env var names):

Vault key	Meaning	When
`MCP_AUTH_TOKEN`	inbound bearer the RCA agent must present	always
`<TOKENENV>` per backend, e.g. `SUPPLY_MCP_TOKEN`	bearer the hub uses to call that backend	multi mode

Single-mode deploy

Per cluster, set the Vault path and (optionally) the ingress host. Example override (helm-overrides/<cluster>/kubectl-mcp-server/custom-values.yaml):

mcp:
  mode: single
  transport: http
  host: "0.0.0.0"
  port: 8000

auth:
  allowAnonymous: false

externalSecrets:
  enabled: true
  dataFrom:
    secretKey: "meesho/stg/cntr/devop/kubectl-mcp-server"   # must hold MCP_AUTH_TOKEN

Seed the Vault path:

vault kv put meesho/stg/cntr/devop/kubectl-mcp-server \
  MCP_AUTH_TOKEN=$(openssl rand -hex 32)

Multi-mode (hub): adding a cluster + its token

Adding a backend is exactly two coordinated steps — a registry entry that names an env var, and a Vault key of that same name holding the value.

1. Add the cluster to the registry (`federation.clusters`)

In the hub's override:

mcp:
  mode: multi

externalSecrets:
  enabled: true
  dataFrom:
    secretKey: "meesho/stg/cntr/devop/kubectl-mcp-server-hub"

federation:
  forwardTimeout: 30
  clusters:
    - name: self                  # the hub's own cluster, served natively
      self: true
    - name: supply                # the name the RCA bot passes as cluster=
      endpoint: https://kubectl-mcp-server.supply.stg.meesho.int/mcp   # must end in /mcp
      tokenEnv: SUPPLY_MCP_TOKEN   # env var the hub reads this backend's bearer from
    - name: payments
      endpoint: https://kubectl-mcp-server.payments.stg.meesho.int/mcp
      tokenEnv: PAYMENTS_MCP_TOKEN

tokenEnv is a name you choose (convention: <CLUSTER>_MCP_TOKEN). It is the env var the hub looks up when forwarding to that cluster.

2. Add the matching token to Vault

Add the backend tokens as keys at the hub's Vault path, named exactly like each tokenEnv:

vault kv put meesho/stg/cntr/devop/kubectl-mcp-server-hub \
  MCP_AUTH_TOKEN=$(openssl rand -hex 32) \
  SUPPLY_MCP_TOKEN=<the supply cluster's own MCP_AUTH_TOKEN> \
  PAYMENTS_MCP_TOKEN=<the payments cluster's own MCP_AUTH_TOKEN>

Critical: the value of SUPPLY_MCP_TOKEN must equal the MCP_AUTH_TOKEN that the supply cluster's own single-mode deployment uses (from its Vault path). That is how the hub authenticates to it.

Nothing else changes — envFrom: secretRef injects every Vault key automatically, so the new *_MCP_TOKEN env var just appears in the container. Adding a cluster = one registry entry + one Vault key of the same name.

Request flow for one backend

RCA bot calls a tool with cluster="supply"
        │
        ▼
hub registry: supply → endpoint + tokenEnv=SUPPLY_MCP_TOKEN
        │                         └── value comes from Vault key SUPPLY_MCP_TOKEN (via envFrom)
        ▼
hub POSTs to https://...supply.../mcp  with  Authorization: Bearer <SUPPLY_MCP_TOKEN>
        ▼
supply single-mode pod checks it == its own MCP_AUTH_TOKEN  ✓  → serves the tool
result returned tagged with cluster="supply"

Structured errors (never silent): missing/empty cluster → missing_cluster; unknown name → unknown_cluster (both list available clusters); backend down or slow → backend_unreachable / backend_timeout, tagged with the cluster.

Key values

Key	Default	Notes
`mcp.mode`	`single`	`single` or `multi`
`mcp.transport`	`http`	streamable-http
`mcp.port`	`8000`	matches probes + service
`auth.allowAnonymous`	`false`	skip the bearer (testing only)
`externalSecrets.enabled`	`true`	sync tokens from Vault
`externalSecrets.dataFrom.secretKey`	`""`	Vault path (set per cluster)
`externalSecrets.secretStoreRef.name`	`vault-backend`	ClusterSecretStore
`externalSecrets.refreshInterval`	`150s`
`envFrom`	`[]`	extra envFrom sources (non-Vault testing)
`federation.forwardTimeout`	`30`	seconds per forwarded call (multi)
`federation.clusters`	`[{name: self, self: true}]`	registry (multi)

RBAC: a read-only ClusterRole (get/list/watch only, secrets excluded) is created for the pod ServiceAccount — the enforcement layer behind the server's own deny-by-default tool allowlist.

README.md

kubectl-mcp-server

Two modes (mcp.mode)