# kubectl-mcp-server

Read-only Kubernetes RCA (root cause analysis) MCP server. A diagnosis-only
tool surface for an RCA bot / AI client (Bifrost) — it can **never** mutate
cluster state. The image entrypoint is `k8s-rca-mcp`, configured **entirely
via env vars** (no CLI args), and exposes only an allowlisted read-only
toolset over streamable-http.

> The chart/release keeps the name `kubectl-mcp-server` for continuity with
> the existing ArgoCD apps and overrides. The underlying server is the
> in-house env-driven RCA build (`prd/.../kubectl-mcp-server:v2`).

## Two modes (`mcp.mode`)

| | `single` | `multi` (hub) |
|---|---|---|
| Scope | one cluster, in-cluster SA | federation router |
| `cluster` tool param | none | **required** on every tool |
| Deploy where | every cluster | one hub |
| Credentials | own ServiceAccount | none for remotes — forwards MCP calls with a per-backend bearer token |

In `multi` mode every tool gains a required `cluster` argument. `cluster == self`
is served natively via the hub's in-cluster SA; any other registered name is
resolved to a DNS endpoint and the identical tool call is forwarded over
streamable-http to that cluster's `single`-mode instance. The hub holds **no
remote kubeconfigs** — only endpoints and bearer tokens.

## Authentication

Inbound: the MCP endpoint requires a bearer token. On http transport the
server **refuses to start** unless `MCP_AUTH_TOKEN` is present (or
`auth.allowAnonymous: true`). Clients send `Authorization: Bearer <token>`.

All bearer tokens (the inbound `MCP_AUTH_TOKEN` and, on a hub, every backend
token) are stored in **Vault** and synced via an `ExternalSecret` — never in
values or git. One ExternalSecret extracts a Vault path into the Secret
`kubectl-mcp-server-creds`, which the deployment injects with `envFrom`, so
each Vault key becomes a container env var verbatim.

```yaml
externalSecrets:
  enabled: true
  refreshInterval: "150s"
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  dataFrom:
    secretKey: "meesho/stg/cntr/devop/kubectl-mcp-server"   # Vault path (configurable)
```

Store these **keys** at that Vault path (key names map 1:1 to env vars, so
they must be valid env var names):

| Vault key | Meaning | When |
|---|---|---|
| `MCP_AUTH_TOKEN` | inbound bearer the RCA agent must present | always |
| `<TOKENENV>` per backend, e.g. `SUPPLY_MCP_TOKEN` | bearer the hub uses to call that backend | multi mode |

## Single-mode deploy

Per cluster, set the Vault path and (optionally) the ingress host. Example
override (`helm-overrides/<cluster>/kubectl-mcp-server/custom-values.yaml`):

```yaml
mcp:
  mode: single
  transport: http
  host: "0.0.0.0"
  port: 8000

auth:
  allowAnonymous: false

externalSecrets:
  enabled: true
  dataFrom:
    secretKey: "meesho/stg/cntr/devop/kubectl-mcp-server"   # must hold MCP_AUTH_TOKEN
```

Seed the Vault path:

```bash
vault kv put meesho/stg/cntr/devop/kubectl-mcp-server \
  MCP_AUTH_TOKEN=$(openssl rand -hex 32)
```

## Multi-mode (hub): adding a cluster + its token

Adding a backend is exactly two coordinated steps — a **registry entry** that
names an env var, and a **Vault key of that same name** holding the value.

### 1. Add the cluster to the registry (`federation.clusters`)

In the hub's override:

```yaml
mcp:
  mode: multi

externalSecrets:
  enabled: true
  dataFrom:
    secretKey: "meesho/stg/cntr/devop/kubectl-mcp-server-hub"

federation:
  forwardTimeout: 30
  clusters:
    - name: self                  # the hub's own cluster, served natively
      self: true
    - name: supply                # the name the RCA bot passes as cluster=
      endpoint: https://kubectl-mcp-server.supply.stg.meesho.int/mcp   # must end in /mcp
      tokenEnv: SUPPLY_MCP_TOKEN   # env var the hub reads this backend's bearer from
    - name: payments
      endpoint: https://kubectl-mcp-server.payments.stg.meesho.int/mcp
      tokenEnv: PAYMENTS_MCP_TOKEN
```

`tokenEnv` is a name you choose (convention: `<CLUSTER>_MCP_TOKEN`). It is the
env var the hub looks up when forwarding to that cluster.

### 2. Add the matching token to Vault

Add the backend tokens as keys at the hub's Vault path, named **exactly** like
each `tokenEnv`:

```bash
vault kv put meesho/stg/cntr/devop/kubectl-mcp-server-hub \
  MCP_AUTH_TOKEN=$(openssl rand -hex 32) \
  SUPPLY_MCP_TOKEN=<the supply cluster's own MCP_AUTH_TOKEN> \
  PAYMENTS_MCP_TOKEN=<the payments cluster's own MCP_AUTH_TOKEN>
```

> **Critical:** the value of `SUPPLY_MCP_TOKEN` must equal the
> `MCP_AUTH_TOKEN` that the **supply cluster's own** single-mode deployment
> uses (from *its* Vault path). That is how the hub authenticates to it.

Nothing else changes — `envFrom: secretRef` injects every Vault key
automatically, so the new `*_MCP_TOKEN` env var just appears in the container.
**Adding a cluster = one registry entry + one Vault key of the same name.**

### Request flow for one backend

```
RCA bot calls a tool with cluster="supply"
        │
        ▼
hub registry: supply → endpoint + tokenEnv=SUPPLY_MCP_TOKEN
        │                         └── value comes from Vault key SUPPLY_MCP_TOKEN (via envFrom)
        ▼
hub POSTs to https://...supply.../mcp  with  Authorization: Bearer <SUPPLY_MCP_TOKEN>
        ▼
supply single-mode pod checks it == its own MCP_AUTH_TOKEN  ✓  → serves the tool
result returned tagged with cluster="supply"
```

Structured errors (never silent): missing/empty `cluster` → `missing_cluster`;
unknown name → `unknown_cluster` (both list available clusters); backend down
or slow → `backend_unreachable` / `backend_timeout`, tagged with the cluster.

## Key values

| Key | Default | Notes |
|---|---|---|
| `mcp.mode` | `single` | `single` or `multi` |
| `mcp.transport` | `http` | streamable-http |
| `mcp.port` | `8000` | matches probes + service |
| `auth.allowAnonymous` | `false` | skip the bearer (testing only) |
| `externalSecrets.enabled` | `true` | sync tokens from Vault |
| `externalSecrets.dataFrom.secretKey` | `""` | **Vault path (set per cluster)** |
| `externalSecrets.secretStoreRef.name` | `vault-backend` | ClusterSecretStore |
| `externalSecrets.refreshInterval` | `150s` | |
| `envFrom` | `[]` | extra envFrom sources (non-Vault testing) |
| `federation.forwardTimeout` | `30` | seconds per forwarded call (multi) |
| `federation.clusters` | `[{name: self, self: true}]` | registry (multi) |

RBAC: a read-only ClusterRole (`get/list/watch` only, `secrets` excluded) is
created for the pod ServiceAccount — the enforcement layer behind the server's
own deny-by-default tool allowlist.