179 lines
6.5 KiB
Markdown
179 lines
6.5 KiB
Markdown
# kubectl-mcp-server
|
|
|
|
Read-only Kubernetes RCA (root cause analysis) MCP server. A diagnosis-only
|
|
tool surface for an RCA bot / AI client (Bifrost) — it can **never** mutate
|
|
cluster state. The image entrypoint is `k8s-rca-mcp`, configured **entirely
|
|
via env vars** (no CLI args), and exposes only an allowlisted read-only
|
|
toolset over streamable-http.
|
|
|
|
> The chart/release keeps the name `kubectl-mcp-server` for continuity with
|
|
> the existing ArgoCD apps and overrides. The underlying server is the
|
|
> in-house env-driven RCA build (`prd/.../kubectl-mcp-server:v2`).
|
|
|
|
## Two modes (`mcp.mode`)
|
|
|
|
| | `single` | `multi` (hub) |
|
|
|---|---|---|
|
|
| Scope | one cluster, in-cluster SA | federation router |
|
|
| `cluster` tool param | none | **required** on every tool |
|
|
| Deploy where | every cluster | one hub |
|
|
| Credentials | own ServiceAccount | none for remotes — forwards MCP calls with a per-backend bearer token |
|
|
|
|
In `multi` mode every tool gains a required `cluster` argument. `cluster == self`
|
|
is served natively via the hub's in-cluster SA; any other registered name is
|
|
resolved to a DNS endpoint and the identical tool call is forwarded over
|
|
streamable-http to that cluster's `single`-mode instance. The hub holds **no
|
|
remote kubeconfigs** — only endpoints and bearer tokens.
|
|
|
|
## Authentication
|
|
|
|
Inbound: the MCP endpoint requires a bearer token. On http transport the
|
|
server **refuses to start** unless `MCP_AUTH_TOKEN` is present (or
|
|
`auth.allowAnonymous: true`). Clients send `Authorization: Bearer <token>`.
|
|
|
|
All bearer tokens (the inbound `MCP_AUTH_TOKEN` and, on a hub, every backend
|
|
token) are stored in **Vault** and synced via an `ExternalSecret` — never in
|
|
values or git. One ExternalSecret extracts a Vault path into the Secret
|
|
`kubectl-mcp-server-creds`, which the deployment injects with `envFrom`, so
|
|
each Vault key becomes a container env var verbatim.
|
|
|
|
```yaml
|
|
externalSecrets:
|
|
enabled: true
|
|
refreshInterval: "150s"
|
|
secretStoreRef:
|
|
name: vault-backend
|
|
kind: ClusterSecretStore
|
|
dataFrom:
|
|
secretKey: "meesho/stg/cntr/devop/kubectl-mcp-server" # Vault path (configurable)
|
|
```
|
|
|
|
Store these **keys** at that Vault path (key names map 1:1 to env vars, so
|
|
they must be valid env var names):
|
|
|
|
| Vault key | Meaning | When |
|
|
|---|---|---|
|
|
| `MCP_AUTH_TOKEN` | inbound bearer the RCA agent must present | always |
|
|
| `<TOKENENV>` per backend, e.g. `SUPPLY_MCP_TOKEN` | bearer the hub uses to call that backend | multi mode |
|
|
|
|
## Single-mode deploy
|
|
|
|
Per cluster, set the Vault path and (optionally) the ingress host. Example
|
|
override (`helm-overrides/<cluster>/kubectl-mcp-server/custom-values.yaml`):
|
|
|
|
```yaml
|
|
mcp:
|
|
mode: single
|
|
transport: http
|
|
host: "0.0.0.0"
|
|
port: 8000
|
|
|
|
auth:
|
|
allowAnonymous: false
|
|
|
|
externalSecrets:
|
|
enabled: true
|
|
dataFrom:
|
|
secretKey: "meesho/stg/cntr/devop/kubectl-mcp-server" # must hold MCP_AUTH_TOKEN
|
|
```
|
|
|
|
Seed the Vault path:
|
|
|
|
```bash
|
|
vault kv put meesho/stg/cntr/devop/kubectl-mcp-server \
|
|
MCP_AUTH_TOKEN=$(openssl rand -hex 32)
|
|
```
|
|
|
|
## Multi-mode (hub): adding a cluster + its token
|
|
|
|
Adding a backend is exactly two coordinated steps — a **registry entry** that
|
|
names an env var, and a **Vault key of that same name** holding the value.
|
|
|
|
### 1. Add the cluster to the registry (`federation.clusters`)
|
|
|
|
In the hub's override:
|
|
|
|
```yaml
|
|
mcp:
|
|
mode: multi
|
|
|
|
externalSecrets:
|
|
enabled: true
|
|
dataFrom:
|
|
secretKey: "meesho/stg/cntr/devop/kubectl-mcp-server-hub"
|
|
|
|
federation:
|
|
forwardTimeout: 30
|
|
clusters:
|
|
- name: self # the hub's own cluster, served natively
|
|
self: true
|
|
- name: supply # the name the RCA bot passes as cluster=
|
|
endpoint: https://kubectl-mcp-server.supply.stg.meesho.int/mcp # must end in /mcp
|
|
tokenEnv: SUPPLY_MCP_TOKEN # env var the hub reads this backend's bearer from
|
|
- name: payments
|
|
endpoint: https://kubectl-mcp-server.payments.stg.meesho.int/mcp
|
|
tokenEnv: PAYMENTS_MCP_TOKEN
|
|
```
|
|
|
|
`tokenEnv` is a name you choose (convention: `<CLUSTER>_MCP_TOKEN`). It is the
|
|
env var the hub looks up when forwarding to that cluster.
|
|
|
|
### 2. Add the matching token to Vault
|
|
|
|
Add the backend tokens as keys at the hub's Vault path, named **exactly** like
|
|
each `tokenEnv`:
|
|
|
|
```bash
|
|
vault kv put meesho/stg/cntr/devop/kubectl-mcp-server-hub \
|
|
MCP_AUTH_TOKEN=$(openssl rand -hex 32) \
|
|
SUPPLY_MCP_TOKEN=<the supply cluster's own MCP_AUTH_TOKEN> \
|
|
PAYMENTS_MCP_TOKEN=<the payments cluster's own MCP_AUTH_TOKEN>
|
|
```
|
|
|
|
> **Critical:** the value of `SUPPLY_MCP_TOKEN` must equal the
|
|
> `MCP_AUTH_TOKEN` that the **supply cluster's own** single-mode deployment
|
|
> uses (from *its* Vault path). That is how the hub authenticates to it.
|
|
|
|
Nothing else changes — `envFrom: secretRef` injects every Vault key
|
|
automatically, so the new `*_MCP_TOKEN` env var just appears in the container.
|
|
**Adding a cluster = one registry entry + one Vault key of the same name.**
|
|
|
|
### Request flow for one backend
|
|
|
|
```
|
|
RCA bot calls a tool with cluster="supply"
|
|
│
|
|
▼
|
|
hub registry: supply → endpoint + tokenEnv=SUPPLY_MCP_TOKEN
|
|
│ └── value comes from Vault key SUPPLY_MCP_TOKEN (via envFrom)
|
|
▼
|
|
hub POSTs to https://...supply.../mcp with Authorization: Bearer <SUPPLY_MCP_TOKEN>
|
|
▼
|
|
supply single-mode pod checks it == its own MCP_AUTH_TOKEN ✓ → serves the tool
|
|
result returned tagged with cluster="supply"
|
|
```
|
|
|
|
Structured errors (never silent): missing/empty `cluster` → `missing_cluster`;
|
|
unknown name → `unknown_cluster` (both list available clusters); backend down
|
|
or slow → `backend_unreachable` / `backend_timeout`, tagged with the cluster.
|
|
|
|
## Key values
|
|
|
|
| Key | Default | Notes |
|
|
|---|---|---|
|
|
| `mcp.mode` | `single` | `single` or `multi` |
|
|
| `mcp.transport` | `http` | streamable-http |
|
|
| `mcp.port` | `8000` | matches probes + service |
|
|
| `auth.allowAnonymous` | `false` | skip the bearer (testing only) |
|
|
| `externalSecrets.enabled` | `true` | sync tokens from Vault |
|
|
| `externalSecrets.dataFrom.secretKey` | `""` | **Vault path (set per cluster)** |
|
|
| `externalSecrets.secretStoreRef.name` | `vault-backend` | ClusterSecretStore |
|
|
| `externalSecrets.refreshInterval` | `150s` | |
|
|
| `envFrom` | `[]` | extra envFrom sources (non-Vault testing) |
|
|
| `federation.forwardTimeout` | `30` | seconds per forwarded call (multi) |
|
|
| `federation.clusters` | `[{name: self, self: true}]` | registry (multi) |
|
|
|
|
RBAC: a read-only ClusterRole (`get/list/watch` only, `secrets` excluded) is
|
|
created for the pod ServiceAccount — the enforcement layer behind the server's
|
|
own deny-by-default tool allowlist.
|