Issue public certificates for the real domain

The deployment has a domain of its own now, which is the first time a
public certificate has been possible here at all. The registry issuer
beside this one explains why: nip.io is not on the public suffix list and
every *.nip.io certificate shares one rate limit, so Let's Encrypt could
never serve the addresses this cluster has been using.

DNS-01 rather than HTTP-01, because every deployed app lives at
<app>.apps.<domain> and only a DNS-01 challenge can issue the wildcard
that covers all of them. HTTP-01 would mean a certificate per app,
requested the moment each one is created.

Two issuers, staging and production. Production allows five duplicate
certificates a week and a failing solver spends that allowance without
issuing anything, which can lock a domain out of certificates for days.
Staging has no meaningful limit and issues from an untrusted root, so a
browser warning is the signal that the plumbing works. Move to the
production issuer once a staging certificate appears.

The token reaches cert-manager the same way every other credential here
does: Vault, through External Secrets. It wants Zone -> DNS -> Edit on the
one zone and nothing else — enough to write the TXT record a challenge
needs, and no more. Until `vault kv put secret/cloudflare/dns-token` has
run, the ExternalSecret stays unfulfilled and the issuers cannot register.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N

extra-manifests/letsencrypt-clusterissuer.yaml

@@ -0,0 +1,72 @@
+# The issuer for this deployment's public certificates, from Let's Encrypt.
+#
+# Two of them: staging first, production second. Let's Encrypt's production
+# endpoint allows five duplicate certificates per week and a failed solver
+# burns that allowance without issuing anything, so a misconfiguration can
+# lock the real domain out of certificates for days. The staging endpoint
+# has no meaningful limit and issues from an untrusted root — a browser will
+# warn, which is exactly what proves the plumbing works before anything
+# depends on it.
+#
+# DNS-01, not HTTP-01, because every deployed app lives at
+# <app>.apps.<domain> and only a DNS-01 challenge can issue the wildcard
+# that covers all of them. HTTP-01 would need a certificate per app,
+# requested the moment each one is created.
+#
+# This is what a real domain buys. The registry issuer beside this file
+# explains why nip.io could never have it: nip.io is not on the public
+# suffix list, and every *.nip.io certificate shares one rate limit.
+#
+# BEFORE THIS WORKS, three things must be true:
+#
+#  1. The zone's records point straight at the load balancer, NOT through
+#     Cloudflare's proxy. A proxied record answers from Cloudflare's own
+#     addresses, so the certificate would protect traffic that never
+#     reaches this cluster.
+#  2. The `cloudflare-api-token` Secret exists in the cert-manager
+#     namespace, created by Terraform from var.cloudflare_api_token. Its
+#     key must be `api-token`; cert-manager reports a mismatch only when a
+#     challenge fails, long after everything else looked fine.
+#  3. The email below is filled in. Let's Encrypt requires one for expiry
+#     notices, and leaving the placeholder makes registration fail.
+---
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-staging
+spec:
+  acme:
+    server: https://acme-staging-v02.api.letsencrypt.org/directory
+    # REPLACE ME — Let's Encrypt registers this address and sends expiry
+    # warnings to it. It is given to a third party, so it is deliberately
+    # not filled in from anyone's account details.
+    email: mukul.sharma909.ms@gmail.com
+    privateKeySecretRef:
+      # cert-manager's own ACME account key, which it creates. Nothing
+      # supplies this; it must differ between the two issuers or they share
+      # an account registration across two different endpoints.
+      name: letsencrypt-staging-account-key
+    solvers:
+      - dns01:
+          cloudflare:
+            apiTokenSecretRef:
+              name: cloudflare-api-token
+              key: api-token
+---
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-prod
+spec:
+  acme:
+    server: https://acme-v02.api.letsencrypt.org/directory
+    # REPLACE ME — see above.
+    email: mukul.sharma909.ms@gmail.com
+    privateKeySecretRef:
+      name: letsencrypt-prod-account-key
+    solvers:
+      - dns01:
+          cloudflare:
+            apiTokenSecretRef:
+              name: cloudflare-api-token
+              key: api-token

secretstores/cloudflare-dns-token.yaml

@@ -0,0 +1,43 @@
+# The Cloudflare API token cert-manager answers DNS-01 challenges with.
+#
+# DNS-01 rather than HTTP-01 because every deployed app lives at
+# <app>.apps.<domain>, and only a DNS-01 challenge can issue the wildcard
+# that covers all of them at once. HTTP-01 would mean a certificate per app,
+# requested the moment each one is created.
+#
+# The token wants Zone -> DNS -> Edit on the one zone and nothing else. It
+# can create and delete TXT records in that zone, which is all the challenge
+# needs; anything wider is a credential in a cluster that did not have to be.
+#
+# Put the value in Vault first — this only copies it, and an ExternalSecret
+# pointing at a path that does not exist stays unfulfilled with the Secret
+# never created:
+#
+#   vault kv put secret/cloudflare/dns-token token='<the token>'
+#
+# The key below MUST stay "api-token": letsencrypt-clusterissuer.yaml in
+# extra-manifests/ names it in apiTokenSecretRef, and cert-manager reports a
+# mismatch only when a challenge fails — long after everything else looked
+# like it had applied cleanly.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: cloudflare-dns-token
+  # cert-manager's own namespace, because a ClusterIssuer always reads its
+  # secrets from there regardless of which namespace asked for the
+  # certificate. That is what lets one issuer serve every namespace without
+  # the token being copied into any of them.
+  namespace: cert-manager
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: vault-backend
+    kind: ClusterSecretStore
+  target:
+    name: cloudflare-api-token
+    creationPolicy: Owner
+  data:
+    - secretKey: api-token
+      remoteRef:
+        key: cloudflare/dns-token
+        property: token