t1k:grafana
| Field | Value |
|---|---|
| Module | t1k-devops |
| Version | 4.1.3 |
| Effort | low |
| Tools | — |
Keywords: alertmanager, cloudflare access grafana, dashboard configmap, dashboard json, datasource provisioning, discord alert, GF_ env, grafana, grafana docker compose, grafana http api, grafana mcp, grafana-deploy, metrics endpoint, observability, prometheus, prometheusrule, provision dashboard, pushgateway, scrape config, service account token, servicemonitor, studio grafana
How to invoke
Section titled “How to invoke”/t1k:grafanainstrument | register | dashboard | alert | operate | self-hostStudio Grafana — observability for everything we build
Section titled “Studio Grafana — observability for everything we build”The studio runs one Grafana. Every self-built service ships its metrics, dashboards and alerts there — never to a per-project Grafana, a hosted Datadog/Honeycomb, or a sidecar.
| What | Where (verified 2026-09-26) |
|---|---|
| Instance | k3s on theone, namespace grafana, Helm chart grafana/grafana (pinned in the deploy repo), persistence off |
| Its source of truth | The1Studio/grafana-deploy (private): Helm values, dashboards, CI |
| Metrics + alerting | kube-prometheus-stack in namespace monitoring, manifests in The1Studio/arc-github-runners k8s/monitoring/ |
| Location / URL / dashboard SSOT | docs/grafana-node-monitoring.md in The1Studio/arc-github-runners |
Read the SSOT doc for the live URL rather than hardcoding one from anywhere else, including this
file. Normative rule: studio-grafana.md in this module. This skill is the how.
Decision Tree
Section titled “Decision Tree”| Intent | Read | Deliverable |
|---|---|---|
| “add metrics to a service” | instrument | /metrics endpoint (or Pushgateway push) in the service repo |
| “register the service / add a datasource” | register | ServiceMonitor in arc-github-runners, or datasource in grafana-deploy |
| “add a dashboard” | dashboard | PR to The1Studio/grafana-deploy |
| “alert me on Discord when X” | alert | PrometheusRule in arc-github-runners k8s/monitoring/ |
| “deploy / roll back / log in / connect an MCP” | operate | a grafana-deploy PR, or a documented command |
| “stand up a Grafana that is NOT the studio one” | self-host | compose + provisioned dirs |
Every step is a file in a repo and a git commit. If you are about to click in the UI, stop — the studio instance has no persistence: a UI-built dashboard, user or setting is gone on the next pod restart and on the next deploy.
Step 0 — read the instance’s own docs first
Section titled “Step 0 — read the instance’s own docs first”G=".claude/scripts/t1k-gh.cjs"; [ -f "$G" ] || G="$HOME/.claude/scripts/t1k-gh.cjs"bun "$G" get repos/The1Studio/grafana-deploy/contents/README.md --pick content | base64 -d | sed -n '1,80p'bun "$G" get repos/The1Studio/arc-github-runners/contents/docs/grafana-node-monitoring.md \ --pick content | base64 -d | sed -n '1,60p'The grafana-deploy README is the operating manual (layout, how a change ships, RBAC consequences); the arc doc is the SSOT for URL and dashboard inventory. Quote them, do not paraphrase a URL.
Auth for every API call below is a service-account token in an env var — never a literal, never
argv (world-readable in ps). See api-and-service-accounts.md (references/api-and-service-accounts.md).
1 — Instrument the service
Section titled “1 — Instrument the service”Prometheus pull first — it needs no Grafana-side config and drops into the existing scrape.
(a) Expose /metrics (preferred). Emit the Prometheus text format on a plain HTTP path. No
SDK is required — lines of name{label="v"} value with a # HELP / # TYPE header:
# TYPE http_requests_total counterhttp_requests_total{route="/health",status="200"} 41Label cardinality is the one thing that ruins a scrape. Never label with a user id, a request id, a raw URL path, or a timestamp — each distinct value becomes a permanent series, an unbounded memory leak in Prometheus that no dashboard can fix afterwards. Label with small closed sets only.
(b) Push to the Pushgateway for anything unpullable (a cron job, a CI step, a machine with no
inbound port). It lives in monitoring (k8s/monitoring/pushgateway.yaml). Pushgateway never
expires a series, so push a *_last_success_timestamp_seconds heartbeat and alert on its age —
a dead pusher otherwise leaves its last values looking healthy forever.
Logs: the studio stack has no Loki today (verified 2026-09-26). Do not stand one up per
project; raise it as an issue against the instance. The Loki push recipe stays in
instrumenting-a-service.md (references/instrumenting-a-service.md) for when one exists.
2 — Register it
Section titled “2 — Register it”| Instrumented how | What to change | Where |
|---|---|---|
/metrics on an in-cluster Service | a ServiceMonitor | arc-github-runners k8s/monitoring/ |
/metrics outside the cluster | Endpoints + Service + ServiceMonitor (see jenkins-exporter-endpoints.yaml) | same |
| Pushgateway | nothing: it is already scraped | — |
| a new Grafana datasource | datasources: in helm/values.yaml | The1Studio/grafana-deploy |
Grafana reaches Prometheus by its cluster FQDN
(prometheus-kube-prometheus-prometheus.monitoring.svc:9090); a short Service name only resolves
inside monitoring, and Grafana is not there.
A target that loads but is not scraped is worse than one that fails — apply reports success
either way. After adding one, prove it: up{job="<job>"} must return rows. Zero rows is a
selector/label mismatch (the ServiceMonitor must carry release: prometheus), not an idle system.
3 — Ship a dashboard
Section titled “3 — Ship a dashboard”A dashboard is a ConfigMap in The1Studio/grafana-deploy, one per dashboard, mounted by name. A change is one PR with three edits — miss the third and the second deploy fails:
dashboards/<name>.yaml— the ConfigMap (namespacegrafana, same shape as the existing ones) holding the dashboard JSON.helm/values.yaml— adashboardProvidersentry for its folder (if new) and adashboardsConfigMapsentry naming the ConfigMap.- arc-github-runners
k8s/grafana-deployer.yaml— add the ConfigMap name to the deployer Role’sconfigmapsresourceNames. Without it the first deploy creates the ConfigMap and every later deploy fails to patch it.
The PR diff job kubectl diffs the rendered change against live; read its summary. Merging runs
Deploy (dashboards → helm upgrade --atomic → /api/health gate).
Generate the JSON from a working dashboard, never by hand: build it once in the UI (it will not
persist), GET /api/dashboards/uid/<uid>, commit the dashboard object. Hand-authored panel JSON
gets the grid, schemaVersion and datasource uid refs wrong. Panel model and reference rules:
provisioning-as-code.md (references/provisioning-as-code.md).
4 — Wire an alert
Section titled “4 — Wire an alert”Studio alerts are Prometheus rules, not Grafana alerts. Grafana holds no alert rules. The path:
PrometheusRule (arc-github-runners k8s/monitoring/*-alerts.yaml, label release: prometheus) -> Alertmanager (k8s/monitoring/alertmanager-config.yaml, receiver discord-webhook) -> discord-webhook-forwarder -> the studio Discord channelCopy an existing *-alerts.yaml. release: prometheus is mandatory — without it the operator
ignores the rule silently. Route by severity; the existing receivers already match
warning/critical, so a new rule needs no routing change.
A valid rule is not a working alert. Prove it: the rule appears under
/api/v1/rules on Prometheus with health: ok, and a rule whose expression returns no series
(absent()-style gaps) is a silent hole — alert on the data’s absence too.
Grafana-managed alerting (contact points, policies) is not used by the studio instance; that
recipe stays in alerting-and-discord.md (references/alerting-and-discord.md) for non-studio
instances.
5 — Operate the studio instance
Section titled “5 — Operate the studio instance”| Task | How |
|---|---|
| Change config / chart / dashboard | PR to grafana-deploy → PR diff → merge → Deploy |
| Roll back | revert the commit (Deploy re-runs), or helm rollback grafana <rev> -n grafana |
| Back up | nothing to back up: state = the repo + the grafana-admin Secret |
| Log in (browser) | the public URL → Cloudflare Access (GitHub, legacy: new internal tools use studio-auth) → auto-signed-in by Grafana JWT; nothing to type |
| Log in (admin fallback) | the LAN address shows Grafana’s own form; creds in Secret grafana-admin, ns grafana — read with kubectl, never paste them anywhere |
Machine client (scripts, the Grafana MCP grafana/mcp-grafana) | the LAN address from the SSOT doc + a Viewer service-account token; MCP runs with --disable-write |
Deploys run on a repo-scoped ARC runner (label grafana-deploy, ServiceAccount
grafana-deployer, rights only in namespace grafana). A chart bump that adds rules to the chart’s
Role fails “forbidden” instead of escalating — apply that one by hand as a cluster admin.
New internal dashboard or tunnel origin: MUST put studio-auth web forward-auth in front (docs/studio-auth.md; internal studio tools only, never customer-facing); MUST NOT add a new Access app. The block below documents the existing Grafana setup.
Auth — verified (first JWT login: tuha@the1studio.org → Admin; dated in grafana-deploy README):
- Cloudflare Access fronts the public URL: one self-hosted app, GitHub IdP only (auto-redirect),
policy “GitHub org The1Studio”, 24h session. Created by grafana-deploy’s
Cloudflare Accessworkflow (scripts/cloudflare-access.sh, repo secretCLOUDFLARE_ACCESS_API_TOKEN). - Grafana
[auth.jwt]signs the user in from theCf-Access-Jwt-Assertionheader: signature checked against the team’s/cdn-cgi/access/certs,expect_claimspinsissANDaud(the app’s AUD, kept in grafana-deployhelm/access.conf),auto_sign_up; Viewer by default, Admin for the owner’s emails (the Accessemailclaim is the GitHub account’s primary verified email, so map that address). A deploy refuses to run without a valid 64-hex AUD — without theaudcheck, a token minted for any other app in the same Zero Trust org would be accepted. - The LAN address has no Access in front of it and shows Grafana’s password form: that is the admin fallback and the path for machine clients. A non-browser client of the public URL gets the Access login page instead of Grafana.
- Known blip: every deploy restarts the pod; the host port-forward feeding the tunnel reconnects within seconds, so the public URL returns 502 briefly. Not an outage.
6 — Self-host a NON-studio Grafana
Section titled “6 — Self-host a NON-studio Grafana”Not for the studio instance — it is Helm-managed from grafana-deploy (§ 5). This is the recipe
for a separate, justified instance (see the rule’s § 1 before building one):
self-hosting-and-backup.md (references/self-hosting-and-backup.md) — compose, GF_* env
(precedence env > grafana.ini > defaults), the __FILE secret variant, and the two-halves
backup (grafana.db + provisioning dirs; restoring the db alone brings back empty folders).
Hard rules
Section titled “Hard rules”- Never commit a token, webhook URL or admin password. They come from the environment or a k8s
Secret — per
~/.claude/rules/security.md.
Executed by
Section titled “Executed by”Runs inline in the main session. Each stage is a small, reviewable file edit plus a verification
query against the live instance — the coordination cost of a hand-off exceeds the work, and the
verification step needs whatever context caused the request. t1k-devops-engineer owns the
CI/CD, compose and datastore-cost halves when a task is genuinely broader than observability.
References
Section titled “References”- instrumenting-a-service.md (
references/instrumenting-a-service.md) —/metricsformat, cardinality rules, push, health probes - provisioning-as-code.md (
references/provisioning-as-code.md) — datasource + dashboard provisioning schemas, dashboard JSON model, folders - alerting-and-discord.md (
references/alerting-and-discord.md) — Grafana-managed alerting (non-studio instances only) - api-and-service-accounts.md (
references/api-and-service-accounts.md) — service-account tokens, HTTP API endpoints - self-hosting-and-backup.md (
references/self-hosting-and-backup.md) — non-studio self-hosting: compose,GF_*env, backup/restore