Skip to content

t1k:grafana

FieldValue
Modulet1k-devops
Version4.1.3
Effortlow
Tools—

Keywords: alertmanager, cloudflare access grafana, dashboard configmap, dashboard json, datasource provisioning, discord alert, GF_ env, grafana, grafana docker compose, grafana http api, grafana mcp, grafana-deploy, metrics endpoint, observability, prometheus, prometheusrule, provision dashboard, pushgateway, scrape config, service account token, servicemonitor, studio grafana

/t1k:grafana
instrument | register | dashboard | alert | operate | self-host

Studio Grafana — observability for everything we build

Section titled “Studio Grafana — observability for everything we build”

The studio runs one Grafana. Every self-built service ships its metrics, dashboards and alerts there — never to a per-project Grafana, a hosted Datadog/Honeycomb, or a sidecar.

WhatWhere (verified 2026-09-26)
Instancek3s on theone, namespace grafana, Helm chart grafana/grafana (pinned in the deploy repo), persistence off
Its source of truthThe1Studio/grafana-deploy (private): Helm values, dashboards, CI
Metrics + alertingkube-prometheus-stack in namespace monitoring, manifests in The1Studio/arc-github-runners k8s/monitoring/
Location / URL / dashboard SSOTdocs/grafana-node-monitoring.md in The1Studio/arc-github-runners

Read the SSOT doc for the live URL rather than hardcoding one from anywhere else, including this file. Normative rule: studio-grafana.md in this module. This skill is the how.

IntentReadDeliverable
“add metrics to a service”instrument/metrics endpoint (or Pushgateway push) in the service repo
“register the service / add a datasource”registerServiceMonitor in arc-github-runners, or datasource in grafana-deploy
“add a dashboard”dashboardPR to The1Studio/grafana-deploy
“alert me on Discord when X”alertPrometheusRule in arc-github-runners k8s/monitoring/
“deploy / roll back / log in / connect an MCP”operatea grafana-deploy PR, or a documented command
“stand up a Grafana that is NOT the studio one”self-hostcompose + provisioned dirs

Every step is a file in a repo and a git commit. If you are about to click in the UI, stop — the studio instance has no persistence: a UI-built dashboard, user or setting is gone on the next pod restart and on the next deploy.

Step 0 — read the instance’s own docs first

Section titled “Step 0 — read the instance’s own docs first”
Terminal window
G=".claude/scripts/t1k-gh.cjs"; [ -f "$G" ] || G="$HOME/.claude/scripts/t1k-gh.cjs"
bun "$G" get repos/The1Studio/grafana-deploy/contents/README.md --pick content | base64 -d | sed -n '1,80p'
bun "$G" get repos/The1Studio/arc-github-runners/contents/docs/grafana-node-monitoring.md \
--pick content | base64 -d | sed -n '1,60p'

The grafana-deploy README is the operating manual (layout, how a change ships, RBAC consequences); the arc doc is the SSOT for URL and dashboard inventory. Quote them, do not paraphrase a URL.

Auth for every API call below is a service-account token in an env var — never a literal, never argv (world-readable in ps). See api-and-service-accounts.md (references/api-and-service-accounts.md).

Prometheus pull first — it needs no Grafana-side config and drops into the existing scrape.

(a) Expose /metrics (preferred). Emit the Prometheus text format on a plain HTTP path. No SDK is required — lines of name{label="v"} value with a # HELP / # TYPE header:

# TYPE http_requests_total counter
http_requests_total{route="/health",status="200"} 41

Label cardinality is the one thing that ruins a scrape. Never label with a user id, a request id, a raw URL path, or a timestamp — each distinct value becomes a permanent series, an unbounded memory leak in Prometheus that no dashboard can fix afterwards. Label with small closed sets only.

(b) Push to the Pushgateway for anything unpullable (a cron job, a CI step, a machine with no inbound port). It lives in monitoring (k8s/monitoring/pushgateway.yaml). Pushgateway never expires a series, so push a *_last_success_timestamp_seconds heartbeat and alert on its age — a dead pusher otherwise leaves its last values looking healthy forever.

Logs: the studio stack has no Loki today (verified 2026-09-26). Do not stand one up per project; raise it as an issue against the instance. The Loki push recipe stays in instrumenting-a-service.md (references/instrumenting-a-service.md) for when one exists.

Instrumented howWhat to changeWhere
/metrics on an in-cluster Servicea ServiceMonitorarc-github-runners k8s/monitoring/
/metrics outside the clusterEndpoints + Service + ServiceMonitor (see jenkins-exporter-endpoints.yaml)same
Pushgatewaynothing: it is already scraped—
a new Grafana datasourcedatasources: in helm/values.yamlThe1Studio/grafana-deploy

Grafana reaches Prometheus by its cluster FQDN (prometheus-kube-prometheus-prometheus.monitoring.svc:9090); a short Service name only resolves inside monitoring, and Grafana is not there.

A target that loads but is not scraped is worse than one that fails — apply reports success either way. After adding one, prove it: up{job="<job>"} must return rows. Zero rows is a selector/label mismatch (the ServiceMonitor must carry release: prometheus), not an idle system.

A dashboard is a ConfigMap in The1Studio/grafana-deploy, one per dashboard, mounted by name. A change is one PR with three edits — miss the third and the second deploy fails:

  1. dashboards/<name>.yaml — the ConfigMap (namespace grafana, same shape as the existing ones) holding the dashboard JSON.
  2. helm/values.yaml — a dashboardProviders entry for its folder (if new) and a dashboardsConfigMaps entry naming the ConfigMap.
  3. arc-github-runners k8s/grafana-deployer.yaml — add the ConfigMap name to the deployer Role’s configmaps resourceNames. Without it the first deploy creates the ConfigMap and every later deploy fails to patch it.

The PR diff job kubectl diffs the rendered change against live; read its summary. Merging runs Deploy (dashboards → helm upgrade --atomic → /api/health gate).

Generate the JSON from a working dashboard, never by hand: build it once in the UI (it will not persist), GET /api/dashboards/uid/<uid>, commit the dashboard object. Hand-authored panel JSON gets the grid, schemaVersion and datasource uid refs wrong. Panel model and reference rules: provisioning-as-code.md (references/provisioning-as-code.md).

Studio alerts are Prometheus rules, not Grafana alerts. Grafana holds no alert rules. The path:

PrometheusRule (arc-github-runners k8s/monitoring/*-alerts.yaml, label release: prometheus)
-> Alertmanager (k8s/monitoring/alertmanager-config.yaml, receiver discord-webhook)
-> discord-webhook-forwarder -> the studio Discord channel

Copy an existing *-alerts.yaml. release: prometheus is mandatory — without it the operator ignores the rule silently. Route by severity; the existing receivers already match warning/critical, so a new rule needs no routing change.

A valid rule is not a working alert. Prove it: the rule appears under /api/v1/rules on Prometheus with health: ok, and a rule whose expression returns no series (absent()-style gaps) is a silent hole — alert on the data’s absence too.

Grafana-managed alerting (contact points, policies) is not used by the studio instance; that recipe stays in alerting-and-discord.md (references/alerting-and-discord.md) for non-studio instances.

TaskHow
Change config / chart / dashboardPR to grafana-deploy → PR diff → merge → Deploy
Roll backrevert the commit (Deploy re-runs), or helm rollback grafana <rev> -n grafana
Back upnothing to back up: state = the repo + the grafana-admin Secret
Log in (browser)the public URL → Cloudflare Access (GitHub, legacy: new internal tools use studio-auth) → auto-signed-in by Grafana JWT; nothing to type
Log in (admin fallback)the LAN address shows Grafana’s own form; creds in Secret grafana-admin, ns grafana — read with kubectl, never paste them anywhere
Machine client (scripts, the Grafana MCP grafana/mcp-grafana)the LAN address from the SSOT doc + a Viewer service-account token; MCP runs with --disable-write

Deploys run on a repo-scoped ARC runner (label grafana-deploy, ServiceAccount grafana-deployer, rights only in namespace grafana). A chart bump that adds rules to the chart’s Role fails “forbidden” instead of escalating — apply that one by hand as a cluster admin.

New internal dashboard or tunnel origin: MUST put studio-auth web forward-auth in front (docs/studio-auth.md; internal studio tools only, never customer-facing); MUST NOT add a new Access app. The block below documents the existing Grafana setup.

Auth — verified (first JWT login: tuha@the1studio.org → Admin; dated in grafana-deploy README):

  • Cloudflare Access fronts the public URL: one self-hosted app, GitHub IdP only (auto-redirect), policy “GitHub org The1Studio”, 24h session. Created by grafana-deploy’s Cloudflare Access workflow (scripts/cloudflare-access.sh, repo secret CLOUDFLARE_ACCESS_API_TOKEN).
  • Grafana [auth.jwt] signs the user in from the Cf-Access-Jwt-Assertion header: signature checked against the team’s /cdn-cgi/access/certs, expect_claims pins iss AND aud (the app’s AUD, kept in grafana-deploy helm/access.conf), auto_sign_up; Viewer by default, Admin for the owner’s emails (the Access email claim is the GitHub account’s primary verified email, so map that address). A deploy refuses to run without a valid 64-hex AUD — without the aud check, a token minted for any other app in the same Zero Trust org would be accepted.
  • The LAN address has no Access in front of it and shows Grafana’s password form: that is the admin fallback and the path for machine clients. A non-browser client of the public URL gets the Access login page instead of Grafana.
  • Known blip: every deploy restarts the pod; the host port-forward feeding the tunnel reconnects within seconds, so the public URL returns 502 briefly. Not an outage.

Not for the studio instance — it is Helm-managed from grafana-deploy (§ 5). This is the recipe for a separate, justified instance (see the rule’s § 1 before building one): self-hosting-and-backup.md (references/self-hosting-and-backup.md) — compose, GF_* env (precedence env > grafana.ini > defaults), the __FILE secret variant, and the two-halves backup (grafana.db + provisioning dirs; restoring the db alone brings back empty folders).

  • Never commit a token, webhook URL or admin password. They come from the environment or a k8s Secret — per ~/.claude/rules/security.md.

Runs inline in the main session. Each stage is a small, reviewable file edit plus a verification query against the live instance — the coordination cost of a hand-off exceeds the work, and the verification step needs whatever context caused the request. t1k-devops-engineer owns the CI/CD, compose and datastore-cost halves when a task is genuinely broader than observability.

  • instrumenting-a-service.md (references/instrumenting-a-service.md) — /metrics format, cardinality rules, push, health probes
  • provisioning-as-code.md (references/provisioning-as-code.md) — datasource + dashboard provisioning schemas, dashboard JSON model, folders
  • alerting-and-discord.md (references/alerting-and-discord.md) — Grafana-managed alerting (non-studio instances only)
  • api-and-service-accounts.md (references/api-and-service-accounts.md) — service-account tokens, HTTP API endpoints
  • self-hosting-and-backup.md (references/self-hosting-and-backup.md) — non-studio self-hosting: compose, GF_* env, backup/restore