Skip to content

t1k:web:devops:cloudflare-internal-netbird

FieldValue
Moduledevops
Version1.31.4
Effortmedium
Tools—

Keywords: -internal host, access app worker destination, cf-access-client-id, cf-access-client-secret, cloudflare internal worker, custom_headers netbird, internal host behind access, netbird service token, non_identity policy, service auth policy, service token access app, worker behind netbird, worker custom domain access

/t1k:web:devops:cloudflare-internal-netbird
[new|verify|rotate|retire] <name>

Worker on <name>-internal behind an Access service token, published on the mesh

Section titled “Worker on <name>-internal behind an Access service token, published on the mesh”

An app is internal but runs on the edge. Three pieces, in this order:

peer --NetBird DNS--> <name>.<zone> (the name people use; mesh-only)
--TLS/WireGuard--> netbird reverse proxy (stamps X-NetBird-User/Groups)
--HTTPS + CF-Access-Client-* headers--> <name>-internal.<zone> (Worker custom domain)
--Cloudflare edge--> Access app (service token ONLY) --> the Worker

The Access app is the only way in; the proxy is the only holder of the token. Same shape as package-registry staging (upm-staging.the1studio.org → upm-internal-staging.the1studio.org).

Not this skill: a self-hosted container on the mesh with no Worker (/t1k:web:devops:selfhost-netbird), a public app on Tunnel + Access (/t1k:web:devops:selfhost-cloudflare), platform choice (/t1k:web:devops:placement).

The script is deterministic and fully specified, so running it is a mechanical step; deciding what to publish, which Access group admits whom, and whether a host may exist at all are not.

StepAgent
Run scripts/internal-netbird.sh <name> --dry-run, then for real once the Worker is deployed, and report its output block verbatim (names, ids, aud, the three verification lines)t1k-ci-watcher — one command, report what it printed
After a deploy, poll https://<name>-internal.<zone>/ for the cf-access-aud header to appear and confirm it equals the aud the script printedt1k-ci-watcher

Stays on a reasoning tier: choosing the group that should reach the service, deciding the -internal hostname, approving a token rotation, and every retire/delete (they change who can reach a host). Never let a cheap tier delete an Access app or a NetBird service.

cloudflare.config.ts, per deployment mode. The -internal host is a Worker custom domain (DNS + TLS created by the deploy), never a Workers Route:

domains: ["<name>-internal.<zone>"],
workersDev: false,
previewUrls: false,

Deploy with an explicit --mode (a bare cf deploy evaluates as production). Nothing else changes: no triggers, no separate script per host — one Worker can serve several -internal hosts.

The script must exist before step 2, and the Access app references it by worker id — so the first deploy necessarily publishes the domain before Access exists. package-registry accepted that window (its app was attached after the fact). To avoid it, deploy a config with no domains first, create the app (step 2), then deploy again with domains. [second order not exercised]

2. One command for the rest: scripts/internal-netbird.sh

Section titled “2. One command for the rest: scripts/internal-netbird.sh”

cf creates the token, the app and its policy; curl creates the NetBird service. The script does all four, is idempotent, and verifies the result:

Terminal window
cd .claude/skills/t1k-web-devops-cloudflare-internal-netbird # installed layout
./scripts/internal-netbird.sh <name> --zone the1studio.org \
--netbird-token "$NETBIRD_TOKEN" # or NETBIRD_TOKEN in the env
FlagDefaultMeaning
--zonethe1studio.orgzone for both names (the -internal host and the mesh name)
--netbird-token$NETBIRD_TOKENNetBird user PAT (/api/reverse-proxies)
--groupAllaccess group, by id or name; narrow it to the team that should see the service
--worker<name>Worker script name, when it is not the base name
--dry-run—print every name and action, touch nothing

Names it creates, all derived from <name> (a rerun finds them again):

ObjectName
service token<name>-netbird
Access app<name>-netbird
Access policy (reusable, attached by id)<name>-netbird
NetBird service<name>.<zone> → <name>-internal.<zone>

Rerunning is safe, and this is the intended repair path: an existing app, policy and service are reused (the policy is updated in place, so its id and its attachment never change), and an existing token is rotated (client_secret_version + 1, previous secret valid 24 h) because a secret is printed only on create. Every other policy on the app is kept.

What it refuses (each is a real API failure it turns into a message first): a worker name that does not exist, a worker already claimed by another Access app (409 12132), a NetBird group that does not exist, a name that is not a legal DNS label, and a name ending in -internal (that is the Worker, not the base name).

Order matters: the Worker and its -internal domain must already be deployed (step 1). The script looks the worker up; it never deploys. [unexercised: deploy the Worker with no domains, run this script, then add domains — the verified order publishes the domain first]

The steps it performs, if you are doing them by hand:

Terminal window
# token — a secret comes back ONCE, on create
cf zero-trust access service-tokens create --name <name>-netbird --duration forever
# policy: create as a REUSABLE account policy, then attach it BY ID
PID=$(cf zero-trust access policies create --name <name>-netbird --decision non_identity \
--include "$(jq -nc --arg t <token-id> '[{service_token:{token_id:$t}}]')" | jq -r .id)
# app + the policy REFERENCE (an inline policy object here would be app-local)
cf zero-trust access applications create --body "$(jq -nc --arg w <worker-id> --arg pid "$PID" '{
name: "<name>-netbird", type: "self_hosted",
destinations: [{ type: "worker", worker_id: $w }],
session_duration: "0s", allowed_idps: [],
policies: [{ id: $pid }] }')"
# NetBird service: mesh name -> the -internal host, token in custom_headers
curl -fsS -X POST https://vpn.the1studio.org/api/reverse-proxies/services \
-H "Authorization: Token $NETBIRD_TOKEN" -H 'Content-Type: application/json' \
--data "$(jq -nc --arg host <name> --arg zone <zone> --arg cid <client-id> --arg sec <secret> '{
name: ($host + "." + $zone), domain: ($host + "." + $zone),
mode: "http", private: true, enabled: true, access_groups: ["<group-id>"],
pass_host_header: false, rewrite_redirects: false,
targets: [{ target_id: $zone, target_type: "cluster", path: "/", protocol: "https",
host: ($host + "-internal." + $zone), port: 443, enabled: true,
options: { direct_upstream: true,
custom_headers: { "CF-Access-Client-Id": $cid,
"CF-Access-Client-Secret": $sec } } }] }')"

The App object, the Decision value and the measurement behind each: references/access-service-token.md.

The three NetBird settings that are not defaults:

SettingValueWhy
pass_host_headerfalseThe Worker routes tenants by Host and answers 404 unknown host for names outside its list. Measured: true → 403, false → 200 for the same backend.
protocol / porthttps / 443The target is the public -internal hostname, not a container.
access_groupsthe team, not AllGroup ids from GET /api/groups; All admits every peer.

Status moves pending → active within seconds; a one-label name uses the zone wildcard, so meta.certificate_issued_at being set immediately is normal.

Verify — the script runs checks 1–3 itself

Section titled “Verify — the script runs checks 1–3 itself”
Terminal window
H=<name>-internal.<zone>
# 1. gated at the edge: 403 and the aud of THIS app (a wrong app answers with its own aud)
curl -sS -D- -o /dev/null "https://$H/" | grep -i cf-access-aud
# 2. the token is sufficient, direct (no proxy in the path): 200
curl -sS -o /dev/null -w '%{http_code}\n' "https://$H/" \
-H "CF-Access-Client-Id: $CID" -H "CF-Access-Client-Secret: $SEC"
# 3. through the mesh name: the Worker's own 200 + headers — proves custom_headers landed
curl -sS -D- "https://<name>.<zone>/" | sed -n '1p;/x-registry\|x-/p'
# 4. from a host that is NOT on the mesh: must fail (NXDOMAIN/403), never 200

Check 3 is the only proof the proxy forwards the token; if 2 passes and 3 answers 403, drop to GET /api/reverse-proxies/services and read targets[0].options.custom_headers. Check 4 cannot be automated from a mesh peer — run it from a machine that is not on the VPN.

ChangeDo
Rotate the tokenRerun the script: it bumps client_secret_version with a 24 h grace on the old secret, then rewrites custom_headers. By hand: cf zero-trust access service-tokens update <id> --name <n> --client-secret-version <v+1> --previous-client-secret-expires-at <ISO8601> — --name is required, an update without it drops the name off the record (measured)
Rename the mesh nameCreate the new service, verify, delete the old
RetireDelete the NetBird service, then cf zero-trust access applications delete <app-id> --force, then the token, then drop domains from the Worker config and deploy. Never the reverse: detaching the domain last keeps the host gated until it is gone
Add a caller (CI, another proxy)A new service token + a second policy on the same app (each policy’s include is one OR-list); never a second app on the same worker id (409)
Add a second WorkerAdd a worker destination to the app (one app then holds several; verified)

Do not delete the app before the Worker’s domain: that leaves the host public with no gate.

  • worker_id, not worker name. cf workers scripts search --name <w> returns the script uuid. A guessed id creates an app that gates nothing, silently.
  • A worker id can belong to exactly ONE app. A second app targeting it is a hard 409 (12132 destination belongs to another application) — e.g. the registry’s staging Worker is already claimed by package-registry-netbird. Add a destination to the existing app instead.
  • A hostname already claimed by another app is the same 409. Read the live apps (cf zero-trust access applications list) before creating; a stale app you forgot is the usual claimant. public → worker also cannot be flipped in place on a claimed hostname, so create the app with the worker destination from the start.
  • Re-sending a policy object without its id DETACHES the old one. An app update whose policies[] entry has no id creates a NEW policy; the previous one is silently left behind as an orphaned reusable policy with app_count: 0 — the app keeps working, so nothing looks broken while the account accumulates dead policies (this is exactly what a rename done naively caused). Attach by {id} and mutate the policy through cf zero-trust access policies update.
  • Policy attachment is by id; an inline policy object makes an app-local policy (reusable: false). Use the reusable form (cf zero-trust access policies create, then policies: [{id}]): it can back several apps and it survives a rerun with the same id.
  • The client secret is readable by anyone holding a NetBird user PAT (GET /api/reverse-proxies/services prints custom_headers). Treat that PAT as a secret.
  • service-tokens update without --name erases the token’s name. The record survives with name gone, so the next lookup by name misses it and a rerun creates a duplicate. Always pass --name, as the script does.

More with symptoms: ledger/gotchas/t1k-web-devops-cloudflare-internal-netbird.md.

Script: scripts/internal-netbird.sh — --dry-run first; its header lists every name it creates.

  • references/access-service-token.md — exact API bodies, the JWT verification contract, verification transcript
  • /t1k:web:devops:selfhost-netbird — the plain mesh service (no Worker, no Access)
  • /t1k:web:devops:cloudflare-access — creating the app/policy/token correctly, auditing orphans
  • /t1k:cloudflare — cf itself, deploy flags, config shape
  • package-registry — cloudflare.config.ts (staging domains + AUD), src/web/guard.ts (service-token-only assertion), src/access/proxy-identity.ts