Skip to content

t1k:web:devops:hard-compute-dispatch

FieldValue
Moduledevops
Version1.31.4
Effortmedium
Tools—

Keywords: backfill, concurrency guard, derived table rebuild, dispatch job, hard compute, hard-compute dispatch, heavy job, job registry, one-off job, reconcile, repository dispatch, rollup rebuild, scheduled heavy job, self-hosted runner job, vendor sync, wide sync, workflow dispatch

/t1k:web:devops:hard-compute-dispatch
[add-job|run|make-dispatchable|debug] [job-key]

Hard Compute Dispatch — Registry-Driven, Runner-Bounded Heavy Jobs

Section titled “Hard Compute Dispatch — Registry-Driven, Runner-Bounded Heavy Jobs”

One-off heavy work (vendor data syncs, derived-table rebuilds, backfills, reconciles) used to be a human typing a command into a production box at 2am. This pattern gives it a trigger, a timeout, a concurrency guard, a log, and an audit trail — without ever handing the production box to a free-form command.

repository_dispatch ──▶ resolve (ARC, disposable pod) ──▶ run (self-hosted) ──▶ POST ──▶ running service
or workflow_dispatch validate key+params curl the endpoint in-process guard owns
against the registry concurrency (busy if taken)

Three files:

FileRole
.github/hard-compute-jobs.jsonThe allowlist. Job key → runner, method, url, query, per-param regex patterns. SSOT.
.github/workflows/hard-compute-dispatch.ymlValidates the payload on the ARC pool, routes to the declared self-hosted one, invokes, uploads the response.
.claude/rules/hard-compute-dispatch.mdThe invariants you must not break (mirror this file’s Gotchas into your project).

Security model — the registry is the only place a command is assembled

Section titled “Security model — the registry is the only place a command is assembled”

A dispatch payload names a job key and supplies params. It never supplies a command, a URL, or a host. The registry declares the method, the URL, and a regex pattern for every param; the command is built from the registry alone, and every param is validated against its pattern before anything touches the self-hosted box. An unknown key, or a param that fails its pattern, fails on the resolve job, on its own disposable pod.

This is the injection boundary. Do not add a job whose target is interpolated from the payload — that reintroduces exactly the injection this shape exists to prevent.

SituationDo this
Run an existing job nowgh workflow run … -f job=<key> -f params='{…}' (below)
Trigger from code / a Worker / an external schedulerrepository_dispatch with a fixed event_type (below)
Add a job for a service that already has an endpointAdd a registry entry via PR. No workflow edit needed. The registry is a command allowlist that must not be runtime-editable, so it stays reviewed code, not a table.
Add a job for a service with no endpointBuild the endpoint first (§ Making a service dispatchable)
Job returned the busy statusNot a failure — a scheduled run holds the guard. Re-dispatch later.
Job needs the private network / a local databaseDeclare the runner that can reach it (§ Runner capability)
Pure CPU/build work, no local stateThe CI/build pool is usually enough
Terminal window
# Manual
gh workflow run "Hard Compute (dispatch)" \
-f job=<job-key> \
-f params='{"startDate":"YYYY-MM-DD","endDate":"YYYY-MM-DD"}'
# From anything holding a repo-scoped token (a Worker, a script, another workflow)
curl -X POST https://api.github.com/repos/<owner>/<repo>/dispatches \
-H "Authorization: Bearer $GH_TOKEN" \
-H 'Accept: application/vnd.github+json' \
-d '{"event_type":"<your-event>",
"client_payload":{"job":"<job-key>",
"params":{"startDate":"YYYY-MM-DD","endDate":"YYYY-MM-DD"}}}'

The response body lands as a run artifact and in the job summary.

Validation and execution run on different runners on purpose:

  • resolve — our ARC pool: runs-on: ${{ fromJSON(vars.RUNNER_CI_MEDIUM) }} (never a label array). Reads the registry, rejects an unknown job key, rejects a non-object params payload, and runs the jq validation (§ Adding a job) so a missing required param, an undeclared param, or a pattern mismatch fails on a fresh ephemeral pod — never on the privileged box and never a hosted runner. It shares the pool’s 4 GiB per-pod ceiling (memory.oom.group=1, measured 2026-08-22), which is ample for a jq validation and is why this job must stay read-and-validate only; anything heavy belongs in run. Passes the payload into the shell only via --arg/--argjson, never by interpolation. Emits job, runner, timeout as job outputs.
  • run — [self-hosted, <runner>]. needs: resolve, so it cannot start unless resolve passed. Reads method/url/query from the registry, assembles the query string from the declared query order using curl --get --data-urlencode (no string concatenation of user input), invokes, uploads the response as an artifact, and treats the busy status as exit 0.

An unknown key or a bad param never reaches the privileged box.

Runner capability is a boundary, not a preference

Section titled “Runner capability is a boundary, not a preference”

Declare a per-job runner in the registry. Different runner pools have different reach: the self-hosted box that can reach the private network / local database may be the only pool that can, while our shared ARC pool — and a GitHub-hosted runner — cannot. A data job placed on a pool without the route fails at connect time, long after the run started — so the constraint is data in the registry, read before dispatch, not tribal knowledge rediscovered per incident.

Fleet fact (2026-10-09): ARC pods cannot reach sv-2 by configuration (inferred 2026-10-08, no pod-level test; a private-network-only host: NetBird peer since 2026-10-20, Tailscale retired; see t1k-vpn), so a job that must reach sv-2 needs a runner on the far side. A dedicated heavy runner there is a plan only (arc-github-runners docs/heavy-runner-sv2-plan.md, not built).

The registry’s defaults.runner gives most jobs a sensible default; only jobs that genuinely need the other pool override it.

Which variable (RUNNER_CI_<size>, RUNNER_CD), is owned by t1k-web-devops-placement → references/runner-tiers.md. Read it before declaring a new runner label here rather than inventing one.

Append to .github/hard-compute-jobs.json:

"<job-key>": {
"description": "One line: what it does, and when you would reach for it.",
"runner": "<runner-label>", // capability boundary — see above
"timeoutMinutes": 60, // > the job's realistic p99, not its median
"method": "POST",
"url": "http://<service-host>:<port>/api/<endpoint>",
"query": ["startDate", "endDate"], // order matters; these become the query string
"params": {
"startDate": { "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}$", "required": true },
"endDate": { "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}$", "required": true }
},
"notes": "Anything the next person needs: window caps, guard behaviour, cost."
}

Every param needs a pattern. It is the injection boundary, not a typo-catcher — an unmatched param is rejected on the resolve runner before the self-hosted box is touched.

Test the validation before you push. The workflow’s check is a single jq expression; run it against your new entry directly:

Terminal window
jq -e -n --slurpfile reg .github/hard-compute-jobs.json \
--arg j '<your-key>' --argjson p '{"startDate":"2026-08-01"}' '
($reg[0].jobs[$j].params // {}) as $spec
| (($spec|to_entries|map(select(.value.required))|map(.key)) - ($p|keys)) as $missing
| ($p|keys|map(select($spec[.]==null))) as $undeclared
| ($p|to_entries|map(select(. as $e
| ($spec[$e.key] != null)
and ((($e.value|tostring)|test($spec[$e.key].pattern))|not)))|map(.key)) as $bad
| if ($missing|length)>0 then error("missing: "+($missing|join(", ")))
elif ($undeclared|length)>0 then error("undeclared: "+($undeclared|join(", ")))
elif ($bad|length)>0 then error("bad pattern: "+($bad|join(", ")))
else true end'

Confirm it REJECTS a missing required param and a value with a shell metacharacter ($(...), backticks, ;) — a validator never seen failing is unproven.

jq gotcha worth knowing: inside the $bad filter, . as $e must bind the entry before test(...). After the pipe into test(), . is rebound to the string being matched, so a bare $spec[.key] there indexes a string and errors out. Binding the entry first keeps $e.key referencing the real param name.

The heavy work must already be reachable as an HTTP endpoint on the service that owns the concurrency guard. That is the whole trick, and it is not optional:

  1. Add the endpoint to the owning service, alongside its scheduler. It validates its inputs, caps its window, and delegates to a method that shares the scheduler’s in-process isRunning guard, so it returns the busy status rather than overlapping the scheduled run.
  2. Expose the service’s port on the host so the self-hosted runner can reach it.
  3. Add the registry entry. No workflow change.

Booting a second application context against the same database starts a second scheduler and a second writer, exhausts the connection pool, and races the live one. A script also bypasses the in-process guard entirely, so it can corrupt a derived table mid-rebuild — and it leaves no trigger, no log, no artifact.

  • Decision tree for where long-running work lives at all: t1k-web-devops-placement
  • Keep the triggered job’s page/API reads fast: t1k-web-devops-core
  • The consumer-side invariants belong in the project’s .claude/rules/ — mirror this skill’s Gotchas into a hard-compute-dispatch.md rule there so every session sees the boundary.