Skip to content

t1k:test

FieldValue
Modulet1k-base
Version5.15.36
Effortmedium
ToolsAgent, Bash, Glob, Grep, Read, Task

Keywords: compile, coverage, failing, flaky, run-tests, test, unit

/t1k:test
[context] OR compile OR coverage OR --flaky OR --diff

Delegate to the registered t1k-tester agent. Never ignore failing tests.

FlagModeBehavior
(default)FullRun entire test suite
--flakyFlaky detectionRe-run failing tests up to 3 times, report retry rate per test
--diffDiff-awareOnly run tests for files changed since main branch
--coverageCoverageRun with coverage reporting, flag uncovered critical paths
--qualitySuite qualityAudit, write-standard check, speed, prune: use /t1k:test-quality (no logic here)
  1. Run test suite normally
  2. For any failing test, re-run it up to 3 times in isolation
  3. If test passes on retry: mark as FLAKY (intermittent)
  4. If test fails all retries: mark as FAILING (genuine failure)
  5. Report: flaky tests with retry rate, genuine failures separately
  1. Run git diff --name-only origin/main...HEAD to find changed files
  2. Map changed source files → corresponding test files
  3. Run only mapped test files
  4. If no mapping found: run full suite (fallback)

Filtered tests while developing, ONE full run per PR on the merged head after review, no heavy run for a docs/plans/.claude-only diff, generated files regenerated after the merge. Decide with bun .claude/scripts/t1k-verify-tier.cjs classify --base origin/main; policy and the lane procedure: references/verification-tiers.md.

Run the shard as CI does (Node major, env, selector), count skips, keep test titles stable, classify each red as failure, flake or infra: references/ci-parity-and-silent-greens.md.

NEVER IGNORE FAILING TESTS. Fix root causes, not symptoms.

Full-Verify Contract — What “Tests Green” Actually Means

Section titled “Full-Verify Contract — What “Tests Green” Actually Means”

“Tests passed” is insufficient evidence to claim done. Three gate types must all pass before reporting complete:

GateQuestion it answersTypical command
Type-checkDoes it compile?bun run typecheck / tsc --noEmit / cargo check
Lint / FormatDoes it meet project style?bun run lint / biome check / eslint . / ruff check
TestsDoes behavior match spec?bun test / bun test / pytest / cargo test

These gates are orthogonal. A change can pass tests and fail lint (or vice versa). CI runs all three; claiming “done” after running only one or two is a false positive that surfaces as a red PR later.

Before claiming work complete:

  1. Run the kit’s type-check script — read the output
  2. Run the kit’s lint/format script — read the output
  3. Run the kit’s test command — read the output
  4. Only then claim “done” / “green” / “verified”

If the kit has no lint script: note this explicitly (“no lint gate in this project”) so it’s visible rather than silently skipped.

Check package.json:

Terminal window
jq -r '.scripts | to_entries[] | select(.key | test("^(test|lint|typecheck|check|format|build)$")) | "\(.key): \(.value)"' package.json

If a repo has lint + typecheck + test scripts, run all three. Don’t assume one covers the others.

Invoking Watch-Capable Runners (vitest / jest) — foreground + timeout, never background

Section titled “Invoking Watch-Capable Runners (vitest / jest) — foreground + timeout, never background”

Vitest and Jest default to watch mode when invoked bare and never exit — always use the one-shot form (vitest run, jest --ci).

Run the one-shot form in the foreground with a Bash timeout, not as a background job:

Terminal window
CI=1 timeout 300 npx vitest run path/to/suite # one-shot, exits on its own; default reporter works on every vitest version
  • Don’t background vitest run yourself. Output capture is unreliable under a non-TTY background pipe. A FULL or HEAVY run is a session-owned background run: Bash with run_in_background: true, t1k gate run --class <gateClass> -- <heavyRun.argv> (the gate queues it), stdout and stderr redirected to a file, never a pipe. The orchestrator waits for it in-turn or with a session-owned Monitor until-loop on the exit sentinel, and never ends the turn while it is live. No detached process: t1k never starts one that outlives the session (see rules/agent-completion-discipline.md).
  • Force non-interactive output: CI=1 and/or --no-color. Do NOT pass --reporter=basic: vitest 5 no longer ships it and exits 1 at startup (Failed to load custom Reporter from basic, core#5321); --reporter=dot or the default reporter work on every version.
  • Genuine-hang fallback: re-run with --pool=forks --no-file-parallelism; kill the stuck process rather than polling its empty log.

MANDATORY — Process-count watchdog + worker caps for suites that may spawn children

Section titled “MANDATORY — Process-count watchdog + worker caps for suites that may spawn children”

A suite whose tests spawn scripts (spawnSync, bun run, a deploy/release script with a test gate) can re-enter itself and fork-bomb the host (2026-10-06: ~7,400 pids, 18 GiB, 383 OOM kills, a whole herdr session’s agents killed). Always cap workers (bun test --max-concurrency=4, vitest --maxWorkers=4, jest --maxWorkers=4, pytest -n 4) and run such a suite under a watchdog: if ps -eo args | grep -c "[n]ode --test" exceeds ~30, abort and SIGKILL the whole tree (run it under setsid, kill -KILL -- -PGID). Never use ulimit -u as the guard — it counts every process of the user. Before running, check no test spawns a script that re-runs its own suite (recursion-guard env var: rules/test-must-not-rerun-its-suite.md). Recipe: docs/test-must-not-rerun-its-suite.md.

Testing with the real user profile is allowed

Section titled “Testing with the real user profile is allowed”

A test or agent MAY run against the real user profile (real HOME, browser profile, gh/cf/npm logins) when it needs real authentication; do not block on “sandbox HOME” for that. Why (user, 2026-10-22): this is an all-users machine, so working against the real profile makes development faster, and the auth a test needs already lives in the browser profile and the gh/cf/npm logins, which a sandboxed HOME cannot reproduce.

One carve-out, from an incident: a harness that deliberately WRITES or MUTATES home files (break-test mutations that write ~/.npmrc, git/npm config, tokens) still redirects HOME/USERPROFILE/NPM_CONFIG_USERCONFIG for those writes, and proves the real file is unchanged with a stat before and after (2026-10-20: a mutation overwrote the real ~/.npmrc). Reading the real profile is fine; letting a destructive test write to it is not. Never print a token, cookie or assertion read from the profile.

A sandbox HOME is not isolation when the child reads a keyring

Section titled “A sandbox HOME is not isolation when the child reads a keyring”

Redirecting HOME isolates a child only as far as the child respects it. Four things it does not, each measured:

  • gh reads its login from the OS keyring, not from GH_CONFIG_DIR. A sandbox with an empty gh config dir still answers gh auth token with the maintainer’s real token (2026-10-23: a registry engine inside a “sandbox” minted real registry tokens this way). Put a gh shim FIRST on PATH that answers gh auth token only from GH_TOKEN/GITHUB_TOKEN in the child env and fails every other call; the shim must never read the keyring. Reference implementation: .claude/skills/t1k-registry/tests/helpers/sandbox-home.cjs.
  • bun and node disagree about HOME. os.homedir() honours HOME under bun on Linux; os.userInfo().homedir ignores it under node. Set HOME AND USERPROFILE, run every child under one runtime, and have a probe child report where os.homedir() actually points before the helper creates anything. Under bun, os.homedir() resolves once and ignores a later process.env.HOME change: hooks/__tests__/bun-os-homedir-shim.cjs is the preload that makes it follow.
  • The hook wrapper resets HOME from os.userInfo() (hook-gateway.ts), so it is NOT a safe way to run a hook in a sandbox. Run the hook file itself.
  • XDG and gh read their own variables (XDG_CONFIG_HOME, XDG_CACHE_HOME, GH_CONFIG_DIR), so set them together with HOME/USERPROFILE or a write escapes through one of them.

Prove the isolation instead of assuming it: a helper that returns a sandbox should have already shown the child’s home is inside the scratch dir, and a test should hash the real credential files before and after so it can show they did not change. Git config is a separate leak: use emptyGitConfigEnv() from hooks/__tests__/git-test-env.cjs (an empty regular file, never /dev/null, core#3945) rather than inheriting the developer’s identity.

MANDATORY — Kill vitest processes on completion (avoid OOM / IDE crash)

Section titled “MANDATORY — Kill vitest processes on completion (avoid OOM / IDE crash)”

--pool=forks (and --pool=threads on older versions) leaves orphaned bun workers alive on Windows when the parent exits via timeout, Ctrl-C, or normal completion. Accumulated orphans eat RAM and have crashed the editor / Claude Code host repeatedly.

Every vitest invocation MUST end with a kill sweep — pass, fail, foreground, or background. Sweep = kill the captured PID, kill its child tree, then name-fallback against node.*vitest. Wrap in trap EXIT (Bash) / try-finally (PowerShell) when scripted. Apply after run returns, after timeout 124, and before launching the next vitest.

The sweep walks DOWN, never UP. Kill the captured PID and the processes it spawned. Never walk the parent chain to find “the tree”: a background run’s ancestry runs past your shell into the Bash tool, Claude, herdr and systemd --user, so a ps -o ppid= loop signals the user’s session manager. pkill -P <pid> / %<id> are child-of (safe); a while … ps -o ppid= loop, kill $PPID or a bare killall node is not. Never signal PID 1, -1, 0, a session manager, or any ancestor of this process. 2026-11-03: an agent cleaning up its own background run walked ppid up to systemd --user and killed it — every herdr and Claude session in that exit.target died with the desktop.

Snippets: see references/vitest-process-management.md.

MANDATORY — Cap concurrent background vitest runs by device hardware

Section titled “MANDATORY — Cap concurrent background vitest runs by device hardware”

A static cap is wrong — a 4-core/8GB laptop chokes at 2, a 16-core/64GB workstation idles at 2. Compute the cap from the host:

cap = max(1, min(floor(logicalCores / 4), floor(freeMemGB / 4), 6))

cores/4 and freeMemGB/4 because one run already saturates ~25% of cores and 2-4 GB RAM (worker pool ≈ cores/2). Ceiling 6 prevents IDE/host paging. Floor 1 keeps low-end boxes functional.

Compute once per session (hardware doesn’t change). Before launch, count live node.*vitest processes — if ≥ cap, STOP (wait or kill oldest). Prefer foreground + timeout; at most cap - 1 background runs so a foreground slot stays free.

Compute snippets + reference table: see references/vitest-process-management.md.

Self-Teach on CI Gate Failures — MANDATORY

Section titled “Self-Teach on CI Gate Failures — MANDATORY”

Every time a CI quality gate (lint, format, typecheck, test, build, security scan) fails on a PR, you MUST update the responsible skill to prevent the same failure in future sessions. This is the development-principles.md “Update Skills After Every Error” rule applied specifically to CI.

Failure → lesson loop:

  1. CI gate fails on PR
  2. Identify which local-verify command would have caught it (the CI step’s name + what it runs)
  3. If that command is NOT in the skill’s verify checklist → add it
  4. If it IS in the checklist but was skipped → note why, reinforce the checklist
  5. Commit the skill update in the SAME session as the CI fix, not “later”

If you are about to push a fix-up commit for a CI failure without updating a skill, stop and update the skill first. The fix-up without the skill update guarantees the same person (or future you) will hit the same gate again.

PR theonekit-cli#79 (2026-04-21) had bun run typecheck clean + 433 tests green, but CI’s Lint stage failed on 3 OS matrices because biome check wasn’t in the local verify step. Cost: one extra round trip. Had a full-verify contract been followed, the formatter violations would have been caught and fixed locally before push. This skill section exists BECAUSE that happened.

Follow protocol: skills/t1k-cook/references/routing-protocol.md This command uses role: t1k-tester

Follow protocol: skills/t1k-cook/references/activation-protocol.md

  1. Compilation check (read console or build output)
  2. Run tests via registered t1k-tester agent
  3. Analyze test results for failures
  4. If failures → spawn registered t1k-debugger for root cause
  5. Report structured results
  6. Plane work-item update (see below)

Comment-only — no state transition. Full contract: skills/t1k-plane/references/workflow-enforcement.md (../t1k-plane/references/workflow-enforcement.md) § Stage 3. Governing rule: modules/t1k-plane/rules/plane-workitem-workflow.md.

Post the pass/fail counts to every work item bound to this session. A red suite does NOT move the item backwards — a failing test is information, not a state regression. If the project defines an extra started-group state named In Review / Testing / QA, move there instead of commenting alone; never create such a state to satisfy this. Plane enforcement now blocks, not warns, on an unbound repo or absent plane server — see the cited Stage 0 contract. Escapes: T1K_PLANE_MODE=advisory|off, --no-plane, recorded optout.

Module Context for Tester (if installedModules or modules present in metadata.json)

Section titled “Module Context for Tester (if installedModules or modules present in metadata.json)”

Follow protocol: skills/t1k-cook/references/subagent-injection-protocol.md Before spawning t1k-tester agent, inject:

  • Which module’s files are being tested (from .claude/metadata.json)
  • Module’s test skills if available
  • Boundary: “Test files in module {name} should not test cross-module behavior”

Sub-agent forking: see skills/t1k-architecture/references/fork-hygiene.md.