Skip to content

pinned-baseline-test-companion

Pinned-Baseline Tests Ship a Companion β€” and Invert, Never Re-Pin

Section titled β€œPinned-Baseline Tests Ship a Companion β€” and Invert, Never Re-Pin”

A pinned-baseline test asserts the currently broken state so the suite stays green while a known gap exists (a count pin, allow-list pin, absence pin, or identity-keyed pin).

Every pinned baseline MUST ship, in the same commit, a companion assertion that fails when the pin stops being accurate β€” and when the pin fails, invert it; never re-pin. Assert by IDENTITY, not count (78 misses can become 78 different misses and stay green); assert BOTH directions (no undocumented new entry AND no documented entry silently fixed); the failure message states what a rise means, what a fall means, and forbids re-pinning; verify the resolver that locates the target.

Is it a pin? If closing the gap tomorrow would turn the test RED, it owes a companion. Read the direction first: grew = regression, fix the code; shrank to zero = invert (assert the healthy state); shrank partway = shrink the identity list, then the count. Never re-pin to whatever the run just reported. Mechanics, naming and the resolver repair: docs/pinned-baseline-test-companion.md.