A check that has never failed is not a check you have tested
Two of my own checks were reporting PASS while looking at almost nothing. The inputs built to catch exactly that were green at the same time.
// the first one
It stopped reading at an apostrophe
One check measures page descriptions against a length limit. It pulled the text out of the attribute with a pattern that would accept either quote character as the closing one. On a double-quoted attribute containing an apostrophe, it stopped reading at the apostrophe.
So it measured the opening fragment and compared that to the limit. Everything after the apostrophe was invisible to it. Thirteen real pages were being measured wrong, and two more were failing for reasons that were not real.
// the second one
Eighty-eight declarations it never looked at
Another check enforces a minimum font size. It looked for the font-size property. The type scale on this site is written almost entirely in the font shorthand, and there were 88 shorthand declarations it had never once looked at.
Underneath them sat one real breach: a table header at 11px, below the 12px floor, on pages that were live.
// the part that was supposed to catch this
The broken samples were green too
Both checks come with deliberately broken inputs — samples built to be wrong, so the check has something to report. All 34 of those were green the entire time.
That is the finding, and it took me longer to accept than either pattern took to fix. The broken samples were written by the same person, in the same sitting, from the same idea of what "wrong" looks like as the check they test. A sample can only contain a failure its author thought of. When the check has a blind spot, its sample inherits the same one — not by accident, by construction.
// what actually changed
Watched to go red, before it is believed
Not the two patterns. Those were the easy part.
The rule now is that a check is not trusted until it has been pointed at a known-bad input and watched to go red. Not "has a sample." Watched to go red, before the check is believed.
I had two checks that had never once returned anything but PASS, and I read that as evidence they were working. It was evidence of nothing. A green light from something that has never been shown a real failure is not a result.