What “ready” means here

Every check suite ships with an input that is deliberately wrong, and I don’t trust the suite until I have watched it fail on that input. A check that has never gone red hasn’t been tested. It has only been run.

The same rule covers a website’s link check, a game’s audio-file move and an AI model’s claim that it’s done.

The note on why: two of my own checks once passed while looking at almost nothing →

Before I trust a check, I feed it what it should catch. On Devgenome’s release, the checks were run against a copy seeded with planted defects: every one had to be reported, and nothing could be reported on the clean copy.

In a data system I’m building, each new detector is tried on planted fakes it must flag, clean decoys it must leave alone and the lookalike it is most likely to be confused with, and its real detections are checked one by one before it is used.

Devgenome, release 1.2.0, 29 September 2026: 12 planted, 12 reported, 0 on the clean copy.

Live data isn’t where a change is tried first. The change is rehearsed on a copy of that data, and a rollback that only pretends to work is refused.

When ChampChase moved 119 audio files, 97 of them out of its base install, the move was rehearsed on 5 of them first: deliberately broken, caught by the check, reverted and proven identical. Only then did the 119 move.

Changes to the game’s save format ship with migration tests, and a save from a newer version opened on an older build loads read-only instead of being overwritten.

ChampChase, 24 July 2026: 57 of 57 save-migration tests passed. 9 August 2026: 119 files moved after the 5-file rehearsal.

The rules for a test are written before the data is read, so a result can’t move the goalposts. A number nobody can work out again isn’t a measurement.

When a claim turns out to rest on a check that was never seen to fail, the claim is struck on the record, not quietly reworded.

The CityRoamer 1.3.0 package had to clear every one of these before it went to the client, between its packaging step and its release checks. Nothing shipped around them.

On a full release, every page is loaded in Chromium, WebKit (Safari’s engine) and Edge and checked for errors, failed requests and one main heading. Layout and zoom are checked on six representative pages, and the home page also gets the keyboard, JavaScript-off, lost-graphics, slow-connection and reduced-motion runs.

CityRoamer, release 1.3.0: 119 checks. Before these results were trusted, each suite had been watched to fail on a planted wrong answer.

Forms are tested on the same server software the host runs, sending into a local test mailbox: duplicates, sending limits, spam tokens and a mail server that refuses. A visitor whose message can’t be sent keeps what they typed.

3D is checked against the file it came from. The Devgenome scene was compared with its Blender export frame by frame, 3,002 frames, before it shipped, and how long it takes to appear was measured on three connections.

Some bugs that reached a review got a check of their own: phone scrolling, the helix settling after a spin, loader progress and the iPhone toolbar resizing the page mid-scroll (Devgenome), and the first frame of a looping video (CityRoamer).

Then the site is walked like a visitor would walk it: load, scroll at reading pace, scroll back, reload halfway down, scroll while it loads, and submit an empty form on the contact page. It’s judged by looking at the frames, not by a pass count.

The frames below are the CityRoamer home page walk, recorded on 29 September 2026 at the screen sizes of three iPhones in WebKit, a tablet, a laptop and a desktop. Earlier builds of the same site were also checked on my own iPhone and Mac.

The whole live site is backed up first, with a hash of every file. Then each new file is uploaded over an encrypted connection, read back, and its hash checked before it takes the old one’s place. Pages go last. The home page goes last of all.

After that come the live checks, 63 on Devgenome and 108 on CityRoamer on 30 September 2026, and one labelled enquiry sent through each live form, with its acknowledgement received.

Checked again on 2 October 2026: all 62 public files Devgenome serves match the delivered files (release 1.2.0 plus the 30 September update) byte for byte.

iziwerk, a software product for freelancers, kept every customer’s data apart with rules enforced inside the database, not only in the app’s code. Every phase added hostile-customer tests to one suite: a script that logs in as someone else and tries to read or change what isn’t theirs, alongside checks that the owner still can.

The day after it went live, a full-product audit tried to break it the way a determined user would. It raised 355 findings, and a twelve-pass fix program followed. The few it didn’t close were parked and named on the record, not hidden.

A change that can’t be undone waits. When a feature was retired, removing access shipped first, because one grant undoes it; deleting its data was designed and parked until one of four named triggers fires, and even then only after a backup has been proven by a test restore.

21 July 2026, against the live database: 556 of 556 isolation checks passed. It has since been archived as a portfolio project. The full case →

An AI model saying a task is done is a claim, not a result. In my lab, a model’s “done” is checked against its visible tests and against hidden checks it never saw.

In the data system I’m building, code decides whether a rule was met: a model can explain the evidence, but it can’t waive it.

155 runs across four models, 23–24 September 2026: no run said it was done while its checks failed, and the trap set for a false claim caught no one in 31 attempts. The write-up →

Before anyone else sees the work, it’s reviewed by AI reviewers that didn’t build it, each given its own part to go through, and then by me. Where a fix can be checked automatically, the check is first seen failing on the old build.

For a website, the package is then unpacked into an empty folder, away from the development copy, served fresh, and its manifest of file hashes checked; on CityRoamer 1.3.0 all 109 files matched. The review link is kept out of search engines.

Not every job needs every step, and I don’t pretend otherwise. The quote says which tier a job gets.

None of this has to be taken on trust. On a call I can show you a check going red on its broken input before it went green, and the record of the labelled enquiry sent through a live form.

Book that call →