The benchmark was wrong before the models were

SWE-bench gives a model a real GitHub issue from one of 12 Python repositories and asks for a patch. The patch is applied and the task's tests are run. A task counts as resolved "if all tests across FAIL_TO_PASS and PASS_TO_PASS pass," and the score is the share of tasks resolved.1

A score like that is only as good as the tests and the task descriptions behind it.

Terminal-Bench measures something different: "how well AI agents perform complex tasks in the terminal," with each task checked by its author's tests.6

In 2024 OpenAI had 93 developers review 1,699 SWE-bench samples. They flagged "38.3% of samples" for underspecified problem statements and "61.1%" for "unit tests that may unfairly mark valid solutions as incorrect." In total "68.3% of SWE-bench samples" were filtered out, leaving the 500-task SWE-bench Verified.2

In February 2026 OpenAI stopped reporting SWE-bench Verified. An audit of 138 Verified problems that o3 did not consistently solve found that "59.4% of the 138 problems contained material issues in test design and/or problem description." OpenAI also found that the frontier models it tested could reproduce the original bug fix, or the problem statement verbatim, for certain tasks.3

Independent studies found passes that shouldn't count. SWE-Bench+ checked 251 passed instances by hand: "32.67% of the successful patches involve cheating as the solutions were directly provided in the issue report or the comments," and "31.08% of the passed patches are suspicious patches due to weak test cases."4

UTBoost added missing tests and "uncovered 345 erroneous patches incorrectly labeled as passed," affecting 24.4% of SWE-bench Verified leaderboard entries and producing 11 ranking changes there.5 Terminal-Bench's own maintainers later fixed "issues in 28 of the 89 tasks" of version 2.0.6

The harness matters too. Anthropic wrote that performance "can vary significantly based on this scaffolding, even when using the same underlying AI model."7 Terminal-Bench's own table lists GPT-5.4 at 76.0% in Codex CLI and 55.1% in Terminus 2.6

My lab runs Claude Code and Codex on a fixed set of five coding tasks in locked-down containers. It records what the model claims separately from what the checks find, and it grades in fresh containers the model never touches. How it keeps the claim apart from the check is its own article.

All twelve were caught on 23 and 24 September 2026, before any model result was trusted.

The scorer.

• The first live run was scored as claiming checks that weren't in its log. Every one was there. A stray backtick in the model's own heading had shifted the parser. The correction was recorded, and the original receipt was left as it was.
• The scope rule was stricter than the prompts. Eleven runs were rejected for making existing tests stricter (10 of 155 lab runs, plus 1 smoke run), and none had weakened a test. The lab's first reading said a model "fell into the trap." The lab's correction: "That was wrong: the scope rule rejected them, the trap did not."
• A completion claim written in bold was scored as missing, in two runs.
• "No checks claimed" was mostly the parser. 45 of the 46 runs scored that way had listed their checks, under a heading it didn't recognise.
• For a while, any claimed command counted as observed whenever the transcript held an empty command, because an empty string is inside every string. A read-only review found it, and the next scoring version fixed it.

The fixtures and the harness.

• A safety fixture went red and green under load, because a 3-second timer raced a restart. It now waits for a flag that only exists after the restart, and its positive control was seen failing.
• Harness files were edited during a live batch, so three runs recorded fingerprints of code they didn't run. They were corrected in the record, not rewritten.
• A container image was missing a helper binary. The model reported that it couldn't run anything. That run is kept out of the model results.
• Codex's sub-agents made tool calls that never reached the transcript, in five runs. The final-state checks were unaffected.

The records and the rubric.

• Setting up version control overwrote one of the lab's configuration files. It was restored byte for byte before the first commit.
• The rubric judges accepted 55 of 55 runs before they had ever been seen rejecting anything. Known-answer controls were added afterwards.
• One rubric contradicts itself. Fresh judges said "unsure" twice against a known-answer case. The wording fix is planned for the next bench version.

The corrected bench states the test rule in its prompts. In the second batch, on the same models, scope rejections went from 10 to 0. Harder tasks are planned for the next bench version.

• That any particular leaderboard number is wrong. The studies above measured specific benchmarks at specific dates.
• A ranking of the models in my lab. This article doesn't give one.
• That these twelve catches are all there were. They are the ones found.