Research · replication

Rerun this study, or break it

A study you cannot rerun is an anecdote. This page is the recipe: the machine readable record behind all 14 claims, the rules that assigned every verdict, and, most usefully, exactly what evidence would prove this study wrong. If you find it, open an issue and the record changes.

The record, machine readable

Every claim, verdict, source URL and check date lives in one portable JSON file, the same file this site renders from and cross checks in CI on every push. Take it, parse it, re open every source.

evidence.json (raw) ↗versioned release ↗rendered claims table

The decision rules

A claim is confirmed only when its primary source was opened directly: the repository, the CI configuration file, the arXiv full text or the DOI record, never a search snippet or an abstract of an abstract.
A claim of absence (no tool does X) is recorded with the searches that failed to find it, and stays falsifiable by a single counterexample.
When the evidence contradicted the starting hypothesis, the hypothesis moved, not the evidence. Two starting assumptions are recorded as false.
Vendor numbers stay labeled vendor reported. Preprints stay labeled preprints. Nothing is promoted by enthusiasm.

How to falsify it

The load bearing conclusions, and the single finding that would break each one:

The surviving gap claim. Find one tool, released before the check date, that is pytest native, works across SDKs, automatically detects equivalence or regression failures, and ships as reusable CI packaging, all four at once. One example ends the central claim, and the study says so in its own gap analysis.
Cross version regression testing already exists. Show that QUTest does not do what its paper and repository state, or that the cited CI output is not real.
Benchpress detects nothing. Find assertion or detection logic in its harness beyond benchmarking, with a file path.
The corrected bug corpus count. Produce a resolvable version of Bugs4Q with a different entry count than the recorded 36 and 42.

Found one? Open an issue on the research repository. The record versions forward; being corrected in public is the point of keeping one.

Known limits

The search was English language, so non English tooling may be underrepresented; one German language adjacent result is noted in the record. No empirical prototype comparison has been run yet, the companion tool exists but has not been benchmarked head to head against the systems in the prior art matrix. Those limits are stated in the paper too, because a reviewer would find them in five minutes anyway.