Can AI Write Tests That Catch Bugs in Quantum Code?
A test that waves broken code through is worse than no test. This study checks whether AI-written tests for quantum programs can tell right from wrong.
In one paragraph
Language models increasingly write the unit tests that guard code, but a test that passes broken code is worse than none. QTestBench asked five models to write tests for eight PennyLane QHack challenges and ten Qiskit HumanEval tasks, and scored each verdict against the benchmark's own checker. The tests almost always ran but were better at accepting correct code than rejecting wrong code. The best test quality was 0.71 on QHack and 0.34 on Qiskit HumanEval.
How did QTestBench check AI-written quantum tests?
Quantum programs are probabilistic, depend on numerical tolerances and are judged by special checkers, so a well-formed test can still disagree with the official verdict. Existing benchmarks grade whether models write correct quantum solutions, but not whether their tests can tell correct code from faulty code.
In QTestBench, five models (Claude Opus 4.6, DeepSeek V3.2, Gemini 3 Pro, GPT-5.4 and Qwen3-Coder) wrote syntactic, semantic and behavioral pytest modules in a single pass, without seeing the reference solution. The framework then injected candidate solutions and compared each test verdict with the benchmark's official checker. Three pools were used: the trusted reference only, the reference plus solutions written by other models, and the reference plus mutants, copies with deliberate bugs that the checker confirms are wrong.
Did the AI-written tests catch bugs?
The tests nearly always ran: on QHack, executability was at least 0.79 for every model and 1.00 for three of them. But across both benchmarks, models were better at accepting correct code than at rejecting incorrect code.
With only correct code, Gemini 3 Pro accepted almost all of it. Once wrong code was added, bug detection reordered the ranking: Claude Opus 4.6 reached the best QHack test quality, 0.71, while Gemini's lead did not survive. On Qiskit HumanEval the best was only 0.34, from Gemini, with wide variation between tasks.
Coverage was no guarantee: the Coffee challenge tests included nearly all checker inputs, 0.97, yet scored only 0.13. On three of ten Qiskit tasks no model wrote a wrong solution, and mutants made bug detection measurable there.
Why do tests that only run fall short?
A suite that runs green says little until it has been seen to fail something. The authors' case studies show two mirror-image failures: one suite accepted every correct answer and every wrong one, while another rejected the correct human reference. Both end with a test quality of zero, which is why acceptance and rejection need to be reported separately.
On QHack, results with synthetic mutants agreed closely with results from real model mistakes, suggesting that planted bugs are a fair stand-in when natural wrong solutions are scarce.
A fair reading for anyone relying on AI-written tests is to plant a few verified bugs on purpose and check which ones the tests catch, rather than trusting a suite because it runs.
What are the limits of this quantum test benchmark?
Read before you quote it
This is a preprint that has not been peer reviewed, and the authors list its limits:
- Single pass. Each model got one attempt at a fixed temperature of 0.8, with no repair.
- Small task set. Eight QHack challenges and ten selected Qiskit HumanEval tasks.
- Checker-dependent. Ground truth comes from each benchmark's own checker.
- Pool-dependent. Absolute scores depend on which correct and wrong candidates are in the cohort.
The study does not show how well models write quantum tests when they can repair them using execution feedback, which the authors name as the natural next step, and it does not test quantum software outside these two benchmarks.
- It does not show how well models write quantum tests when allowed to repair them with feedback.
- One generation pass at a fixed temperature of 0.8, with no repair or execution feedback.
- Only ten selected Qiskit HumanEval tasks and eight QHack challenges.
- Ground truth depends on benchmark-specific checker oracles.
- Absolute test quality depends on the mix of probes in each cohort.
Takeaways
- saysAI-written quantum tests almost always run, but they're better at accepting correct code than rejecting wrong code.
- saysThe best test quality was about zero point seven one on QHack and only about zero point three four on Qiskit HumanEval.
- saysPlanted bugs, called mutants, made bug detection measurable where real wrong solutions were missing.
- explainsMutation testing plants deliberate bugs in correct code to check whether a test suite catches them.
- interpretsA fair reading is that tests which run are easy to get, while tests which tell right from wrong are not.
Ask the paper
In PaperSays you can ask this paper anything, by text or by saying “Hey Aiden”. Three questions listeners asked:
Can language models write good tests for quantum code?
Their tests usually run, but in this benchmark they were better at accepting correct code than catching wrong code. Hear the full answer in the app →
What is a test oracle?
Something that decides whether an output is correct; here, each benchmark's official checker. Ask in the app →
Why use synthetic mutants?
When real wrong solutions are scarce, planted bugs let you measure whether tests catch errors. Ask in the app →
Briefing reviewed by an editor before publication. The paper belongs to its authors; this page summarises it and links to the original. Paper licence: CC BY 4.0. Corrections: support@papersays.com.