--- title: "Can AI write tests that catch bugs in quantum code?" canonical: https://papersays.com/p/can-ai-write-tests-that-catch-bugs-in-quantum-code date: 2026-10-13 modified: 2026-10-13 hub: "AI & Machine Learning" doi: "10.48550/arXiv.2610.13811" license: "CC BY 4.0" --- # Can AI Write Tests That Catch Bugs in Quantum Code? > A test that waves broken code through is worse than no test. This study checks whether AI-written tests for quantum programs can tell right from wrong. Language models increasingly write the unit tests that guard code, but a test that passes broken code is worse than none. QTestBench asked five models to write tests for eight PennyLane QHack challenges and ten Qiskit HumanEval tasks, and scored each verdict against the benchmark's own checker. The tests almost always ran but were better at accepting correct code than rejecting wrong code. The best test quality was 0.71 on QHack and 0.34 on Qiskit HumanEval. ## How did QTestBench check AI-written quantum tests? Quantum programs are probabilistic, depend on numerical tolerances and are judged by special checkers, so a well-formed test can still disagree with the official verdict. Existing benchmarks grade whether models write correct quantum solutions, but not whether their tests can tell correct code from faulty code. In QTestBench, five models (Claude Opus 4.6, DeepSeek V3.2, Gemini 3 Pro, GPT-5.4 and Qwen3-Coder) wrote syntactic, semantic and behavioral pytest modules in a single pass, without seeing the reference solution. The framework then injected candidate solutions and compared each test verdict with the benchmark's official checker. Three pools were used: the trusted reference only, the reference plus solutions written by other models, and the reference plus mutants, copies with deliberate bugs that the checker confirms are wrong. ## Did the AI-written tests catch bugs? The tests nearly always ran: on QHack, executability was at least 0.79 for every model and 1.00 for three of them. But across both benchmarks, models were better at accepting correct code than at rejecting incorrect code. With only correct code, Gemini 3 Pro accepted almost all of it. Once wrong code was added, bug detection reordered the ranking: Claude Opus 4.6 reached the best QHack test quality, 0.71, while Gemini's lead did not survive. On Qiskit HumanEval the best was only 0.34, from Gemini, with wide variation between tasks. Coverage was no guarantee: the Coffee challenge tests included nearly all checker inputs, 0.97, yet scored only 0.13. On three of ten Qiskit tasks no model wrote a wrong solution, and mutants made bug detection measurable there. ## Why do tests that only run fall short? A suite that runs green says little until it has been seen to fail something. The authors' case studies show two mirror-image failures: one suite accepted every correct answer and every wrong one, while another rejected the correct human reference. Both end with a test quality of zero, which is why acceptance and rejection need to be reported separately. On QHack, results with synthetic mutants agreed closely with results from real model mistakes, suggesting that planted bugs are a fair stand-in when natural wrong solutions are scarce. A fair reading for anyone relying on AI-written tests is to plant a few verified bugs on purpose and check which ones the tests catch, rather than trusting a suite because it runs. ## What are the limits of this quantum test benchmark? This is a preprint that has not been peer reviewed, and the authors list its limits: - **Single pass.** Each model got one attempt at a fixed temperature of 0.8, with no repair. - **Small task set.** Eight QHack challenges and ten selected Qiskit HumanEval tasks. - **Checker-dependent.** Ground truth comes from each benchmark's own checker. - **Pool-dependent.** Absolute scores depend on which correct and wrong candidates are in the cohort. The study does not show how well models write quantum tests when they can repair them using execution feedback, which the authors name as the natural next step, and it does not test quantum software outside these two benchmarks. ## Takeaways - **SAYS** AI-written quantum tests almost always run, but they're better at accepting correct code than rejecting wrong code. - **SAYS** The best test quality was about zero point seven one on QHack and only about zero point three four on Qiskit HumanEval. - **SAYS** Planted bugs, called mutants, made bug detection measurable where real wrong solutions were missing. - **EXPLAINS** Mutation testing plants deliberate bugs in correct code to check whether a test suite catches them. - **INTERPRETS** A fair reading is that tests which run are easy to get, while tests which tell right from wrong are not. ## Ask the paper ### Can language models write good tests for quantum code? Their tests usually run, but in this benchmark they were better at accepting correct code than catching wrong code. ### What is a test oracle? Something that decides whether an output is correct; here, each benchmark's official checker. ### Why use synthetic mutants? When real wrong solutions are scarce, planted bugs let you measure whether tests catch errors. ## The paper - Title: QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code - Authors: Meriem Terki, Nouhaila Innan, Minghao Shao, Muhammad Kashif, Alberto Marchisio, Muhammad Shafique - Venue: arXiv (preprint), 9 October 2026 - DOI: [10.48550/arXiv.2610.13811](https://arxiv.org/abs/2610.13811) - Licence: CC BY 4.0 [Read this page on the site](https://papersays.com/p/can-ai-write-tests-that-catch-bugs-in-quantum-code)