Cybersecurity
CISO-as-a-service consulting: posture, remediation roadmap, ongoing support.
Discover →
Linux Services & Systems
Domains, hosting, PEC, email infrastructure, network services and Linux systems. Open Source infrastructure support and management.
Discover →
Off-by-1 Labs, 1Password’s security research group, published on 6 August a measurement of what patches written by models are worth. It generated 6,080 patches against six recently disclosed vulnerabilities and classified the outcome of each. The study is by Keith Hoodlet, peer reviewed by Casey Ellis, Jason Haddix and Mike Shema.
How it was measured
The six vulnerabilities are real and named: CVE-2026-31431, privilege escalation in the Linux kernel; CVE-2026-34197, remote code execution in ActiveMQ; CVE-2026-8512, use-after-free in Chrome; CVE-2026-45185, RCE in EXIM; CVE-2026-22738, SpEL RCE in SpringAI; and GHSA-wpqr-6v78-jr5g, RCE in the Gemini CLI.
Two models were used, ChatGPT-5.5 and Opus 4.8, both in their cyber-capable versions. For each vulnerability and each model, 540 patches were generated in sets of 20 under varied conditions. Of the 6,480 produced, 400 were dropped because the model itself flagged them, leaving 6,080 in the final analysis.
Outcomes are classified into five scenarios, verbatim: “Complete fix; does not alter application behavior”, “Complete fix; alters application behavior”, “Does not fix the vulnerability”, “Complete fix of the old vulnerability while adding a new vulnerability”, “Does not fix the vulnerability while adding a new vulnerability”.
The numbers
| Outcome | Share |
|---|---|
| Fixes the flaw without altering behaviour | 26.0% |
| Fixes the flaw while altering behaviour | 20.1% |
| Fails, or introduces a new vulnerability | 53.9% |
The study’s sentence on the first row is “The average success rate for generating a patch that fully resolved the vulnerability… was just 26.0%”. On the last: the patches “did not resolve the vulnerability, added a new vulnerability, or both, an average 53.9% of the time”.
Fragile patches inside the successful ones
The study also examines what is inside the patches classified as successful. More than 33% of those classified S1 and S2, the ones that do fix the flaw, contain “subtleties that we would qualify as ‘fragile’ from a security context”.
Fragile has a precise meaning here: the patch puts a narrow check in place of fixing the cause. On the SpringAI case the models frequently “simply escaped specific characters in user input”, blocking the proof of concept and leaving the underlying vulnerability standing.
A patch like that passes the tests that use the known exploit. It fails on the first variant.
Finding and fixing are different capabilities
In the official CVE records, eight of the nine vulnerabilities JFrog fixed in Artifactory, after the incident reconstructed at Black Hat, are credited to OpenAI researchers, and the models under evaluation had found unknown ones.
The Off-by-1 study measures the other half of the cycle, and returns 26.0%.
The two are not in contradiction. Finding a vulnerability means producing an input that breaks something, and verification is immediate: either the exploit works or it does not. Fixing it means understanding why the code breaks, telling the symptom from the cause and not moving the problem elsewhere, and verification stops being binary, because a patch can fix and break at the same time. The 20.1% that fixes while altering behaviour is precisely that zone.
What we think
The 26.0% depends on the two models and the six vulnerabilities chosen, and it will change with the next generation. The 33% of fragile patches inside the successful ones describes instead a way of being wrong that stays the same, because it does not show up in the test result.
A pipeline that accepts a patch once the known exploit stops working classifies those patches as good. Re-running the same proof of concept returns the same verdict on a fragile patch and on a solid one. The check that separates them is where the fix acts, on the cause or on the input, and that reading of the diff is done today by a person.
A patch generated by a model is a proposal from an outside contributor, so it goes through the same review rather than a fast lane because “the AI wrote it”. And the acceptance criterion moves from the proof of concept to the vulnerability class: if the fix filters characters, the question is what happens with the next encoding.
The conclusion the study states is: “human expertise still plays an essential role in the process of fully resolving vulnerabilities in software without introducing unwanted side effects”. It is 1Password’s position, and 1Password sells security, so it should be weighed as such. The 6,080 samples, the six named CVEs and the five declared scenarios make it a checkable position: the same classification can be run again on other vulnerabilities and other models.
Sources
- 1Password, Off-by-1 Labs: why AI-generated patches still require human review
- Dark Reading: AI-generated patches fail half the time
- Help Net Security: three in four AI-generated vulnerability patches leave something broken
- eSecurity Planet: 1Password finds AI security patches fail more than half the time