The fix is usually known. Getting it in safely is the whole job.
Snyk’s numbers make the gap hard to argue with. Over six months in 2026, Snyk Open Source recorded 459 million vulnerability findings across roughly 300,000 organizations. 94% of them already had a known fixed version. Only about 10% had been resolved. When Snyk built a benchmark to test whether AI agents fix vulnerable dependencies any better than the manual process does, that spread between “fix is known” and “fix is applied” was the problem it was aimed at.
So the idea that teams leave vulnerabilities open because nobody knows which version fixes them does not hold up. The fixed version is usually sitting right there in the advisory, listed alongside the CVE it closes. What stalls the work is shipping the bump, and the bump almost never lands where you expect. Most vulnerable packages arrive transitively. You did not choose the package, you chose something that chose it, and the fix advice points at a line in your lockfile that nothing in your import statements mentions. On a WordPress site running WooCommerce plus a handful of add-ons, you are trusting each plugin author to have bumped the library underneath, and you have no clean way to override it without patching the plugin yourself. About a quarter of the fixes in this dataset need a major version bump, and minor and patch releases break builds more often than the version number promises.
That is the setting for VulnBump (opens in new tab), published in October 2026. It measures 307 real upgrades drawn from 43 repositories across six package managers: Maven, Gradle, npm, Yarn, pip and Poetry. An attempt passes only if the vulnerability is verifiably gone, the project still installs, and every test that passed before still passes. Four gates, checked from the agent’s diff in a clean container, with nobody taking the agent’s word for it.
The part I keep returning to is the spread. All three agents ran on the same model, Claude Sonnet 5.5, with the same 110-turn budget, and their pass rates ranged from 59.6% to 90.2%. Nothing about the model changed between those runs. What changed was the information and the tooling each agent was handed. That is a tooling result wearing model clothing, and it should shape where a team spends its remediation effort before it spends anything on a new model.
Background: why remediation stalls after the scan finishes
Part of the reason is that version numbers lie to us in a specific, predictable way. A caret range like ^1.4.2 is a promise that nothing below 2.0.0 will break your code. Maintainers break that promise often, usually without meaning to, by changing how a function behaves on malformed input or by dropping a Node version you still run. In the VulnBump dataset, roughly a quarter of the recommended fixes require a major version bump, which is at least an honest signal that something might break. The minor and patch bumps are the ones that catch people out, because you apply them in a routine dependency refresh and find out three deploys later.
The second reason is that the vulnerable package rarely belongs to you. Run npm ls qs on a mid-sized Node project and you will get a tree of paths, something like express > body-parser > qs, where the package you actually import is two levels above the thing with the CVE. The advisory tells you to move qs, and you cannot move it, because express pins it. Your real options are a major upgrade of the framework, or an overrides block in package.json that forces a version the parent library was never tested against. Both are decisions with consequences, and neither is a one-line change. On the WordPress side the same shape appears: your WooCommerce add-on bundles a PHP library with a known flaw, and you have no supported way to override it short of patching the plugin and re-patching it after every update.
Snyk has been circling this problem for a while. VulnBench, their June 2026 security review benchmark, found that Claude and Snyk Code tend to miss different vulnerabilities and concluded the two work better combined. An earlier set of remediation results in May ran on an older model and covered fewer ecosystems, so the numbers were never going to transfer cleanly.
What makes VulnBump worth reading separately is the pass condition. Earlier upgrade benchmarks only checked whether the project still built after a bump. An agent could downgrade a package, edit the lockfile cosmetically, or change nothing at all and still score well, because nothing rescanned for the vulnerability. VulnBump requires the CVE to be gone and the build green, which turns a build-stability test into a security test. That is a harder bar, and it produces very different numbers.
What’s happening now: how AI agents fix vulnerable dependencies on the VulnBump benchmark
The dataset was assembled in stages. Snyk pulled 767 candidate repositories off GitHub, mixing popular open source projects with repos that deliberately pin known-vulnerable packages. Each candidate had to install, build, and pass better than 85% of its own tests before anyone touched it, and Snyk had to find at least one vulnerability inside. That eliminated most of the pile and left 99. From those, 43 were selected to mirror Snyk’s production data: the same spread of ecosystems, severity levels, direct versus transitive dependencies, and size of version jump.
Those 43 repositories yield 307 tasks covering 2,066 fixable vulnerabilities across Maven, Gradle, npm, Yarn, pip, and Poetry. The six package managers account for roughly 92% of production findings, so the sample is not a random slice of GitHub.
Each task is narrow by design: one package, moved to one version that fixes its vulnerabilities. The scoring is where this gets serious. When an agent declares it is finished, its diff is copied into a clean container and rebuilt from nothing by a verifier that takes no part of the agent’s story on trust. Four gates, and all four have to pass. The dependencies have to install using the same command CI would run. A fresh Snyk scan has to confirm the vulnerabilities are actually gone. The project has to build. Every test that passed before the change still has to pass after it. Anything short of that is a fail.
The uses all ran on Claude Sonnet 5.5 with a 110-turn limit. Snyk’s Evo Remediation Agent passed 90.2% of upgrades. Claude Code, given Snyk’s recommended target version plus the ability to run its own scan, passed 77.5%. The same model with no Snyk data at all, left to work out the fixing version by searching vulnerability databases, passed 59.6%. And the non-AI baseline, a plain package manager applying Snyk’s version and nothing else, passed 46.9%.
Build stability was not the differentiator. All three agents kept builds green around 93% of the time. The separation came entirely from whether the vulnerability was gone, and the failure pattern is worth sitting with. In 70 of its 124 failed tasks, unaided Claude Code reported that it had fixed vulnerabilities that a rescan still found. Those are the false successes the clean container exists to catch, and they line up with what the VulnBump research (opens in new tab) describes as the core difficulty of the task.
Do AI agents fix vulnerable dependencies without breaking the build?
The benchmark’s title asks one question. In practice the data answers two, and they diverge sharply.
Can AI agents fix vulnerable dependencies without breaking the build?
Mostly yes, and more consistently than I expected. Every agent in the run kept the build passing around 93% of the time, including the unaided one. But a PASS here is not “the build survived.” It requires a clean rescan showing the vulnerability is gone as well. Once you insist on both, the rates spread out across more than 30 points. An agent that refuses to touch anything and returns a green build is 93% of the way to a passing build and nowhere near a passing fix.
Does the model choice decide the outcome?
No, and this is the finding that should reframe how teams buy tooling. All three agents ran the same model, Claude Sonnet 5.5, with the same 110-turn budget. Unaided, it scored 59.6%. Given Snyk’s recommended target version and a scanner it could execute itself, the same model scored 77.5%. Same weights, same turns, a 17.9-point swing. Whatever you believe about model capability, the information and feedback loop around the model accounted for more of the result than the model did.
Is a non-AI baseline pointless?
Not remotely. Applying Snyk’s recommended version with the project’s own package manager, no agent involved, passed 46.9% of tasks. That baseline beats unaided Claude Code on this benchmark. For a site owner with a handful of straightforward direct dependencies to bump, the scripted path is not a joke, and any agent tooling has to clear that bar before it earns its inference cost.
What is the single most common failure mode?
An agent reporting success on a vulnerability it did not remove. Unaided Claude Code claimed a fix in 70 of its 124 failures, a false success rate above half. Nothing in the agent’s own summary distinguishes that case from a genuine fix. Only the clean-container rescan does.
One caveat worth holding onto: Snyk Open Source acted as the judge, and the run used a single model with each upgrade attempted once. A second judge, or a second model from a different family, might shift these numbers, and I cannot say from the published data which direction.
What it means in practice on your repo
The cheapest change you can make costs nothing and does not involve picking a different model. When you open an agent session to fix a CVE, paste in the Snyk recommendation: the package, the target version, and the advisory ID it closes. Then leave the scanner available so the agent can run it against its own work. That is the entire delta between the 59.6% and 77.5% runs in VulnBump (opens in new tab). No fine-tuning, no new vendor, no migration. I have watched teams spend a week evaluating agent frameworks while skipping this step, which is roughly like hiring a locksmith and not telling them which door.
The second change matters more than the first. Treat the agent’s summary as a request, not a receipt. Unaided Claude Code claimed success in 70 of its 124 failures, and the only thing that caught it was re-running the scan in a fresh container built from the agent’s diff. On a small VPS host with a handful of Node apps, that looks like this: the agent edits package.json and package-lock.json, says the CVE is resolved, and you push. What it actually did was bump a direct dependency while a transitive copy of the vulnerable package stayed put two levels down. npm ls lodash tells a different story than the agent did. The rescan tells the same story louder.
The third change is about where the risk concentrates. Most vulnerable packages arrive transitively, and about a quarter of fixes require a major version bump. When the recommended target is two major versions ahead of what your app actually imports, do not let the agent apply it and walk away. Pull that upgrade into its own branch, in its own deploy, and watch it for a day before the next one goes out. Two CVEs fixed in one commit with a broken build is a worse week than one CVE fixed slowly.
I expect scanner-in-the-loop tooling to become the default shape of these agents over the next year or so, because the benchmark makes the feedback loop, not the model, the thing that moves numbers. But that is a guess about vendors, not a reason to wait.
Before your next agent session, add one step to CI: rebuild from the proposed diff in a clean container, run the scanner, and fail the job if the CVE you set out to fix still appears in the results.