Benchmarking Hypervex against real CVEs

Every security tool claims it catches vulnerabilities. The honest way to test that claim is against vulnerabilities that actually shipped: the OpenSSF CVE Benchmark — hundreds of real CVEs from real open-source projects, each with the vulnerable commit and the fix. We published our first run in July and promised the follow-up either way. This is it, misses included.

The headline number

On Claude Sonnet 5, base review only — no codebase context, no scanner fusion, no investigation pass — Hypervex scores 72.4% F1 on a 30-CVE sample: 72.4% precision, 72.4% recall. The original Gemini 2.5 Flash baseline was 47.1%. Same dataset, same sample, same scoring rule — the only changes are the model and four prompt fixes.

The false-positive rate on patched, already-safe code fell from 56.0% to 27.6%. On a security reviewer that number matters more than the headline: a tool that cries wolf on fixed code gets muted, and a muted tool catches nothing.

Why there is no competitor column

Other tools publish F1 scores on this same benchmark, and it is tempting to put them side by side. We are not going to, because the numbers are not comparable. Ours uses strict scoring: a finding counts only if the CWE number matches or it lands within ±15 lines of the documented weakness. Published figures elsewhere are typically judged by an LLM on semantic equivalence, which credits a finding that describes the same bug in different words.

Those two rules can differ by more than the gap between two tools — so a side-by-side table would mostly measure the scoring rule, not the reviewers. An honest comparison needs one judge over one dataset with both tools scored identically. That is a separate piece of work, it is underway, and we will publish it when it holds up — including if it goes against us.

Methodology

For each CVE we build two pull requests from the two commits: a vulnerable diff (the fix reverted) and a patched diff (the fix applied). Hypervex reviews both. A finding counts as a true positive if its CWE number matches the CVE's CWE, or it lands in the documented weakness file within ±15 lines. A finding on the patched, safe diff counts as a false positive. We sampled 30 CVEs (fixed random seed, drawn across CWE categories). Both runs above use this identical rule, sample and seed — only the model and prompt differ, which is what makes the before/after a fair comparison.

What it caught well

Classic injection and access-control flaws are the consistent strength across both runs: path traversal (CWE-22), command injection (CWE-78), code injection / sandbox escape (CWE-94), XSS (CWE-79), CSRF (CWE-352), open redirect (CWE-601), and unbounded resource allocation (CWE-770). Where a vulnerability has a recognisable dangerous sink, Hypervex is reliable.

What it missed

The first run had three blind spots. Two are addressed; the third is honest about needing more than a diff:

• Cryptographic weakness (CWE-327 broken crypto, CWE-338 weak PRNG) — missed both. The review prompt had no cryptography section at all. It has one now.
• Denial-of-service / ReDoS (CWE-400/399) — a catastrophic-backtracking regex has no injection sink, so it fell outside the model's idea of "a vulnerability." The prompt now evaluates the pattern itself even when the diff does not show where the input comes from.
• Protocol & logic flaws (HTTP request smuggling CWE-444, SSRF CWE-918) — these need multi-step data-flow reasoning across files, which a base review of a diff cannot do however the prompt is written. This is the gap the Trace exists to close, and it is not reflected in the numbers above: every figure here is base review only.

One honest wrinkle in our favour: several "misses" were cases where Hypervex flagged the same real vulnerability under a more specific CWE than the dataset's generic tag — for example, correctly calling out prototype pollution (CWE-1321) on a CVE tagged with the broader CWE-94. Strict number-matching scores those as misses, so the real detection rate is somewhat higher than the 72.4% recall above. We report the strict number anyway.

The finding that surprised us: confidence didn't separate signal from noise

Hypervex attaches a 0–100 confidence score to every finding, and the bet was that high confidence would concentrate the true positives. On the Gemini baseline it did not: true positives averaged 94% confidence, false positives on fixed, safe code averaged 91%, and nearly every false positive cleared the 80% bar. Confidence was a near-constant high number — decorative, not a triage signal.

This is the number that moved most. On Sonnet 5 the gap between true-positive and false-positive confidence widened to +43.6 points, and findings at or above 80 confidence are 92.9% precise. That is what makes the score worth showing you — and it is the mechanism behind the Trace, which escalates only above that threshold. Selective escalation is only affordable if the threshold means something.

What's next

Everything above is the base review working alone. The shipped pipeline adds three things this benchmark never exercised: scanner fusion, retrieval over your indexed codebase, and the Trace. Measuring those honestly needs a harder dataset than 30 CVEs judged on line proximity, which is the next piece of work — along with the like-for-like cross-tool comparison described above. Both get published, higher or lower. The runners are deterministic and the dataset is public, so anyone can check the arithmetic.