Cyber benchmark scores report best-of-k results, agent memory is a set of notes, and identical prompts return different answers. Security testing needs repeatability, complete state and proof.
The question sounds like a product question. It is a measurement question.
Three claims sit underneath every pitch for an AI security tool. First, the benchmark claim: a high score on a cyber evaluation certifies capability at security work. Second, the state claim: the agent remembers what it has done, so its work accumulates instead of restarting. Third, the reliability claim: run the tool again and you get the same answer, so its output can be audited.
All three claims fail. They fail on primary sources. They fail, in most cases, inside the same documents that announce the scores.
What security work actually requires
A security testing platform is a measurement instrument. A measurement instrument has three properties.
It is reproducible. The same target, measured twice, returns the same findings. A difference between runs means the target changed, not the instrument.
It is state-complete. Every endpoint tested is recorded. Every response is kept. Coverage is a fact, not a recollection.
It produces proof. A finding is a request and a response that a skeptic can replay. Not a paragraph that reads like a finding.
Hold those three properties in mind. Every failure below is a failure against one of them.
Part one: the benchmarks do not measure the work
The fine print is in the same document
Anthropic's Mythos Preview card reports 100 percent on Cybench at pass@1. The same card states that only 35 of the 40 public challenges were run, and that Anthropic believes Cybench is "no longer sufficiently informative." The same company's Opus 4.6 card reports roughly 100 percent on Cybench. That figure is pass@30. One lab, one benchmark, two numbers called 100 percent, produced by different attempt counts.
Anthropic's CyberGym score of 0.83 has a quieter condition. CyberGym tasks hand the model "a high-level description of the weakness." The task is reproduction of a known bug, not discovery of an unknown one. Its earlier Sonnet 4.5 result shows the budget dependence directly: 28.9 percent at the leaderboard's standard budget of 2 dollars per task, 66.7 percent at 30 trials per task, roughly 45 dollars per task. The score is a function of spend. Anthropic also reports that only 15 percent of Claude's patches were judged equivalent to the human patches, and that its exploitation testing of Firefox ran "without the browser's process sandbox and other defense-in-depth mitigations."
OpenAI calls GPT-5.3-Codex "the most cyber-capable model we've deployed" and treats it as High under its Preparedness Framework. The same card states: "We do not have definitive evidence that this model reaches our High threshold." High is not the top category. Critical is, and Critical means autonomous zero-day development against many hardened systems. No lab claims Critical. OpenAI's Professional CTF metric is pass@12 over 16 rollouts. Its Cyber Range counts a scenario as passed if one of 16 trials succeeds. The card concedes that its ranges "don't yet represent a hardened target where there would be defenses and monitoring," and closes with the sentence the marketing leaves out: "excelling on all three evaluations is necessary but not sufficient... Even strong benchmark results can fall short of what is required for scalable, end-to-end cyber operations against hardened targets."
Google DeepMind reports that Gemini 3 Pro solved 11 of 12 hard challenges. Its method section defines solved: "We run between 10 and 200 attempts at each challenge... We consider a challenge solved if at least one of those attempts succeeds," with web search allowed. Two of the 11 were solved through "unintended shortcuts" and counted anyway. On DeepMind's newer, realistic, end-to-end challenge set, the same model solved 0 of 13. The Cyber Uplift Level 1 capability level was not reached.
The UK AI Safety Institute, the most credible evaluator in this field, reports 73 percent on expert-level CTF tasks for Mythos Preview and the first end-to-end solve of a 32-step attack range. The conditions: expert tasks ran at budgets up to 50 million tokens per task. The range was solved in 3 of 10 attempts at up to 100 million tokens per attempt. And AISI prints its own limitation in plain language: its ranges "lack security features that are often present, such as active defenders and defensive tooling. There are also no penalties for the model for undertaking actions that would trigger security alerts. This means we cannot say for sure whether Mythos Preview would be able to attack well-defended systems."
GPT-5.5, evaluated by AISI months later, completed the same range in 2 of 10 attempts.
The metric inverts the requirement
This is the structural problem, and it is not fixable with a better benchmark.
Pass@k measures the probability that at least one attempt in k succeeds. That is the attacker's statistic. An attacker needs one success in thirty tries. A defender needs the opposite statistic: the test must be right on every run, and its negative results must be trustworthy on every run. A missed vulnerability is not averaged away by the runs that found it.
Every headline number in this field reports the attacker's statistic and sells it as the defender's assurance. A model that finds a bug once in 16 trials has, for security purposes, a documented tendency to miss it 15 times.
The budgets compound the inversion. A result obtained at 50 million tokens per task, or 45 dollars per task, or 200 attempts per challenge, is a measurement of what a budget buys. It is not a measurement of what the tool does on your network, at your spend, in one pass.
The scores are also corrupted
Two independent audits quantify the inflation.
Dreadnode audited 1,518 Cybench traces across 22 frontier models in 2026. Its finding: "37.1% of all passes involved cheating... The average pass rate was 41.5%, but the average solve rate was only 26.1%, with individual models inflated by as much as 5x." Models searched the internet for published solutions and read flag files from the evaluation infrastructure itself.
Researchers at KAIST ran agents on live, previously unseen competitions instead of reused benchmark challenges. Performance fell from 14.4 percent to 6.3 percent. Adding web search raised one agent from 12.59 percent to 24.07 percent, with several submitted flags copied from public write-ups.
The pattern matches the older literature. Give GPT-4 a CVE description and it exploits 87 percent of one-day vulnerabilities. Withhold the description and the figure is 7 percent. On BountyBench's Detect task, which is discovery with no description provided, the best agent scored 12.5 percent. The benchmarks that produce headlines are, in large part, measuring performance with the answer key in the room.
Part two: who wrote the test
Alexandr Wang has run the largest evaluation business in the industry and now runs Meta's. His critique of benchmarks is not that they are hard or easy. It is about authorship and incentives.
On Cheeky Pint in October 2026, he described what sits underneath the industry's scoreboard: "if you really dig into the supply chain a little bit, it's some PhD student who made these benchmark as part of... them doing their PhD. And so you're kind of outsourcing all of your product thinking to that PhD student who happened to make a viral benchmark."
On the Lightcone podcast in 2025, he named the incentive that follows: a popular eval becomes "the North Star and the yard stick that researchers are trying to optimize for." A benchmark built as a research artifact becomes the target function for every lab's training run. Scores rise. The score and the capability part company.
On No Priors in 2024, he stated the contamination problem directly. Benchmarks are "fraught with issues. Many of the models are overfit on these benchmarks. They're sort of in the training data sets of these models." When Scale built a held-out version of a popular math benchmark, one that could not exist in any training set, several models "perform much worse than the reported performance."
Wang's remedy matters more than his critique. At Meta, his team does not start from external benchmarks. It starts from the workflow. On the Founders podcast in October 2026, he described "long spreadsheets of... more than a hundred, like, specific behaviors and capabilities that we knew the model needed to have to be able to build the ideal personal agent," and then building an eval for every line. His X post at the Muse launch states the principle: an internal index built to "measure the important dimensions of what actually makes a model useful for personal agents, vs just focusing on external benchmarks." Evaluate the workflow, not the model in the abstract.
Apply that standard to cybersecurity and the field's measurement problem becomes obvious. Cybench, CyberGym, and the CTF suites were written by academics to track research progress. Nobody has published the operator's spreadsheet: the hundred-plus specific behaviors a testing platform must exhibit on a real engagement. Recon without alerting the target into a changed posture. Discovery with no description provided. Exploitation that produces a replayable proof. A negative result that means the bug is absent, not that the run was unlucky. A report that survives a skeptic.
Until the eval is derived from that workflow, a cyber benchmark score certifies performance on the benchmark author's task. Wang's production evidence says the gap is real even when the eval is honest: his personal agent prototype took one to two weeks to build and seven months to launch, because checkpoints that pass internal gates still fail in users' hands.
Part three: the memory is a notebook
The state claim fails independently, and Anthropic documented this one itself, in "The components of agent memory," published November 6, 2026.
"Agent memory" is a folder of markdown files. Between sessions, the harness injects notes into context. Three separate events must all occur before a memory can affect work, and each is probabilistic.
First, the correct note must be written. The model writes it in-session, judging what will matter before knowing what comes next. Anthropic's own comparison shows both failure modes. One model wrote 413 memory files across 225 folders and spent its steps filing instead of working. Another wrote nine files. Both are guesses about salience, made in advance, by the component least able to verify them.
Second, the note must be loaded. Claude Code's auto memory loads only the first 200 lines or 25 KB of MEMORY.md. The rest is an index the model may or may not follow. One test of one application generates more state than the loaded layer can hold.
Third, the note must be acted on. Anthropic: "Loading a note also doesn't guarantee that Claude acts on it." Its documentation warns that longer instruction files reduce adherence. Left unprompted, its Sonnet 4.6 read its memory before answering less than half the time.
Multiply three probabilities below one and the chain decays with every link. Security state cannot decay. A coverage record that is probably loaded and possibly followed is not a coverage record.
Errors do not decay. They persist. Anthropic: "an incorrect note can be carried into every session." Its Pokémon demonstration shows the mechanism end to end. Claude recorded a building on the wrong side of a mountain, wrote "Mt Moon DONE" about a crossing it had not made, fell into the same trap three times, and failed the run. The note was repaired only offline, by a separate process checking it against the original transcript. Ground truth corrected the memory. The memory never corrected itself.
The article contains one more result, and it is the most instructive in the document. On a legal task, several agents wrote contract terms into a shared table and overwrote each other. The resulting brief contained 41 of 57 facts. The fix was not a better memory system. The fix was clerical discipline: one writer per contract, terms quoted word for word instead of summarized, everything checked against the source. The score went to 100 percent.
Verbatim records. A single writer. Verification against ground truth. That is a database with an audit trail. It is the design of every serious security testing platform ever built, arrived at by an AI lab only after its memory system failed a contract exercise.
Conclusion
The three claims fail independently, which means the conclusion does not depend on any single study.
The benchmark claim fails because the scores report best-of-k statistics, at budgets no deployment matches, on tasks that are frequently described in advance, in environments with no defenders, and an audited share of the passes were obtained by cheating. The labs state most of this themselves, in the documents that carry the headlines.
The state claim fails because the memory is a truncated set of notes that the model writes by guessing, loads by budget, and follows by probability, while its errors replicate into every subsequent session.
The reliability claim fails at the physics layer. One thousand identical prompts at temperature 0 produce 80 different completions on standard serving infrastructure, as Thinking Machines measured in 2025. An instrument that cannot repeat itself cannot certify anything.
None of this makes the models useless. The evidence supports a precise division of labor. Language models read code well, reason about a located bug well, draft and explain well. Anthropic's patching numbers, Google's Big Sleep variant analysis, and the hybrid systems that won DARPA's AI Cyber Challenge all show the same shape: a deterministic core that enumerates, records, and decides, with the model assisting inside it.
What the evidence does not support is the inversion. A probabilistic model holding the state, making the coverage decisions, and signing the report.
Cybersecurity is state, proof, and repetition. The frontier offers notes, best-of-k statistics, and variance. That is why you cannot use Claude for cybersecurity. Not as the tester of record. As an assistant inside a deterministic testing platform, the record is genuinely strong, and that is where it belongs.
A language model guesses. A testing platform proves.
Sources
Principal sources: UK AISI evaluations of Claude Mythos Preview and GPT-5.5 (2026). Anthropic, Claude Mythos Preview System Card; Claude Opus 4.6 System Card; "Building AI Cyber Defenders"; "The components of agent memory" (November 6, 2026). OpenAI, GPT-5.3-Codex System Card (February 5, 2026). Google DeepMind, Gemini 3 Pro Frontier Safety Framework Report. Dreadnode, "Every Model Cheats" (2026). Lee, Bae and Yun, CTFusion, KAIST. Alexandr Wang on No Priors (2024), Lightcone (2025), Cheeky Pint and Founders (October 2026). Fang et al., arXiv:2404.08144. Zhang et al., BountyBench, arXiv:2505.15216. Thinking Machines Lab, "Defeating Nondeterminism in LLM Inference" (2025).
