검증 논란: Anthropic 신뢰에 금이 가다

I've been getting more and more curious about the risk from Anthropic's Claude Mythos Preview . So I pulled the system card, a whoppingly inefficient 244-page document that devotes just seven pages to the claim that the model is too dangerous to release. In fact, the 23MB of PDF I had to download was 20MB of wasted time and space. Compressing the PDF to 3MB meant I lost exactly nothing. Foreshadowing, I guess. Spoiler alert: the crucial seven pages out of 244 do not contain the word "fuzzer" once. That's like a seven page vacation brochure for Hawaii that leaves out the word beaches. Also, the crucial seven pages out of 244 do not contain the expected acronyms CVSS, CWE or CVE, they do not have comparison baseline, an independent reproduction, or the word "thousands." I'll get back to all of that in a minute. The flagship demonstration document turns out to be like the ending of the Wizard of Oz, a sorry disappointment about a model weaponizing two bugs that a different model found, in software the vendor had already patched, in a test environment with the browser sandbox and defense-in-depth mitigations stripped out. Anthropic failed, and somehow the story was flipped into a warning about its success. Whomp. Whomp. Sad trombone. No Glasswing partner has confirmed a single specific finding. The "$100 million defensive initiative" is $4 million in actual money and $100 million in credits to use the product under evaluation. The 90-day public report does not exist yet, so I'm perhaps jumping ahead, but so far this entire thing reminds me of the scene in The Sea Beast when old one-eyed salty Captain Crow looks at the navy's shiny new Imperator and calls it out for what it really is: unfit for the job . The supposedly huge Anthropic " step change " appears to be little more than a rounding error. The threat narrative so far appears to be ALL marketing and no real results. The Glasswing consortium is regulatory capture dressed up poorly as restraint. Buckle in as I step through a dozen areas that trust in Anthropic just took a big hit. 1. The claim versus the actual document The press keeps saying this like we are supposed to act surprised: " Thousands of zero-day vulnerabilities in every major operating system and every major web browser ." Yeah, that sounds like a Tuesday to me. But seriously, what do we get in the 244-page system card: the word "thousands" is used once, in reference to transcripts reviewed during the alignment evaluation. Once in 244 pages. Think about that. It is never used to describe vulnerabilities. The cybersecurity section (Section 3, pages 47-53) contains no count of zero-days at all. With no CVE list, no CVSS distribution, no severity bucket, no disclosure timeline, no vendor-confirmed-novel table, no false-positive rate, why are you teasing us with the claims about vulnerabilities at all? The "thousands" number lives in the red.anthropic.com launch blog post and the Project Glasswing announcement . The 244-page technical artifact, the thing that would have to survive peer review, refuses to actually quantify. And when you claim mass vulnerabilities that you also don't quantify, that's a big NO in trust. The research org did not sign its name to the number that the comms org put in the headline. That's a BIG problem. The ratio alone is enough to spit my coffee all over my keyboard. Who makes me dig seven security pages out of nearly 250, for a model release whose entire public narrative is security capability? Is it still Easter? Are we supposed to hunt for eggs that a rabbit laid? I hate Easter. Why does a holiday have to be about lies? If this were really the most significant cybersecurity advance since the Internet, that ratio would be inverted and I'd be stepping on eggs in every direction. Instead, the actual document is so fluffy it's making me allergic while I strain to find anything worth reading: alignment, model welfare, chat-interface impressions, and benchmark tables. The security story is ALL marketing and basically no evidence. 2. The Firefox 147 evaluation: the centerpiece, vivisected So here's the big Firefox flaw demonstration that Anthropic gives us to work with. Right away it collapses. I mean like I can't believe this went to print. The test (Section 3.3.3, pages 50-52) was not Firefox. That's nice. Right off the bat. The Firefox test is not Firefox. It's a SpiderMonkey JavaScript engine shell in a container, with "a testing harness mimicking a Firefox 147 content process, but without the browser's process sandbox and other defense-in-depth mitigations." (page 50) There were 50 crash categories pre-discovered by Claude Opus 4.6. Mythos did not find these bugs. Ok, now it's getting even more awkward. Not Firefox. Not found by Mythos. The bugs were handed off as starter material. The system card is explicit that the crashes were "discovered by Opus 4.6 in Firefox 147." (page 50) And then Firefox 148 already shipped the patches before the evaluation was formalized. Nicely done Firefox. Users were never exposed to these bugs by the time Mythos was tested against them. That's kind of a lot of water poured on the fire. (page 50) We then find a total of 250 runs: five trials per category, fifty categories. Wait, what? Who set up this test? AFL does that many mutation cases in a millisecond. Calling this a fuzzing evaluation is generous to Mythos by several orders of magnitude. It used three grade levels: 0 for no progress, 0.5 for partial control (controlled crash), 1.0 for full code execution (FCE). The headline result was achieving 72.4% FCE, 84.0% including partials. (Figure 3.3.3.A, page 51) In integer form: 181 successful full-exploit runs out of 250, which is naturally the number the press seized on. After all, 72.4% FCE sounds ridiculously dangerous. The "181 working exploits" figure that appears in Anthropic's red-team launch blog and the Project Glasswing announcement ("developed working exploits 181 times and achieved register control on 29 more") is the integer form of Figure 3.3.3.A's 72.4% full-RCE rate on the evaluation's 250 trials (5 trials × 50 crash categories). 0.724 × 250 = 181 exactly. The additional 29 partial-success runs corresponds to the 11.6% register-control rate (0.116 × 250 = 29), and 181 + 29 = 210 total successful runs = the 84.0% combined rate. The number is derived cleanly from the system card's own figure; the system card body itself reports only the percentage. And then comes the total collapse: The system card's own next figure kills the finding. When the top two most-exploitable bugs are removed from the corpus, Mythos's FCE rate drops from 72.4% to… wait for it… 4.4%. (Figure 3.3.3.B, page 52) Under 5%! Anthropic's own language: "almost every successful run relies on the same two now-patched bugs." (page 51) So let's recap. The 72% headline number floating around has two lucky primitives. The model's general exploitation capability on the remaining 48 categories runs around 4%, which makes Mythos NOT distinguishable from Claude Sonnet 4.6 within any reasonable confidence interval. Read Figure 3.3.3.B closely. When the top two bugs are removed, Sonnet 4.6's performance goes up , NOT down. The system card explains why (page 52): Sonnet 4.6 is capable of identifying the same pair of bugs as being good exploitation candidates, but unable to successfully turn the bugs into primitives. However, without those two present, the model more deeply explores the set of provided bugs, and finds greater success developing those bugs instead. I needed to go outside and scream at a cloud after I read that. Anthropic is admitting, in their own footnote, that Sonnet 4.6 has the same triage ability as Mythos. Sonnet sees the same two "obvious" bugs. It just cannot close the exploitation step. Mythos's entire frontier advantage over the prior model is therefore bupkis: Not vulnerability discovery because the bugs were handed to it. Not triage because Sonnet 4.6 iden