모질라, AI로 취약점 271개 발견 "오류 거의 없어"

Text settings Story text Size Small Standard Large Width * Standard Wide Links Standard Orange * Subscribers only Learn more Minimize to nav The disbelief was palpable when Mozilla’s CTO last month declared that AI-assisted vulnerability detection meant “ zero-days are numbered ” and “defenders finally have a chance to win, decisively.” After all, it looked like part of an all-too-familiar pattern: Cherry-pick a handful of impressive AI-achieved results, leave out any of the fine print that might paint a more nuanced picture, and let the hype train roll on. Mindful of the skepticism, Mozilla on Thursday provided a behind-the-scenes look into its use of Anthropic Mythos—an AI model for identifying software vulnerabilities—to ferret out 271 Firefox security flaws over two months. In a post , Mozilla engineers said the finally ready-for-prime-time breakthrough they achieved was primarily the result of two things: (1) improvement in the models themselves and (2) Mozilla’s development of a custom “ harness ” that supported Mythos as it analyzed Firefox source code. “Almost no false positives” The engineers said their earlier brushes with AI-assisted vulnerability detection were fraught with “unwanted slop.” Typically, someone would prompt a model to analyze a block of code. The model would then produce plausible-reading bug reports, and often at unprecedented scales. Invariably, however, when human developers further investigated, they’d find a large percentage of the details had been hallucinated. The humans would then need to invest significant work handling the vulnerability reports the old-fashioned way. Mozilla’s work with Mythos was different, Mozilla Distinguished Engineer Brian Grinstead said in an interview. The biggest differentiating factor was the use of an agent harness , a piece of code that wraps around an LLM to guide it through a series of specific tasks. For such a harness to be useful, it requires significant resources to customize it to the project-specific semantics, tooling, and processes it will be used for. Grinstead described the harness his team built as “the code that drives the LLM in order to accomplish a goal. It gives the model instructions (e.g., ‘find a bug in this file’), provides it tools (e.g., allowing it to read/write files and evaluate test cases), then runs it in a loop until completion.” The harness gave Mythos access to the same tools and pipeline that human Mozilla developers use, including the special Firefox build they use for testing. He elaborated: With these harnesses, so long as you can define a deterministic and clear success signal or task verification signal, you can just keep telling it to keep working. In our case when we’re looking for memory safety issues we have our sanitizer build of Firefox and if you make it crash you win. We point that agent off to a source file and say: “we know there’s an issue in this file, please go find it.” It will craft test cases. We have our existing fuzzing systems and tools to be able to run those tests. It will say: “I think there’s an issue here if I craft the HTML exactly so.” It sends it off to a tool, the tool says yes or no. If the tool says yes then there’s some additional verification. The additional verification comes in the form of a second LLM that grades the output from the first LLM. A high score gives developers the same confidence they have when viewing reports generated through more traditional discovery methods. “In terms of the bugs coming out on the other side, there are almost no false positives,” he said. Thursday’s behind-the-scenes view includes the unhiding of full Bugzilla reports for 12 of the 271 vulnerabilities Mozilla discovered using Mythos and, to a lesser extent, Claude Opus 4.6. The test cases—meaning the HTML or other code that triggers an unsafe memory condition—are provided in each one and meet the same criteria Mozilla requires for all bugs to be considered security vulnerabilities in Firefox. At least one researcher said Thursday that a cursory look at the reports showed they were “ pretty impressive .” Unlike previous vulnerability disclosure slop, Grinstead said, the details provided by its harness-guided Mythos analysis, and confirmed by the second LLM, and ultimately included in the reports, provide a level of confidence his team didn’t have before. “That’s the key thing that has unlocked our ability to operate at the scale we’ve been operating at now,” he said. “It gives the engineer a crank they can pull that says: ‘Yep, this has the problem,’ and then you can iterate on the code and know clearly when you’ve fixed it and eventually land the test case in the tree such that you don’t regress it.” As noted earlier, Mozilla’s characterization of AI-assisted vulnerability discovery as a game changer has been met with massive, vocal skepticism in many quarters. Critics initially scoffed when Mozilla didn’t obtain CVE designations for any of the 271 vulnerabilities. Like many developers, however, Mozilla doesn’t obtain CVE listings for internally discovered security bugs. Instead, they are bundled into a single patch. Normally, Bugzilla reports detailing these “rollups” are hidden for several months after being fixed to protect those who are slow to patch. Now that Mozilla has revealed a dozen of them, the same critics will surely claim they too were cherry-picked and conceal less accurate results. Of the 271 bugs found using Mythos, 180 were sec-high, Mozilla’s highest designation for internally reported vulnerabilities. These types of vulnerabilities can be exploited through normal user behavior, such as browsing to a web page. (The only higher rating, sec-critical, is reserved for zero-days.) Another 80 were sec-moderate, and 11 were sec-low. The critics are right to keep pushing back. Hype is a key method for inflating the already high puffed-up valuations of AI companies. Given the extensive praise Mozilla has given to Mythos, it’s easy for even more trusting people to wonder: What’s it getting in return? Far from settling the debate, Thursday’s elaborations are likely to only further stoke the controversy. To hear Grinstead tell it, however, the details are clear evidence of the usefulness of AI-assisted discovery, and Mozilla’s motivation is simple. “People are a bit burned from the last year of these slop commits so we felt it was important to show some of our work, open up some of the bugs, and talk about it in a little more detail as a way to hopefully spur some action or continue the conversation,” he said. “There’s no sort of marketing angle here. Our team has completely bought in on this approach. We are trying to get a message out about this technique in general and not any specific model provider, company, or anything like that.” Dan Goodin Senior Security Editor Dan Goodin Senior Security Editor Dan Goodin is Senior Security Editor at Ars Technica, where he oversees coverage of malware, computer espionage, botnets, hardware hacking, encryption, and passwords. In his spare time, he enjoys gardening, cooking, and following the independent music scene. Dan is based in San Francisco. Follow him at here on Mastodon and here on Bluesky. Contact him on Signal at DanArs.82. 47 Comments