AI 모델의 숨겨진 '내면의 생각'을 해킹하는 새로운 기법

Comment Loader Save Story Save this story Comment Loader Save Story Save this story Computer scientists recently discovered a way to extract the hidden “thinking” that frontier AI models perform as they work through complex problems. The findings provide some evidence—although not conclusive proof—that certain Chinese models may have been trained by “ distilling ” reasoning information from US models that was supposedly hidden because of how closely some of their thinking or reasoning patterns seem to match. The researchers have also demonstrated that the method could be used to recover personal information, like passwords and API keys, from a model’s inner reasoning, although this vulnerability has been fixed. “All major frontier model providers we tested share this vulnerability,” says Alexander Panfilov,⁩ a computer scientist at University of Tübingen in Germany who was involved with the work. “It can lead to personal information leakage, and it enables large-scale reasoning distillation attacks.” Panfilov and colleagues from the University of Tubingen, the Max Planck Institute, the AI safety institute MATS Research, and the security company Snyk identified the same issue with frontier models from OpenAI, Anthropic, and Google that are accessed via an application programming interface or API. In a paper laying out the work, the researchers show that the open-weight or downloadable Chinese model Kimi K3 from Moonshot AI produces a strikingly similar output to the hidden reasoning traces—the written-out reasoning steps involved in solving a problem—of Claude Opus 4.8 and GPT 5.6 Sol for certain prompts. Despite the similarities, they note that the work “cannot causally establish distillation.” They found that two other open-weight models, China’s DeepSeek and Inkling from the US company Thinking Machines, did not exhibit this kind of reasoning similarity with Claude Opus. Moonshot AI and Z.ai did not respond to a request for comment by time of publication. Distillation is a well-established, widely used technique for efficiently copying the capabilities of existing models over to new ones, and is especially common in the development of open-weight or fully downloadable models. Lately, however, distillation has become a controversial topic, because of claims that Chinese AI companies use it to essentially copy the best US models. In February, OpenAI told US lawmakers that DeekSeek seemed to have copied one of its models to build a reasoning model called R1. In June, Anthropic told lawmakers that Alibaba had systematically distilled its models in order to build its own, called Qwen. There’s no indication that Chinese AI companies used this specific technique to distill US-based AI models. But Panfilov and collaborators say that using their method would make it possible to distill more information from closed models than previously realized. Mini-Me Models Advanced AI models solve difficult problems by breaking them into constituent parts that are analyzed in turn in a kind of artificial reasoning or “chain of thought.” Companies tend to keep a proprietary model’s reasoning secret to prevent others from using them to train new ones. However, they typically also send an encrypted version of that reasoning to a user’s computer in a way that offloads some computation. The researchers’ attack relies on the fact that most AI companies also provide related models of different sizes. Larger models are more capable but also more computationally expensive to run and more expensive to access. Users may choose smaller, weaker models for certain tasks to lower costs. Panfilov and his colleagues found that feeding encrypted reasoning traces to a smaller version of the same model can reveal the hidden reasoning inside. The smaller models have received less alignment training, meaning that, unlike the bigger ones, they are less likely to refuse to reveal their inner thoughts. “The idea of swapping out messages to a weaker model variant which has the same decryption key but weaker alignment is very cool,” says Florian Tramer, a computer scientist at ETH Zürich in Switzerland who specializes in computer security. “Its definitely becoming an issue.” The same method also revealed secret information including API keys and passwords embedded in reasoning traces captured from a user’s machine. Panfilov and coauthors alerted OpenAI, Anthropic, and Google to the vulnerability last month. Each company has adjusted its API to mitigate the problem. While it is no longer possible to extract private information this way, Panfilov says some reasoning traces can still be uncovered using the same method. Fixing the distillation entirely would require a fundamental overhaul to the way these companies’ APIs work, he says. “We value independent research on our models and have begun building short-term mitigations for the replay behaviors described in the report,” says Michael Aciman, a spokesperson for Anthropic. He adds that the research did not involve recovering encryption keys, accessing Anthropic’s infrastructure, or recovering personal data from its systems. Google and OpenAI both declined to comment. Distillation has become a matter of geopolitical importance in recent months as US and Chinese companies vie for AI supremacy with increasingly powerful models. China hawks claim that the country gains a strategic advantage by distilling US technology to build open-weight models that are less expensive to run. Others, however, argue that distillation is a widely used way to help quickly boost an AI model’s abilities in certain areas. Mark Zuckerberg, CEO of Meta, said in a blog post this week that distillation “is an important principle of how the open source ecosystem works,” and warned restricting the practice would put the US at a disadvantage. Kyle Miller, a researcher at the Center for Security and Emerging Technologies (CSET), a tech policy think tank, says it is unclear how much distillation really helps China. This is because it only enhances the capabilities of existing models to a limited degree, and because Chinese companies appear to have the expertise required to build cutting-edge models entirely from scratch if needed. “Nobody here in the US knows how much distillation is benefiting the Chinese labs,” Miller says. “If you removed the ability for Chinese labs to distill, it's my view that it wouldn't dramatically change the competitive landscape.” To test whether open-weight models may have distilled from closed ones, the researchers fed 90 questions to each of the models. When they gave some open-weight models the first few words of reasoning traces captured from the proprietary group, they sometimes saw those open models generate remarkably similar answers. This was particularly pronounced with Kimi K3, the researchers say. There had previously been some speculation on Chinese social media that it might be possible to discover and use hidden reasoning traces for distillation. Yarin Gal, a computer scientist at Oxford University, says distillation is not only widely used but has also helped AI advance more rapidly. “If it's the norm that everyone blocks everyone [from doing distillation], then that also will have implications on the rate of progress,” he says. Companies and policymakers may introduce measures aimed at limiting distillation, but even so, AI models could continue to reveal their inner thinking in surprising ways.