AI 추론, 정답을 맞히는 잘못된 이유

Home Is AI Reasoning Right for the Wrong Reasons? Comment Save Article Read Later Share Facebook Copied! Copy link Email Pocket Reddit Ycombinator Comment Comments Save Article Read Later Read Later Qualia Is AI Reasoning Right for the Wrong Reasons? By John Pavlus July 31, 2026 The idea that artificial intelligence can “reason” is more intuitive than ever. But intuitions can be wrong, and the science is far from settled. Comment Save Article Read Later Introduction I ’ll just say it: What the hell is going on with AI “reasoning”? Sorry for the air quotes. That punctuational side-eye was more common in 2024, when the specially trained cousins of LLMs now known as “large reasoning models,” or LRMs, were still new. Nowadays it may seem downright churlish, though, given that a “general-purpose reasoning model” from OpenAI solved a famous open mathematical research problem in one shot in May 2026. Still, I’m not sure how else to acknowledge my intellectual whiplash over the scientific interpretation of what these AI systems are actually doing. Reasoning comes in many technically defined forms , but the basic procedure is easily recognizable: arriving at a sound conclusion by linking together intermediate steps that logically follow from each other. We do this with thoughts; LRMs use so-called chains of thought, a term of art for the streams of synthetic text that the models emit before arriving at an answer to a complex query. One minute, the idea that AI could reason via these chains was being prominently and credibly critiqued (by a team of researchers from Apple) as an “ Illusion of Thinking ” subject to “complete accuracy collapse” under surprisingly simple conditions. The next minute, LRMs were bagging gold medals at the International Mathematical Olympiad, a feat so challenging that “even very successful mathematicians and scientists may well highlight [it] on their CVs all their lives,” as the scientist and AI critic Gary Marcus and Ernest Davis wrote in 2025. If that’s not a sign of “real” reasoning, what is? But wait — soon after, more research, from the Santa Fe Institute, showed that LRMs can crush even carefully designed benchmarks for reasoning (like a collection of analogy-like visual puzzles) using mere “ surface-level ‘shortcuts.’ ” What they were doing looked less like generalizable reasoning than just gaming the system. Then, as if on cue, another “hold my beer” moment: Google DeepMind and the mathematician Terence Tao ( the GOAT! ) used AI to rediscover or improve the solutions to 67 problems “spanning mathematical analysis, combinatorics, geometry, and number theory.” Deal with it, haters! What about additional evidence that LRMs can’t reason reliably , even when they possess the necessary algorithm and computational budget to do so, and suffer from a list of scientifically documented failure states long enough to use as a Slip ’N Slide? Whatever — I guess that’s just “jagged intelligence” for you (AI-speak for “when it works, it works”). And so it went from late 2025 into 2026. I’ve been a science journalist for 20 years and an AI journalist for half of that, so I know better than to expect tidy consistency out of rapidly advancing research. But even for me, this back-and-forth has been a bit much. To quote Al Pacino in The Insider , “I’m getting two things: pissed off, and curious.” I don’t believe there’s fraud to be found here. I just want to know which way is up. Can AI reasoning somehow be both BS and not at the same time? And if so, how on Earth does that work? I knew just who to call first. Melanie Mitchell’s career in AI stretches back to the 1980s, but lately she’s earned a reputation as an au courant AI truth teller, penning lucid explainers for Science and her widely read newsletter , as well as conducting research at the Santa Fe Institute. (The study about “surface-level ‘shortcuts’” is hers.) When I asked her what we actually know about AI reasoning, her answer was brief enough to fit on an index card. “Number one: It works. It improves things,” she said, referring to LRMs’ superior accuracy on reasoning tasks compared to LLMs. “Number two: The actual text that’s generated” — i.e., the chain of thought that every LRM is trained to produce to improve its performance — “isn’t necessarily faithful to what’s going on [inside the model]. And number three: A lot of that text isn’t even useful. You can actually take it out.” Let’s unpack numbers two and three, because that’s where the superposition of “BS and not” actually lives. Chains of thought were half-discovered, half-devised in 2022 as a prompting hack for LLMs: Provide them with examples of written-out reasoning (or, famously, just ask them to “ think step by step ”), and they’ll suddenly give less boneheaded answers to simple logic and math problems. LRMs, starting with OpenAI’s o1 model in 2024, are trained to automate this trick by generating such prompts — also called reasoning traces or thinking tokens — and then feeding them back to themselves. Because LRMs are essentially just language models, those extra bits of text create what looks convincingly like a paper trail of the model’s “thought process.” Except it’s not that simple. A growing body of academic and industry research has cast doubt on whether these “intermediate tokens” are a faithful representation of an LRM’s inner workings. Instead of being auditable receipts or accurate reports, they can appear more like what the Arizona State University researcher Subbarao Kambhampati calls “mumblings” — bits of language, yes, but ones whose meaning may be entirely incidental to any reasoning that might have occurred. Kambhampati’s lab showed in 2025 that fully replacing a model’s correct “traces” with incorrect or irrelevant ones didn’t degrade its performance on a formal reasoning task. Meanwhile, training the model only on correct trace data still led it to occasionally generate invalid records of its reasoning — even when it produced a correct solution to the original problem it was given. A 2024 paper from researchers at New York University showed that “meaningless filler tokens” — literally, strings of dots — could function effectively in place of a human-readable “chain of thought.” William Merrill , one of the authors on that paper and currently a professor at the Toyota Technological Institute at Chicago, put the matter plainly: “There’s no guarantee the chain of thought has to be meaningful in any sense.” Pavel Izmailov , a researcher at NYU who also works for Anthropic (and was part of its original reasoning-model team), said he doubts that reinforcement learning — a typical training method for LRMs — even incentivizes models to produce faithful chains of thought in the first place. “I mean, maybe it will,” he told me. “But I would say the chances are not very high.” OK, so the linguistic content of reasoning traces may be dubious. But surely the tokens themselves must play a role in producing the model’s outputs? (Think of a pinball machine: It runs on coins, not the words “In God We Trust.”) Not so fast. A 2025 paper from Northeastern University and the University of California, Berkeley on frontier open-source LRMs showed that between 30% and 60% of their “thinking steps” had “minimal causal impact” on the answers the models produced to benchmark math questions. Chop half of them out, and a model’s performance barely suffers. “We want to be careful when we review these chain-of-thought prompts because they may not be linked to the final output,” said Weiyan Shi , one of the study’s authors. So reasoning traces, the very things that supposedly distinguish LRMs from the mere next-word-predicting LLMs, are not necessarily either meaningful or causal to a model’s … reasoning? I’m no philosopher, but this seems to stretch the meaning of “reasoning” beyond its tensile strength. Kambhampati’s research group sounded frankly fed up in the title of their position paper on the subject (presented at the 2026 I