교과서 이론과 달리 ML 벤치마크를 반복 평가하며 개선해도 과적합(overfitting)이 잘 일어나지 않는 이유를 설명한 연구입니다. 핵심은 성공적인 에이전트의 전략이 압축 가능한 형태로 표현될 수 있어 데이터를 암기할 여지가 없다는 점이며, 압축 테스트가 과적합 여부를 판별하는 진단 도구로 활용될 수 있습니다. 이는 LLM이 방대한 세계 지식을 바탕으로 간결한 지시만으로도 완성된 ML 파이프라인을 재구성하는 강력한 '압축 디코더'라는 관점도 제공합니다.
번역된 본문
머신러닝 리서치 에이전트는 왜 과적합되지 않는가?
새로운 연구에 따르면, AI 에이전트는 데이터를 암기할 만큼의 공간이 부족한 '압축 가능한 모델'을 학습한다고 합니다.
마르틴 베르트란 로페즈(Martin Bertran Lopez), 애런 로스(Aaron Roth) — 2026년 9월 10일, 11분 분량
핵심 요점:
ML 모델은 여러 차례 반복적 개선을 거쳐도 벤치마크에 과적합되지 않습니다. 이는 동일한 홀드아웃 데이터에 대해 반복적으로 평가하면 과적합이 발생해야 한다는 교과서적 예측과 모순됩니다.
ML 리서치 에이전트 실험에 따르면 성공적인 전략은 매우 압축 가능합니다. 성공한 에이전트의 전략을 정보 병목(최소 16개 토큰)으로 압축하면, 메모리가 전혀 없는 새 에이전트가 원래 에이전트의 성능을 재현할 수 있습니다. 즉, 해당 전략이 암기한 데이터가 아니라 실제 구조를 포착한 것입니다.
압축은 설명이자 진단 도구를 모두 제공합니다. 실제로 과적합한 전략은 압축 테스트에서 실패합니다. 즉, 검증 세트 특유의 성능 향상이 병목을 통과하면 사라집니다.
LLM은 강력한 압축 디코더입니다. 방대한 세계 지식을 갖고 있어 간결한 전문가식 약어 프롬프트만으로도 완전한 ML 파이프라인을 재구성할 수 있으며, 이것이 LLM이 왜 그렇게 유능한지 구체적으로 이해하는 방법입니다.
머신러닝의 본질은 암기가 아니라 일반화입니다. 학습 알고리즘에 훈련 예제를 주입해 모델을 학습시킵니다. 하지만 목표는 훈련 예제에서 잘 수행하는 것이 아닙니다. 그건 쉽습니다. 정답을 그냥 외워버리면 되니까요. 목표는 한 번도 본 적 없는 새로운 예제에서 잘 수행하는 것입니다. 모델이 훈련 데이터에서는 잘하지만 새로운 데이터에서는 성능이 나쁘다면, 실제로 아무것도 학습하지 못한 것이며, 학습했다고 스스로를 속인 것일 뿐입니다.
이러한 실패 양상에는 이름이 있습니다. 바로 과적합(overfitting)입니다. 입문 통계학이나 머신러닝 수업을 들어본 사람이라면 표준적인 방어법을 알 것입니다. 데이터 일부를 떼어놓고 훈련에 사용하지 않는 것입니다. 실무에서 이 홀드아웃 데이터는 두 가지 역할을 합니다. 검증 세트(validation set)는 모델을 만드는 동안 반복적으로 참조하는 세트로, 후보를 비교하고 하이퍼파라미터를 튜닝하며 다음에 무엇을 시도할지 결정하는 데 씁니다. 최종 테스트 세트(또는 홀드아웃)는 맨 마지막에 단 한 번만 건드리는 것이 원칙입니다. 훈련 절차가 이를 본 적이 없기 때문에, 여기서 좋은 성능이 나오면 실제 환경에서 마주칠 새로운 예제에 대한 정확한 대리 지표가 됩니다.
머신러닝의 본질은 암기가 아니라 일반화입니다.
하지만 '홀드아웃' 조건이 결정적입니다. 정확한 대리 지표라는 보장은 홀드아웃 세트가 진정으로 보이지 않은 상태로 유지될 때만 성립합니다. 그 성능을 확인하고, 그에 맞춰 훈련 절차를 조정하고, 다시 확인하고, 더 좋은 숫자를 좇으며 반복한다면, 그 세트는 더 이상 보이지 않는 상태가 아닙니다. 훈련 절차의 일부가 되어버린 것입니다. 이를 충분히 반복하면 훈련 세트에 과적합하듯 홀드아웃 세트에도 과적합할 수 있으며, 미지 데이터에 대한 대리 지표를 잃게 됩니다. 이는 설계상 반복 사용되는 검증 세트를 포함해 이런 식으로 재사용되는 모든 홀드아웃 세트에 해당합니다.
머신러닝의 심장부에 있는 수수께끼
실제 머신러닝 연구는 방금 설명한 반복적 개선 루프와 정확히 똑같이 생겼습니다. 모두가 수년간 개정되지 않는 소수의 벤치마크 데이터셋으로 성능을 평가합니다. 연구 커뮤니티는 거대하고 분산된 루프를 반복합니다. 벤치마크에서 모델을 평가하고, 훈련 절차를 수정하고, 재평가하고, 논문을 내고, 다음 연구팀이 조금 더 개선을 짜냅니다. 이것이 바로 교과서적 설명대로라면 만연한 과적합을 낳아야 할, 홀드아웃 세트에 대한 언덕 오르기(hill-climbing)입니다. 이쯤 되면 리더보드는 벤치마크에서는 훌륭하지만 그 외 모든 곳에서는 평범한 모델들로 포화되어 있어야 합니다.
그러나 실제로는 그렇게 되지 않습니다. 오래되고 많이 재사용된 벤치마크에 대해 완전히 새로운 테스트 세트를 구축한 연구들은...
Machine learning Why don’t machine learning research agents overfit? New research indicates that AI agents learn compressible models of data, which don’t have enough space to enable memorization. By Martin Bertran Lopez , Aaron Roth September 10, 2026 11 min read Share Share Copy link Email X LinkedIn Facebook Line Reddit QZone Sina Weibo WeChat WhatsApp 分享到微信 x Key takeaways ML models don't overfit benchmarks, even after many rounds of iterative improvement. This contradicts textbook predictions that repeatedly evaluating against the same held-out data should lead to overfitting. Experiments with ML research agents indicate that successful strategies are highly compressible. When a successful agent's strategy is squeezed through an information bottleneck (as few as 16 tokens), a fresh agent with no memory can reproduce the original agent's performance, meaning the strategy captured real structure, not memorized data. Compression provides both an explanation and a diagnostic tool. Strategies that genuinely overfit fail the compression test: their validation-specific gains vanish when passed through the bottleneck. LLMs are powerful compression decoders. Because they carry vast world knowledge, they can reconstruct full ML pipelines from terse, expert-shorthand prompts, which is a concrete way of understanding why they're so capable. Was this answer helpful? Machine learning, at its core, is about generalization, not memorization. You hand your learning algorithm a pile of training examples and use them to fit a model. But the goal is not to perform well on the training examples — that's easy, you could just memorize the answers. The goal is to perform well on new examples that you have never before seen. If a model does well on the data it was trained on but poorly on fresh data, it hasn’t actually learned anything; you have only fooled yourself into thinking it has. This failure mode has a name: overfitting. Anyone who has taken an introductory statistics or machine learning class knows the standard defense. You hold out some of your data and refuse to train on it. In practice, this held-out data plays two roles. A validation set is one you consult repeatedly while building the model — to compare candidates, tune hyperparameters, and decide what to try next. A final test set (or holdout ) is meant to be touched only once, at the very end: because the training procedure never saw it, strong performance there is a correct proxy for the new examples you will encounter in the wild. Machine learning, at its core, is about generalization, not memorization The “holdout” condition is crucial, though. The correct-proxy guarantee holds if the held-out set stays genuinely unseen. If you check your performance on it, tweak your training procedure in response, recheck, and iterate, chasing better and better numbers, that set is no longer unseen; it has become part of your training procedure. Do this enough times, and you can overfit it just as you might have overfit the training set, and you have lost your proxy for unseen data. This is true of any held-out set you reuse this way, including a validation set, which is reused by design. A puzzle at the heart of machine learning Real machine learning research looks exactly like the iterative improvement loop we just described. Everyone gauges performance using a handful of benchmark datasets that go unrevised for years. The research community repeats an enormous, distributed loop: evaluate a model on the benchmark, revise the training procedure, re-evaluate, publish, and let the next group eke out a little more improvement. This is precisely the kind of hill-climbing against a held-out set that, by the textbook account, ought to produce rampant overfitting. By now, the leaderboards should be saturated with models that look great on the benchmark and mediocre everywhere else. And yet that is not what happens. Studies that build entirely fresh test sets for old, heavily reused benchmarks have found that improvements largely transfer : on the new data, models demonstrate the same gains they did on the old benchmark. Benchmark-driven machine learning, against the textbook's prediction, has produced rapid and largely real progress. Why? There is no shortage of hypotheses, but they have been hard to test empirically, because the "subject" of the experiment is the entire human research community. You cannot reset a field, wipe its memory, and rerun the last decade under controlled conditions. But we can do something similar. We now have capable, LLM-based research agents that can autonomously run the same machine-learning optimization loops that human communities run. They engage in the same benchmark hill-climbing — and, intriguingly, they too seem not to overfit. The difference is that an agent, unlike a research community, is something you can reset. You can clear its memory, control exactly what information it sees, and run the experiment again. In a recent paper, " What fits (into few tokens) doesn't overfit: Compression and generalization in ML research agents ", we do exactly that — and in the process offer a concrete explanation for the long-standing mystery. Occam's razor, made precise The explanation begins with a very old idea. Occam's razor says that among hypotheses that explain the data equally well, the simpler one is more likely to be correct. It turns out this intuition has a precise mathematical form, and it is what underlies the whole story. Suppose you can describe your hypothesis — your model, your strategy — in a small number of bits, far fewer than it would take to memorize the training data. If that compact hypothesis performs very well on the training data, it must also perform well on new data. Occam's razor, formalized: among hypotheses that explain the data equally well, the simpler one — describable in fewer bits — is more likely to generalize to new examples. The reasoning runs through a counting argument. There simply are not very many short descriptions, because there are not very many short strings. The fewer candidate hypotheses there are, the less likely it is that any one of them fooled you on the training set by luck — even though you used the training set to guide your search. Another way to get the intuition: if your compressed description is too small to secretly record the training data, then when it performs well on the training data, it cannot be because it memorized the answers — it didn't have space to do that. It must be because it captured something true about the data's structure. Short descriptions cannot cheat because there isn't room. Here is an attractive hypothesis: successful machine learning strategies are highly compressible. A researcher might stare at thousands of benchmark scores over the course of a project, but the strategy that ultimately survives is usually a short list of familiar choices — an architecture family, an optimizer, a learning-rate schedule, a data-handling recipe, a regularization scheme. If that final recipe can be communicated in just a few bits, then the model's true dependence on the benchmark is far smaller than the long, winding transcript of experiments would suggest. The hill-climbing was extensive, but the thing that came out the other end was — or could have been — tiny. Compression, intelligence, and the power of a knowledgeable listener Imagine trying to explain a specific machine learning pipeline to a bright high-school student, in enough detail that they could actually reproduce it. It would be a long, laborious conversation. You would have to explain what gradient descent is, what a neural network is, what PyTorch or JAX or TensorFlow does, what a learning rate is, and on and on. Almost none of that is specific to your problem; it is general background about how machine learning works. Now imagine explaining the same pipeline to an expert ML engineer. The conversation now collapses to a few sentences. You skip everything th