연구 중심 에이전트: 코딩 전 논문을 읽을 때

Table of Contents Where code-only context works Where code-only context breaks down Adding a research phase The experiment log What the research turned up The pivot: from compute to memory Optimizations that landed 1. Softmax fusion 2. RMS norm fusion 3. Adaptive from_float parallelization 4. Graph-level RMS_NORM + MUL fusion 5. Flash attention KQ fusion Results What didn’t work Experiments that failed The benchmark bug Cloud VMs are noisy The code review What this means for coding agents Try it on your own project TL;DR: Coding agents generate better optimizations when they read papers and study competing projects before touching code. We added a literature search phase to the autoresearch / pi-autoresearch loop, pointed it at llama.cpp with 4 cloud VMs, and in ~3 hours it produced 5 optimizations that made flash attention text generation +15% faster on x86 and +5% faster on ARM ( TinyLlama 1.1B ). The full setup works with any project that has a benchmark and test suite. Key takeaways: Agents that read papers and study competing projects before writing code find optimizations that code-only agents miss. The literature research pointed the agent at operator fusions present in CUDA/Metal backends but absent from CPU. 5 of 30+ experiments landed: 4 kernel fusions and an adaptive parallelization. The biggest win fused three passes over flash attention’s QK tile into a single AVX2 FMA loop. Studying forks and other backends was more productive than searching arxiv. ik_llama.cpp and the CUDA backend directly informed two of the five final optimizations. Total cost: ~$29 ($20 in CPU VMs, $9 in API calls) over ~3 hours with 4 VMs. Where code-only context works # Karpathy’s autoresearch showed that a coding agent can autonomously improve a neural network training script. In our previous post , we scaled that to 16 GPUs and watched the agent run ~910 experiments in 8 hours, driving val_bpb down 2.87%. The agent brainstormed ideas from code context alone, and the experiments were all variations on the same train.py . Since then, pi-autoresearch generalized the loop into a reusable extension for any benchmarkable target. Shopify CEO Tobi Lütke ran it on Liquid , the Ruby template engine that processes $292B in annual merchandise volume. The agent ran ~120 experiments, producing 93 commits that cut parse+render time by 53% and allocations by 61% with zero regressions across 974 unit tests ( Simon Willison’s writeup , Tobi’s post ). In that case, the optimization surface was visible in the source. The Liquid agent could read the tokenizer, see that StringScanner was the bottleneck, and brainstorm alternatives from the codebase alone. Where code-only context breaks down # Not every optimization problem works this way. A codebase tells you what the code does , but not why it’s slow or what alternatives exist outside of this codebase . When the answer lives outside the source (in arxiv papers, in competing projects, e.g. in the domain knowledge a senior engineer would bring), an agent working from code alone will generate shallow hypotheses. We saw this when we pointed the agent at llama.cpp ’s CPU inference path. The optimization search space isn’t “try a different learning rate.” It’s “should I fuse these two memory passes?”, “is this workload compute-bound or memory-bound?”, “what has ik_llama.cpp already tried?” The agent’s first wave of experiments showed the problem. Working from code context alone, it went straight for SIMD micro-optimizations in the quantized dot products that sit in GGML’s matrix multiplication hot path. It tried: AVX2 prefetching in the Q4_0 dot product inner loop (+0.8%) 2x loop unrolling with dual accumulators (+0.9%) Eliminating a temporary buffer in mul_mat (-2.8%, regression) Hoisting block boundary calculations (+0.6%) All within noise. The agent’s postmortem: “Wave 1 results show that micro-optimizations in the compute path give negligible returns because text generation is memory-bandwidth bound, not compute bound.” A 606 MiB model at ~49 tokens/s consumes ~30 GB/s of memory bandwidth, close to the c6i.2xlarge’s DRAM limit. No amount of SIMD tricks will help when the CPU is stalled waiting for model weights to arrive from DRAM. But the code alone doesn’t tell you this. You need to know the memory bandwidth of the target hardware, understand the roofline model, and recognize that batch-size-1 inference is memory-bound. That’s domain knowledge the agent didn’t have. Adding a research phase # If the bottleneck is hypothesis quality, give the agent better inputs. Before running any experiments, have it read papers, study forks, and look at what other projects have already tried. The same preparation a senior engineer would do before touching unfamiliar code. The original autoresearch loop is: edit code -> run experiment -> check metric -> keep or discard. pi-autoresearch generalized this to any project with a benchmarkable metric. Our version builds on that and adds a research step and parallel cloud execution: The agent writes its own benchmark script ( autoresearch.sh ) and correctness checks ( autoresearch.checks.sh ), then uses SkyPilot to fan out experiments across cloud VMs. Each experiment runs on its own VM: build the project, run the benchmark, run correctness checks, report metrics. The agent checks results via sky logs , commits winners, and queues the next wave. experiment.yaml : SkyPilot task template for one experiment resources : cpus : 4 + memory : 8 + workdir : . envs : EXPERIMENT_ID : baseline EXPERIMENT_DESC : "baseline measurement" BUILD_CMD : "make -j$(nproc)" BENCH_TIMEOUT : 300 CHECK_TIMEOUT : 300 setup : | cd ~/sky_workdir if [ -f setup_deps.sh ]; then bash setup_deps.sh else eval "${BUILD_CMD}" fi run : | cd ~/sky_workdir # Build, benchmark, run checks, report METRIC lines eval "${BUILD_CMD}" 2>&1 | tail -30 BENCH_OUTPUT=$(timeout "${BENCH_TIMEOUT}" bash autoresearch.sh 2>&1) echo "$BENCH_OUTPUT" # ... extract METRIC lines, run autoresearch.checks.sh ... echo "EXPERIMENT_STATUS: done" No GPU needed for CPU-bound code optimization. Override with --gpus if your target needs GPU benchmarking. The experiment log # We pointed Claude Code at llama.cpp, gave it 4 AWS VMs via SkyPilot, and told it to make CPU inference faster. Target: CPU inference throughput for TinyLlama 1.1B (Q4_0 quantization), benchmarked on two architectures: x86: AWS c6i.2xlarge (Intel Xeon Ice Lake, 8 vCPUs, AVX-512) ARM: AWS c7g.2xlarge (Graviton3, 8 vCPUs, NEON) Metric: tokens/second for prompt processing (pp) and text generation (tg), measured with llama-bench -p 512 -n 128 -t 8 -r 5 . It started with 4 x86 VMs to establish baselines and run experiments. Later it provisioned ARM VMs to check portability; each kernel fusion includes both AVX2/FMA and NEON paths, with scalar fallbacks. What the research turned up # Between experiment waves, the agent ran two parallel research threads: Competing projects: ik_llama.cpp (a performance-focused fork), llamafile’s tinyBLAS, PowerInfer, ExLlamaV2 (the author of this post wasn’t even aware of some of these projects) Arxiv papers: FlashAttention (IO-aware tiled attention), Blockbuster (block-level operator fusion), LLM Inference Acceleration via Efficient Operation Fusion , Online normalizer calculation for softmax , Inference Performance Optimization for Large Language Models on CPUs (Intel’s cache-aware thread partitioning) The top findings: ik_llama.cpp’s row-interleaved quantization repacking gave 2.9x PP improvement. It was already upstreamed to mainline llama.cpp via the Q4_0_8x8 repack format. The agent confirmed it was active in the benchmark. The Blockbuster paper proposes fusing the entire FFN block (RMSNorm + gate matmul + up matmul + SwiGLU + down matmul)