998달러로 3.8B LLM 학습시켜 CORE 0.384 달성

Somewhere between “nanoGPT toy” and “you need a research lab” there’s a large, under-described region where one person with a few thousand dollars can train a meaningful model. I wanted to see language and understanding emerge from random weights for myself, and to learn the parts you can only learn by starting from scratch. This project was written in the evenings, debugged on a 5090 and finished on rented B200s. It was heavily inspired by Andrej Karpathy’s nanochat . The result is a 3.8B-parameter model scoring 0.384 on CORE , trained on 65B tokens in 43 hours for $998 . What follows is what worked, what didn’t, and what I still don’t know. Model Params Tokens Hardware Time Cost CORE GPT-2 (OpenAI) 1.5B — — — — 0.2565 nanochat d26 ~561M 11.2B 8× H100 ~3h — ~0.258 nanochat d32 ~1B — 8× H100 ~33h ~$1000 0.310 little-lm 3.8B (1024 ctx) 3.848B 57.3B 8× B200 35.9h $820 0.338 little-lm 3.8B (2048 ctx) 3.848B 65.3B 8× B200 43h $998 0.384 My model is larger than nanochat d32 and took similar wall-clock time. B200s were better value per unit of work than H100s. But for roughly the same money as nanochat’s $1,000 configuration, this lands meaningfully ahead of it. An encouraging data point about what’s reachable outside a lab or a mega company with millions in compute budget. As the frontier moves, $1,000 takes you further and further. Setup I’ve built little-lm as a config-driven framework for training small decoder-only LLMs. Every run is fully specified by a YAML file: model, dataset, optimizer, schedule, callbacks. Components self-register into a global registry and get resolved by name, so swapping an optimizer or a dataset is a one-line config change. Good infrastructure pays for itself almost immediately. Ordinary software engineering discipline (Things like separation of concerns, clean interfaces, components you can swap in) matters a lot in AI work. It cost me a little at the start, and a couple more times afterward to fix bad contracts or suboptimalities. But this time investment pays for itself at the first convergence problem you encounter. I found that a great infra is the infra that almost never requires you to edit code manually. If you can read the config and understand exactly what happens, and there are no hidden mechanics, it means you have done a good job. The following report is the result of being able to express experiments as a three-line YAML diff rather than a branch. The final model is Llama-style: RMSNorm, RoPE, GQA (24 query heads, 8 KV heads), relu² MLPs, QK-norm, logit softcap, per-layer learnable residual scalars, and ResFormer-style value embeddings. Component Params Token embeddings 154.5M LM head (untied) 154.5M 28 decoder layers 2,818.7M Value embeddings (14 tables) 721.2M Total 3.848B Worth noting that the value embeddings are 19% of the parameter count. 14 tables of vocab × kv_dim , one on every other layer. Results Early experiments Before good runs there were many bad ones. I trained an 858M Llama on FineWeb-Edu for 16.4B tokens, 5.8 days on a single A100. AdamW at 2.5e-4, cosine decay to zero, 5% warmup, batch 256 via gradient accumulation, 2048 context. The result: PIQA 60.45% . GPT-2 124M scores about 63%. I had spent six days of compute to build something worse than a model seven times smaller, from 2019. Generations were repetitive and borderline nonsensical. The loss curve told the story. Cosine decay to zero. The curve went completely flat after about 70% of the steps. The final 30% of the compute budget produced essentially nothing as the learning rate might be too low. Linear cooldown holds a useful rate much later. Peak LR too conservative. 2.5e-4 is low for 858M parameters. You can be quite aggressive for those small models. AdamW on everything. Muon should be meaningfully better per-token for the matrix parameters at this scale. In fact this was demonstrated pretty quickly in ablation runs. The data. FineWeb-Edu is decent. It is not the best available. Five changes came out of that post-mortem. Together they are the difference between the run above and a model that beats GPT-2 by a wide margin. Trapezoidal LR schedule. Warmup for 5%. Hold flat and finish with linear cooldown over the last 50% to 5% of peak. The point is that the model keeps learning until the end instead of coasting through the tail. In the 3.8B run the eval loss was still descending at the final step, which is exactly the behavior the 858M run failed to produce. Muon for matrix parameters , AdamW for everything else. Muon is slower per step (Newton-Schulz orthogonalization isn’t free, about 25% in a shallow-accumulation benchmark) but that cost is paid once per optimizer step: at 7 gradient-accumulation steps it dilutes to ~4%. Measured against total run time the convergence is much faster overall. ClimbMix instead of FineWeb-Edu. This was a tremendous jump in convergence speed. Exactly as Karpathy found as well. FP8 + vocab padding. FP8 training via torch._scaled_mm with dynamic tensorwise scaling on all three GEMMs, and padding the vocab from 50,257 to 50,304 (a multiple of 64) so the tensor cores are happy. Together, +33% throughput mostly from fp8. 1024 context instead of 2048. Halving the context roughly doubles the batch size at fixed memory. Throughput barely changes per token. We are still dominated by the MLPs which is a good sign we are using the hardware effectively. Below we will discuss the impact of the context length on the model. Here is the whole run: Step Tokens Eval loss CORE 2,500 5.7B 2.3278 0.2389 5,000 11.5B 2.2072 0.2752 7,500 17.2B 2.1571 0.2934 10,000 22.9B 2.1269 0.3104 12,500 28.7B 2.1075 0.3147 15,000 34.4B 2.0710 0.3224 17,500 40.1B 2.0395 0.3294 20,000 45.9B 2.0160 0.3267 22,500 51.6B 1.9963 0.3345 25,000 57.3B 1.9868 0.3384 ~480,000 tokens/sec in steady state, which puts 57.3B tokens at 33 hours. The wall clock was 35.9h. The difference is the CORE evaluations, which took about 15 minutes each (ten of them over the run) and consumed 7% of the total. Re-running this identical recipe at 2048-token context scored 0.3840 . Almost all of that gap turned out to be some tasks that were very context dependent. On the GPUs themselves: 92% SM activity, 40% SM occupancy. High activity means the SMs almost never went idle. No dataloader starvation or network waits, which is the payoff for downloading the shards locally instead of streaming, which would leave us vulnerable to a small hugging face network hang. The low occupancy is what back-to-back large GEMMs look like: matmul kernels trade occupancy for register-tile size on purpose. Compute-bound and well fed, great signal we are using the hardware well and we can extend every dollar we spend into a better model. That’s about 1,047 TFLOP/s sustained per B200 , or ~25% MFU against Blackwell’s dense FP8 peak. (Against the bf16 peak it reads as 50%, which is the number that matters a bit more because not even all the linear layers run in FP8.) The distributed strategy is plain old DistributedDataParallel. At 3.8B on a single node, gradient communication was never the constraint, and the sharded-optimizer machinery turned out to be unnecessary. Increasing throughput Renting GPUs isn’t cheap, at work you often think about the quality of the model before its cost. When it’s your own money burning, throughput matters a lot more all of a sudden. This took real work on a single RTX 5090, before I ever rented a node. Baseline 858M model, bf16, compiled: 26,144 tok/s . Final: 37,621 tok/s . FP8 (+25%). All three GEMMs (1 forward and 2 backwards) in FP8 with dynamic tensorwise scaling. Requires SM90+ but that is quite a nice throughput jump. Vocab padding (+33% cumulative). Padding 50,257 → 50,304 costs 47 unused embedding rows and unlocks the fast tensor-core path. Nearly free. Fused linear cross-entropy (+44% cumulative). Liger’s FusedLinearCrossEntropyLoss fuses the lm_head matmul into the loss and chunks internally, so the full (B*T, vocab) logits tensor is never materiali