salmanmohammadi / nanocode Public Notifications You must be signed in to change notification settings Fork 1 Star 26 Introducing nanocode: The best Claude Code that $200 can buy. #1 salmanmohammadi announced in Announcements Introducing nanocode: The best Claude Code that $200 can buy. #1 salmanmohammadi Apr 5, 2026 &middot; 0 comments Return to top Discussion options Uh oh! There was an error while loading. Please reload this page . {{title}} Something went wrong. Uh oh! There was an error while loading. Please reload this page . Quote reply salmanmohammadi Apr 5, 2026 Maintainer - I'm so excited to share nanocode . This is a library showing you how to train your own Claude Code end-to-end. To a first approximation, we will follow the simplest possible approach for training using Constitutional AI - the approach used by Anthropic to train their Claude models. We'll write our own SOUL.md , define the agentic interface which our model will use to interact with the world, generate synthetic data, and use preference optimisation to align the model with our SOUL . nanocode is written entirely in JAX and designed to be trained using TPUs. I adapted the core training infrastructure and philosophy from Karpathy's incredible nanochat project, so if you're familiar with nanochat , nanocode should feel very similar. This is how my d24 1.3B parameter nanocode turned out: nanocode.mp4 You can get started for free using the Google TRC program which gives you free access to pre-emptible TPUs for a month - and I think new Google Cloud accounts also get $300 in credits. I was fortunate to have access to the TRC program for 3 months for this project, and I found most of the time that my spot instances were rarely interrupted and I could easily have the same pod up for a week or more. You can reproduce nanocode-d24 (1.3B params) in around ~9 hours in total on a TPU v6e-8 costing $200, or train nanocode-d20 (477M params) in ~1.5 hours costing $34. If you're using NVIDIA GPUs, nanocode should also work out of the box, but you should be aware that nanocode has been highly optimised for TPUs. Training nanocode : a friendly agentic coding partner Andrej's original release post for nanochat does a great job of explaining what we're doing here, and the commands you'll use in nanocode are virtually identical, so I'd recommend reading through his work first. I'll go over what we've done differently to elicit agentic coding behaviours from our model. Tokenization and Pre-training The pre-training and tokenizer training process is pretty much identical to nanochat 's, but I found that including additional coding data from The Stack-V2 at a ratio of 1:5 in both the pre-training and tokenizer mixture resulted in a stronger coding model and more efficient code tokenization, which helped a ton. Let's first download the dataset shards we'll need for tokenizer training and model pre-training: # we'll be training our d24, 1.3B parameter model. but you can adapt MODEL_TAG for your model size. export NANOCODE_BASE_DIR= " $HOME /.cache/nanocode " export MODEL_TAG=d24 python -m data.pretrain -d fineweb-edu -n 300 # I've pre-packed and sharded The Stack similar to FinewWeb python -m data.pretrain -d the-stack-v2-dedup -n 60 And kick off our tokenizer training script: python -m scripts.tok_train --max-chars=2000000000 python -m scripts.tok_eval For reference, we can compare with nanochat 's tokenizer which is identical aside from the addition of The Stack in the training mixture (well, I've also added special tokens and templating logic to support more sophisticated tool calling, but more on that later). Comparison: nanocode vs nanochat =============================================================================================== Text Type Bytes nanocode nanochat Relative Better Tokens Ratio Tokens Ratio Diff % ----------------------------------------------------------------------------------------------- news 1819 407 4.47 375 4.85 +7.9% nanochat korean 893 558 1.60 712 1.25 -27.6% nanocode code 1259 326 3.86 492 2.56 -50.9% nanocode math 1834 922 1.99 966 1.90 -4.8% nanocode science 1112 259 4.29 228 4.88 +12.0% nanochat fwe-train 4208518 902950 4.66 856883 4.91 +5.1% nanochat fwe-val 4495276 975403 4.61 1010352 4.86 -3.6% nanocode We can see that this gives a big boost for code at the cost of general text tokenization efficiency, but this is okay since we want our model to do one thing very well; agentic coding. Our models are trained with a param:data ratio of 8 (following nanochat's scaling law analysis ). Let's kick off a training run like so: python -u -m scripts.base_train \ --batch-size=32 \ --minibatch-size=1 \ --config=configs.d24 \ --eval-every=500 \ --sample-every=500 You should see something like this: Vocab size: 32768 World size: 8 1342.17728M model parameters 67.108864M wte parameters 1207.959552M h parameters 67.108864M lm_head parameters Training on 10737418240 tokens over 10241 steps ==================== Estimated FLOPs per token: 10066329600 Scaling the LR for the AdamW parameters ∝1/√(2048/768) = 0.612372 Step: 0/10241 | Loss: 10.398 | dt: 104.58s | | tkps: 10026 | mfu: 1.37 | ETA: -1.0 min | lr_multiplier: 1.000 Peak bytes reserved/limit: 14.86/22.27 Step: 1/10241 | Loss: 9.771 | dt: 2.74s | | tkps: 382082 | mfu: 52.37 | ETA: -1.0 min | lr_multiplier: 1.000 Step: 2/10241 | Loss: 8.209 | dt: 2.74s | | tkps: 382220 | mfu: 52.39 | ETA: 234.1 min | lr_multiplier: 1.000 Step: 3/10241 | Loss: 7.327 | dt: 2.74s | | tkps: 382193 | mfu: 52.39 | ETA: 312.1 min | lr_multiplier: 1.000 ... fwe_bpb: 0.7626 | sv2_bpb: 0.4356 | avg_bpb: 0.5991 | dt: 90.53s < | bos | > The capital of France is Paris. It is the largest city in France and the most populous city in < | bos | > The chemical symbol of gold is Au. Gold is a soft, malleable, yellow metal that is < | bos | > The closest planet to the Sun is Mercury, which is the smallest planet in the solar system. It is the closest < | bos | > The opposite of hot is cold. The opposite of cold is heat. The opposite of heat is cold. < | bos | > The second-last day of the week is the day of the Lord. (Leviticus 23:2) ... CORE metric: 0.2352 | dt: 56.86s Total training time: 467.15min Our model has attained some knowledge about the world, which is nice. It still doesn't know about Saturday though : ). Let's look at some more thorough quantitative results, since we only estimate metrics using a smaller subset of the evaluation data during training: python -u -m scripts.base_eval --checkpoint=base --minibatch-size=8 This will print a whole bunch of metrics, but the relevant ones are bits-per-byte across our pretraining sets: sv2 (The Stack V2) and fwe (FineWeb_EDU), and the CORE metric which makes comparing against nanochat 's results and GPT-2 straightforward. I've compiled the results across a few model parameter sizes to get a feel for our scaling laws: | depth | params | CORE | cost | time | MFU | fwe bpb | sv2 bpb | | ------- | -------- | ------ | ------ | -------- | ------- | --------- | --------- | | d12 | 135M | 0.090 | $3 | 9 min | 17.4% | 0.956 | 0.689 | | d20 | 477M | 0.170 | $30 | 1.4 hrs | 45.2% | 0.838 | 0.533 | | d24 | 1.3B | 0.227 | $200 | 9.3 hrs | 52.5% | 0.759 | 0.445 | Since CORE measures general language reasoning capabilities and we've geared our models towards code data, it's expected that our CORE scores drop slightly compared to the corresponding GPT-2 models. Training d24 on FineWeb-EDU alone resulted in a CORE score of 0.261 which lines up with GPT-2 XL below and nanochat-d24 . The tradeoff here is that we expect our models to perform well in coding tasks. | model | params | CORE | | --------------- | -------- | ------- | | GPT-2 Small | 124M | 0.114 | | GPT-2 Medium | 355M | 0.185 | | GPT-2 Large | 774M | 0.215 | | GPT-2 XL | 1.6B | 0.257 | I'll mostly be referring to our d24 model throughout this post, which is similar to nanochat 's d24 model but is trained with twice the context length (4096 vs. 2048) to better sup