AI를 강화학습시켜 AI를 강화학습시키는 에이전트 개발 (약 1,300달러)

🤓 AI Trains AI: Using RL to train an AI agent that trains AI using RL 🔓 Everything is open sourced including: the trained agent's weights ( LoRA adapter on 🤗 HF ), agent harness, task families, reward code, GPU orchestration, tinker RL training scripts, and retro write-ups of every pilot (including the failures). Jump to Getting started ↓ TL;DR: I built a pipeline where an AI agent: Is handed a training task ("teach a model to do X") Writes a complete prime-rl training job, including: environment, reward, dataset, hyperparameters. Submits it to real Runpod GPUs for training. Leveraging Tinker , I then RL-trained the agent itself, rewarding it when it trained better models. Reward climbed from ~0.0 to a ~0.63 peak over 54 training steps. Transferring to a held-out task family it never trained on . An AI in an RL loop, whose action is training AI in an RL loop. (Source: assets/hero.svg .) 📚 Table of Contents 🔁 How it works The episode ⚖️ Reward design 🧩 Task families 📈 Results The reward climbed in two distinct rungs The skill transfers to a held-out task family The agent learned to pick the better base model 🖥️ Infrastructure 💰 Costs 🤗 Model weights 🚀 Getting started 🔮 Future improvements Acknowledgements 🔁 How it works Two RL loops with two entirely separate training stacks. who is being trained what training looks like stack Outer loop the trainer agent ( Qwen3.6-35B-A3B , LoRA) episodes of writing training jobs; episode reward is the policy-gradient signal Tinker + tinker-cookbook (importance-sampling GRPO) Inner loop a small base model ( Qwen3-0.6B / 1.7B ) the job the agent wrote: a verifiers environment + rubric, with a [prime_rl] config table prime-rl (GRPO) on Runpod GPUs Tinker trains the agent. The agent writes verifiers envs, rubrics, and prime-rl configs. prime-rl trains the small model. The inner model's hidden-eval score flows back up as the outer loop's reward. The episode One episode, end to end. (Source: assets/episode.svg .) One episode = one attempt by the trainer agent to produce a valid, high-quality training job for a given task: Task spec — a description of what to train ("teach a small model to resolve multi-hop persona queries"), hard constraint bounds, the eval tool interface, and a handful of dev examples. The agent works — it edits a sandboxed workspace through read_file / write_file / edit_file / list_files , and can call get_baseline_scores to see the untrained base models' scores on the hidden eval. submit_job — triggers a validation probe. Any failures are returned and the agent gets capped retries. Dispatch — a validated job is queued and picked up by a warm pool of Runpod GPU pods, which run GRPO training with prime-rl and score the checkpoint pre/post on the hidden eval. Reward — combines validation efficiency with the trained model's uplift over the best untrained baseline. The outer loop then RL-trains the agent itself on episode reward, using Tinker . Every outer-loop batch spawns 40 real inner training jobs across up to 16 GPU pods. ⚖️ Reward design Episode reward is a weighted sum (live weights 0.35 / 0.60 / 0.05): Validation — 1.0 for a first-try valid submission, decaying per extra attempt; 0 if the episode never validates. (Separately, the outer loop scores an episode −0.1 when the agent never produces a parseable submission at all. The −1.0 values you'll see in the CSVs are a "no post score" logging sentinel, not a reward.) Job quality — a hybrid of the trained model's absolute post-training score and its signed uplift over best_pre , the best untrained model's frozen baseline: 0.25·post + 0.75·uplift_term . A job that dies on the GPU scores 0 here (the episode keeps its validation term). Train speed — a small tie-breaker for faster jobs, gated on job success. Note for close readers: the agent-facing prompt ( template/INSTRUCTIONS.md ) gives the agent a simplified view — the 75/25 uplift/absolute split inside job quality, plus a fewer-attempts nudge — not the full 0.35/0.60/0.05 decomposition. The published adapter was trained against that prompt; the reward actually computed is the one above. 🧩 Task families Six families of tasks, deliberately built so that the untrained models struggled without training, and all require multi-step tool use and reasoning: family shape untrained best_pre (n=200) calc_chain chained arithmetic where each step feeds the next 0.742 multi_hop persona-world queries needing multiple dependent lookups 0.654 string_pipeline composed string transformations 0.545 ledger stateful account bookkeeping via tools 0.242 dispatch conditional routing decisions 0.323 triage (held out) on-call incident triage: correlating services, incidents and deploys across tools 0.352 Five families train the agent; triage is never trained on and serves as the generalisation probe. 📈 Results Setup: Qwen3.6-35B-A3B trainer agent, LoRA rank 8, lr 4e-5, GRPO with group size 8, up to 16 concurrent GPU pods, ~40 real training jobs per batch. Runs: pilot-7 (10 steps) → 7b (24 steps, warm-started) → 7c (20 steps, warm-started) — 54 steps total. 1. The reward climbed in two distinct rungs Decomposing the reward shows what was learned, and in what order: Rung 1 — process reliability (pilot-7). The entire early gain came from converting validation failures and dead-on-GPU jobs into completed episodes. Job quality stayed flat while total reward rose to ~0.26. Showing GRPO taking the steepest gradient first. Rung 2 — making better models (pilot-7b onward). With reliability saturating (~0.75–0.80 validation), job grade rose 0.30 → 0.41 and the hidden-eval post-training score went from ~0.04 noise to a sustained 0.22–0.48. The agent started making better models, not just working ones. 2. The skill transfers to a held-out task family A task family which the agent never trained on showed performance rises with outer-loop training, then plateaus: checkpoint outer steps mean reward validated inner jobs succeeded worst episode base model 0 0.399 9/10 5/9 0.000 pilot-7 final 10 0.438 8/10 8/8 0.000 pilot-7b final 34 0.545 10/10 9/10 0.290 pilot-7c final 54 0.492 10/10 8/10 0.175 3. The agent learned to pick the better base model & better hyperparams Early training runs were model-selection blind: 77/79 episodes chose the weaker 0.6B model. After introducing the get_baseline_scores tool and uplift grading, the policy flipped and remained so during training, deepening across the arc: 1.7B share of job-writing episodes went 42% → 95% . It also adopted the exposed [prime_rl] config surface (21% → ~78% of episodes within one warm-start boundary), with a sensible key mix: sampling temperature, optimizer choice, algorithm variant, scheduler, loss. 🖥️ Infrastructure 16x warm GPU pods training at any one time: Inner loop (the jobs the agent writes): Runpod warm-pod fleet — a capped pool (up to 16) of pods, bootstrap-pinned to exact prime-rl + verifiers revisions so every node is a replica; ~2 min from provision to serving. Idle pods are reaped; the queue is file-backed ( queued/ → running/ → done/ ). GPU selection is data-driven — a benchmark matrix over GPU × base-model found 2× RTX A5000 wins cost at £0.10/job (~$0.13) ; preference ladders are walked at provision time to take whatever's in stock. What the fleet actually ran on (headline arc, ~1,750 jobs — below the nominal 54 steps × 40 because episodes that fail validation never dispatch a job): A40 64% (340 GPU-train-hours) · RTX 4090 32% (151h) · RTX A6000 3% · RTX A5000 1%. The benchmark cost-winner was rarely in stock, so the ladder spent most of the arc on A40s. prime-rl (GRPO) trains the small model; checkpoints are scored pre/post on the hidden eval with vLLM. Outer loop (training the agent): Tinker (Thinking Machines' managed RL API) trains Qwen3.6-35B-A3B with LoRA via tinker-cookbook's importance-sampling GRPO. A control-inversion bridge runs each episode as a background task behind a queue-backed policy, so the cookbook loop drives episodes turn-by-turn wh