AI에게 코드로 그림 그리는 법을 가르치다

Home Work Blog About Training AI to Paint with Code March 2026 · Research, Design, Development. March 2026 · Research, Design, Development. When you make an image with an AI model, the only way to participate is the prompt. You cannot edit the image directly. To change anything you go back to the model and prompt again. That limitation is what started this project. My friend Cameron and I trained a language model to make images by writing code, using reinforcement learning. The code is the artefact, and the code is editable. You can change what the model produced more granularly without going to prompt. The deeper question this project asks is how to do reinforcement learning on creative and design tasks. RL works when the reward is verifiable. A math problem is right or wrong. A game is won or lost. Aesthetic quality is neither. The design problem becomes the reward function and the criteria a judge is asked to apply. Too rigid, and the model converges. Too loose, and the model drifts. [Video of my thesis presentation, for the context behind this project.] My contributions Design Development RL Research The team Surya Cameron Franz Alex Wang A Watercolour painting of a Hibiscus flower made in code. A Watercolour painting of a Hibiscus flower made in code. How it works The system is a four-step loop, run thousands of times during training. The model receives a prompt, something like draw a peach hibiscus in watercolour , and writes a complete p5.brush JavaScript sketch. The sketch is rendered in a sandboxed Puppeteer environment, which produces a PNG. The PNG is judged against two random reference paintings sampled from a hand-rated pool, with a separate judge model picking the better watercolour. The judgment is converted into a reward signal, GRPO updates the model, and the loop runs again. The non-obvious choices live in what is being judged, how the judgment is made, what is in the reference pool, and how the system prompt is written. Each is the subject of a section below. reference pool · 581 refs · 117 love-tier two sampled per rollout thousands of iterations Prompt peach hibiscus in watercolour Model Qwen 3.5 35B writes JS Render Puppeteer → PNG Judge pairwise vs 2 refs Reward GRPO update prompt model tooling judge update The training loop. Reward Functions The first rubric had nine separate signals. A compilation gate. A check that the code actually used p5.brush rather than native p5. A code length ramp targeting around 3,000 tokens. HPSv3, a human preference model. Prompt adherence, judged by a council of GPT-5.4 and Gemini. And four more quality judges: recognisability, aesthetics, technique, depth. The model plateaued around 0.65 reward and stayed there. Every rollout looked the same. A flat, clip-art flower with five rounded petals. The reward kept going up but the capabilities didn't seem to improve. The diagnosis came from looking at the sub-rewards in isolation. The four quality judges plus prompt adherence were correlated with each other at 0.85 to 0.95. They were measuring the same thing five times. Code length, contributing roughly a third of the total reward, had saturated by step thirty and was producing zero gradient afterward. HPSv3, the one signal showing real variance, was weighted at 0.10. The rubric we made was telling the model the same thing over and over again. The fix had two halves. Replace absolute scoring with pairwise judgment. The original rubric asked the judge to score each rollout from zero to ten. The scores came back compressed near zero. Pairwise scoring asks a different question. The judge is shown the rollout, two references from the pool, and a single prompt: which of these is the better hibiscus watercolour? The reward is the fraction of comparisons it wins. The dynamic range opens up. The judge model handles a relative question more reliably than an abstract scale. Build a reference pool of hand-rated examples. 1,664 images, rated one at a time into love, okay, and nope. The 117 love-tier examples seeded the comparison pool. Every rollout from that point onward was being judged against the things I had decided were good. The next step, which we did not get to, would have been training a small reward model on the ratings themselves, (proper RLHF) so the model's sense of good could be applied without needing to compare against the pool every time. The new rubric collapsed all of it into four components: a binary compile-and-uses-brush gate (0.05), a binary length check (0.05), HPSv3 (0.30), and the pairwise judge against the reference pool (0.60). Same base model, same training data. The next run reached the previous plateau three times faster, kept climbing past it, and produced code that compressed from 13,500 tokens to under 2,000. The model learned that winning compositions did not need verbose code. The first rubric had nine separate signals. A compilation gate. A check that the code actually used p5.brush rather than native p5. A code length ramp targeting around 3,000 tokens. HPSv3, a human preference model. Prompt adherence, judged by a council of GPT-5.4 and Gemini. And four more quality judges: recognisability, aesthetics, technique, depth. The model plateaued around 0.65 reward and stayed there. Every rollout looked the same. A flat, clip-art flower with five rounded petals. The reward kept going up but the capabilities didn't seem to improve. The diagnosis came from looking at the sub-rewards in isolation. The four quality judges plus prompt adherence were correlated with each other at 0.85 to 0.95. They were measuring the same thing five times. Code length, contributing roughly a third of the total reward, had saturated by step thirty and was producing zero gradient afterward. HPSv3, the one signal showing real variance, was weighted at 0.10. The rubric we made was telling the model the same thing over and over again. The fix had two halves. Replace absolute scoring with pairwise judgment. The original rubric asked the judge to score each rollout from zero to ten. The scores came back compressed near zero. Pairwise scoring asks a different question. The judge is shown the rollout, two references from the pool, and a single prompt: which of these is the better hibiscus watercolour? The reward is the fraction of comparisons it wins. The dynamic range opens up. The judge model handles a relative question more reliably than an abstract scale. Build a reference pool of hand-rated examples. 1,664 images, rated one at a time into love, okay, and nope. The 117 love-tier examples seeded the comparison pool. Every rollout from that point onward was being judged against the things I had decided were good. The next step, which we did not get to, would have been training a small reward model on the ratings themselves, (proper RLHF) so the model's sense of good could be applied without needing to compare against the pool every time. The new rubric collapsed all of it into four components: a binary compile-and-uses-brush gate (0.05), a binary length check (0.05), HPSv3 (0.30), and the pairwise judge against the reference pool (0.60). Same base model, same training data. The next run reached the previous plateau three times faster, kept climbing past it, and produced code that compressed from 13,500 tokens to under 2,000. The model learned that winning compositions did not need verbose code. 0 50 100 150 200 0.00 0.25 0.50 0.65 0.75 1.00 training reward · 200 steps training step → reward old rubric · 9 signals new rubric · 4 signals 0.65 — old rubric ceiling what changed in the reward function ↓ OLD 9 signals · 5 redundant compile 5 % brush use 5 % length ramp 32 % HPSv3 10 % council 8 % recognise 10 % aesthetics 15 % technique 8 % depth 7 % 5 judges, ρ = 0.85–0.95 NEW 4 signals · pairwise dominates compile gate 5 % length 5 % HPSv3 30 % pairwise 60 % Old rubric vs new rubric, reward curves on the same axes. The reference pool The