에이전트 드로우: 당신이 말하는 동안 AI가 그려주는 도구

We recently built 2draw , a Drawful-style game where players draw on a shared canvas and race to guess each other&#x27;s drawings, on tldraw , an infinite-canvas SDK for React. We started wondering what it would take to put an agent in that loop, as an opponent or a rival guesser, and dug into how an agent could read and draw on a tldraw canvas. That research turned into: Agent draw , a tool that lets an agent draw to the canvas for you while you present. Here you can see the agent assisting me in my demo presentation of a third grade chemistry class. You can try it right now, or grab the source: Live demo: tldraw-agent-draw-demo.james-664.workers.dev Source: github.com/ritza-co/tldraw-agent-draw-demo Agent Draw is an agent that draws while you present ​ Drag a rectangle on the canvas, say what you want inside it, and by the time you look back it&#x27;s there, drawn by an AI agent while you kept talking. Drag a few rectangles in a row and they queue up, each drawing in turn. All of this happens on an infinite canvas tool called tldraw. What is tldraw? ​ tldraw is an infinite-canvas SDK for React: the same editor API a user drives with a mouse, an agent can drive in code, creating shapes, moving them, drawing arrows between them. We did not build our agent from scratch. tldraw already publishes an official Agent starter kit that draws and arranges shapes through a chat panel, backed by a Cloudflare Worker. We built Agent Draw on top of that. How good is an agent at drawing? ​ A simple composition built on top of tldraw&#x27;s primitives, rectangles, diamonds, arrows, is something most models handle well. A more intelligent model tends to get more ambitious with the composition, and is better at placement on the canvas. We tested this by giving each model the same two requests in one session: draw a decision diagram, and draw a person playing cricket. claude-opus-4.8 handled both well, a clean decision diagram built from primitives, and, more interestingly, a fully realised cricket scene sketched with the pen tool: Where it struggles ​ That was claude-opus-4.8 , one of the more capable models available. The result looks different with a smaller model behind the same requests. A smaller model settles for less ​ Give the same two requests to a smaller model, and the ambition drops off fast. claude-haiku-4.5 matches Opus on the decision diagram, but for the cricket request it stays with primitives instead of reaching for the pen, and settles for a simpler, static composition with a label rather than a dynamic sketch: A weaker model can give up on the task ​ Drop down further to google/gemini-2.5-flash-lite , and it seems to give up early on both requests: How we made agent draw ​ The whole feature is a new canvas tool, a speech pipeline, a serialized draw queue, and a prompt section, and here is each piece, with the actual code from the repo . Capturing the region you draw ​ tldraw tools are state machines. You subclass StateNode and define child states; tldraw routes pointer events to whichever state is active. Our AreaCaptureTool has three states ( idle → pointing → dragging ) and does its real work on pointer-up, when the dragged rectangle is final: class AreaCaptureDragging extends StateNode { static override id = &#x27;dragging&#x27; private bounds : BoxModel | undefined = undefined override onPointerUp ( ) { this . editor . updateInstanceState ( { brush : null } ) if ( ! this . bounds ) throw new Error ( &#x27;Bounds not set&#x27; ) // Hand the captured rectangle (in page coordinates) to the capture session. startCaptureSession ( this . bounds ) this . parent . transition ( &#x27;idle&#x27; ) } updateBounds ( ) { if ( ! this . initialPagePoint ) return const currentPagePoint = this . editor . inputs . getCurrentPagePoint ( ) const x = Math . min ( this . initialPagePoint . x , currentPagePoint . x ) const y = Math . min ( this . initialPagePoint . y , currentPagePoint . y ) const w = Math . abs ( currentPagePoint . x - this . initialPagePoint . x ) const h = Math . abs ( currentPagePoint . y - this . initialPagePoint . y ) // Show tldraw&#x27;s native selection brush while dragging. this . editor . updateInstanceState ( { brush : { x , y , w , h } } ) this . bounds = { x , y , w , h } } } We get the live selection-brush rectangle for free by writing to editor.updateInstanceState({ brush }) , the same instance state tldraw&#x27;s own select tool uses. The bounds are in page coordinates, so they stay correct no matter how the user has panned or zoomed. Listening while you talk ​ The moment a capture starts, we open the mic. AreaRecorder is a thin wrapper over the browser&#x27;s MediaRecorder , deliberately with no knowledge of the agent or transcription, just start() and stop() : export class AreaRecorder { async start ( ) : Promise < void > { this . stream = await navigator . mediaDevices . getUserMedia ( { audio : true } ) const recorder = new MediaRecorder ( this . stream , { mimeType : this . mimeType } ) this . chunks = [ ] recorder . ondataavailable = ( event ) => { if ( event . data . size > 0 ) this . chunks . push ( event . data ) } recorder . start ( ) this . recorder = recorder } async stop ( ) : Promise < Blob > { // ...stop the recorder, release the mic, resolve with the recorded clip... return new Blob ( this . chunks , { type : this . mimeType || &#x27;audio/webm&#x27; } ) } } Turning speech into text ​ The audio blob is posted to a new /transcribe route on the same Cloudflare Worker that already serves the agent. The route just forwards the audio to Mistral&#x27;s Voxtral transcription model and returns the text, so a server-held key covers every demo visitor by default: export async function transcribe ( request : IRequest , env : Environment ) { const form = await request . formData ( ) const file = form . get ( &#x27;file&#x27; ) // A visitor&#x27;s own key (if they&#x27;ve added one) takes priority over the server&#x27;s. const mistralKey = request . headers . get ( &#x27;x-mistral-api-key&#x27; ) || env . MISTRAL_API_KEY const outForm = new FormData ( ) outForm . append ( &#x27;file&#x27; , file ) outForm . append ( &#x27;model&#x27; , &#x27;voxtral-mini-transcribe-2507&#x27; ) const mistralResponse = await fetch ( &#x27;https://api.mistral.ai/v1/audio/transcriptions&#x27; , { method : &#x27;POST&#x27; , headers : { Authorization : ` Bearer ${ mistralKey } ` } , body : outForm , } ) const data = ( await mistralResponse . json ( ) ) as { text ? : string } return new Response ( JSON . stringify ( { text : data . text ?? &#x27;&#x27; } ) , { headers : { &#x27;Content-Type&#x27; : &#x27;application/json&#x27; } , } ) } Handling more than one drawing at a time ​ This is the part that makes multiple captures work. You can drag a second rectangle while the first is still drawing, and the captures draw in order rather than fighting over the canvas. The whole thing is a module-level state machine over a tldraw atom : a single active recorder, a FIFO queue, and a single-consumer worker that handles one session at a time. Starting a new capture auto-stops the one still recording and queues it: export function startCaptureSession ( bounds : BoxModel ) : string { // Drawing a new capture ends the audio of the one still recording. if ( recordingId ) finalizeRecording ( recordingId ) const id = nextId ( ) sessions . set ( [ ... sessions . get ( ) , { id , bounds , status : &#x27;recording&#x27; } ] ) const rec = new AreaRecorder ( ) recorder = rec recordingId = id rec . start ( ) . catch ( /* surface a mic-permission error on the session */ ) return id } The consumer drains the queue one session at a time, moving each through transcribing → drawing . Serializing here is the whole point: the agent runs one request at a time, so a newer capture simply waits its turn: async function processQueue ( ) : Promise < void > { if ( processing ) return // single consumer processing = true try { while ( queue . length > 0 ) { const id = queue . shift ( )