트랜스포머 트랜스포머: 동작 특화 로봇 설계 모델

Transformer Transformer A Unified Model for Motion-Conditioned Robot Co-design Huy Ha , Karen Liu , Shuran Song Paper arXiv Video Code Not all robots are created equal Not all robots are created equal — but what if you could design one for a specific task? We propose Transformer Transformer , a unified model that does exactly this: hand it a manipulation demonstration, and it generates a complete robot — every link, joint, motor, and inertial property — optimized for that motion. We fabricated one such design for cloth flinging on an ALOHA 2 bimanual platform; it reduced tracking error by 73% and max joint speed by 30% versus the original. Behind that result is a diffusion transformer trained on RoboTokens , a unified tokenization of robot embodiments, states, and actions. The same architecture spans embodiment spaces (wheeled bimanual, quadrupeds, humanoids) and use cases (embodiment generation, cross-embodiment control). Rather than overfitting to one reward function, it is a dynamics model whose reward-agnostic predictions are converted into reward-specific value predictions at inference time, then used to steer embodiment diffusion through a procedure we call Dynamics Self-Guidance . Experiments across three design spaces show zero-shot optimization of unseen rewards and trajectories, improving performance and runtime over an evolutionary baseline. Technical Summary Video A 17-minute walkthrough — or scroll for the figure-and-caption version. What is the best robot for a given manipulation task? A robot's embodiment decides what tasks it can do well. We pour effort into making robots smarter — bigger datasets, better algorithms, more capable policy architectures — but a bad embodiment can hamper even a great policy. You can collect all the tossing data in the world, but if your robot's shape is far from optimal, it might toss itself instead of the ball. So let's frame the problem concretely. Given as input a target end-effector motion and a reward function, we want to generate a complete robot embodiment for the task — by complete we mean every link, joint, motor, and inertial property, plus a controller that drives the whole thing. We call this motion-conditioned robot co-design . A randomly procedurally-generated robot attempting the target motion: it falls before it can finish 🙃 Demonstrate, Generate, Validate Our framework reframes robot co-design as a three-step process: you demonstrate the desired end-effector motion (e.g., from a UMI 3 human demonstration), our model generates an optimized embodiment, and the same model validates the design by directly controlling it. The same network plays three roles — generator, critic, and cross-embodiment controller — by training jointly on one unified tokenization of robots. The paper has three parts: a unified robot representation (RoboTokens), a unified architecture (Transformer Transformer), and a unified training objective (Dynamics Self-Guidance). You have until the architecture section to figure out the pun 🥸 PART 1 RoboTokens A Unified Robot Representation If we want one model to co-design any robot, the network needs a shared vocabulary that spans every embodiment. RoboTokens are that vocabulary: typed tokens for every link, joint, motor, state, and action — organized so a single sequence describes both a robot's embodiment (time-invariant) and its dynamics (time-varying). RoboTokens has the following key design goals: Complete. A RoboToken sequence captures everything about a rigid articulated robot: 5 embodiment token types (links, fixed joints, rotating/sliding joints, ball joints, motors) plus state and action tokens. Flexible. Tokens reference each other by ID — a joint token knows the two link tokens it connects — so the same format handles a 6-DoF arm or a 35-DoF biped. Consistent. The tokenizer canonicalizes redundant spatial offsets and stores inertial / transform data in a single frame, reducing the variance the model has to learn through. Extensible. New token types slot in without changing the architecture — we add target end-effector pose tokens for trajectory tracking by pointing them at the relevant end-effector and timestep IDs, no retokenization of the rest of the robot required. Optimizable. Unlike autoregressive text formats like MJCF, diffusing continuous-valued RoboTokens gives both global controllability (later tokens influence earlier ones across denoising steps) and differentiability — the two ingredients needed for gradient-based reward optimization at inference time. We tokenized 11 robots from the MuJoCo Menagerie 10 , spanning two orders of magnitude in mass (0.65 kg dexterous hand to 67.5 kg quadruped) and from 6 to 35 active joints. Each became a sequence of 28–101 RoboTokens — small enough to diffuse end-to-end. PART 2 Transformer Transformer A Unified Architecture The model is a diffusion transformer (DiT 1 ) trained on RoboTokens with a DDIM 4 noise schedule. Noise is added to every input token except the ones we condition on — and by changing which tokens are masked, the same network wears different hats. Mask everything to get an unconditional robot generator. Mask the actions and you get a cross-embodiment controller 5 , 6 . Mask the embodiment and you get a motion-conditioned robot designer. Same weights, jointly trained on dynamics. (And about the pun: the first “Transformer” refers to robots whose embodiments can change. The second refers to the self-attention architecture 🥸) An embodiment-aware controller Because the controller sees the full embodiment as input, it can adjust its control strategy to the robot it's driving. Change link inertias, joint ranges, or motor gains, and the same policy keeps tracking — without any retraining per robot. We behavior-cloned whole-body controllers from RL experts 7 , 8 , 9 trained one-per-design on the procedural quadruped space. One policy, many quadrupeds. The same controller tracks the same trajectory across quadrupeds with continuous variations (link dimensions) and discrete variations (DoF, knee direction, spring-loaded vs serial leg). A generator over diverse robot classes Run the model with no conditioning at all and it diffuses out complete robots from Gaussian noise. Trained on diverse robots from Menagerie 10 , a single Transformer Transformer produces fixed-base arms, quadrupeds, humanoids, and dexterous hands. Because diffusion is stochastic, every run gives different robots. Robots, diffused. A single model generates fixed-base arms, quadrupeds, humanoids, and dexterous hands from the same noise schedule. PART 3 Dynamics Self-Guidance Zero-shotting Rewards Now the actual co-design problem: how do we generate a high-reward robot for a reward function the model has never seen during training? Here is the trick. The same model predicts both the embodiment and its dynamics, so we can feed those predictions into the user's reward function and get a predicted reward . With predicted rewards in hand, we can run the model many times in parallel, predict the reward for each candidate, and return the best one. That is best-of-N sampling, but at GPU speed. We can do better. Because the predicted reward is a differentiable function of the embodiment tokens, we can take its gradient and inject it back into the diffusion process — pushing samples toward higher reward at every denoising step, following classifier-guided DDIM 11 . The model is, in effect, asking itself: “how should this robot change to increase the reward?” We call this Dynamics Self-Guidance . Why “self-guidance”: related work that incorporates dynamics into diffusion guidance trains a separate dynamics model 12 or relies on a differentiable simulator 13 . Here, the model uses its own dynamics predictions to steer itself. Reward in, robot out Take a randomly-generated quadruped that falls during a dynamic toss. Ask our model to generate a robot for the same trajectory with only a t