FLUX 3 x mimic: 차세대 비디오-행동 모델의 등장

Back to blog Research Models FLUX 3 x mimic: The Next Generation of Video-Action Models An early version of FLUX 3, our new multimodal foundation model , is now running on robots. We gave mimic robotics early access to FLUX.3. Their strength in robot learning and deployment, combined with the model's world knowledge and BFL's foundation model expertise, produced FLUX-mimic: the next generation of video-action models. FLUX FLUX 1 and FLUX 2 generate images. FLUX 3 expands into multimodality and generates audio-visual content jointly - and, at the same time, provides the foundation of FLUX-mimic: A video-action model, developed in collaboration with mimic, running robots that have been tested and deployed at Audi. At first glance, producing convincing visual content and controlling robots seem to have little in common. One requires generating pixels, the other an understanding of how the physical world responds when you touch and manipulate it. If one model does both, it was never really only a content creation model. It is a model of how the world behaves, and content creation is one thing one can do with it. That is what FLUX 3 is. Video is the hard part FLUX 3 is one model, jointly trained across images, video and audio from the beginning. The most demanding part of that training - accounting for over 95% of the total compute costs - is video prediction. To generate realistic videos, a model has no choice but to learn contact, motion, weight, cause and effect; get any of them wrong and it looks wrong. Learning to render the world accurately means learning how the world behaves. Relatively speaking, audio is the easy modality. Low dimensional and far less detailed than video, it makes up less than 0.5% of the tokens in a 720p video with audio. Once a model has done the hard work of learning video understanding, it will learn the causal relationship between video and audio to predict speech synchronized to lip movement and audio effects synchronized to the physical events causing them. Actions follow the same shape: a low dimensional representation of a robot's state, tightly coupled to visual observations. Actions, audio and video frames are all partial representations of a single underlying physical reality. After the model has learned about the physical processes behind video and audio, action prediction is not a new departure - it is one more view of the reality it already models. A single backbone If that framing is correct, teaching FLUX 3 to predict actions should not incur lasting costs: we expect a brief phase of disturbance as the model has to learn the structure of the action space and align its internal representation of the world to it, before returning to full performance. That is exactly what we observe. In a large-scale training run, we added action prediction to the curriculum and observed the effect on video generation quality. Human ratings on text-to-video and image-to-video initially fell by up to 10% as the model started to incorporate the new action modality. After 3500 steps, the model had regained its full previous quality on video generation tasks while now also predicting actions. Each series is normalized to its own quality before action prediction was added. Higher is better. The model had to integrate actions into its inputs and outputs - but doing so didn't cost it capacity permanently. It merely had to learn how this new modality relates to its existing model of the world. Once this was figured out, the performance penalty on its existing capabilities was gone. Video generation and action prediction don't need separate foundations. The same backbone carries both. This makes Physical AI a natural extension of our roadmap at Black Forest Labs rather than a change in direction. Content creation is what our multimodal FLUX 3 backbone does with image, video and audio. Physical AI is what it does with actions. One foundation model, with visual intelligence at its core, enabling two families of applications. We didn't build a separate foundation model. We focused on the hard thing: building a model that understands the world. Acting in it is what that understanding makes possible. From lab to reality: FLUX-mimic What happens when we point the FLUX 3 backbone at real automation tasks on real production lines? That's the question mimic and BFL created FLUX-mimic to answer. mimic builds their own robots and brings expertise in robot learning, dexterous manipulation and production deployment; BFL builds visual foundation models and brings multimodal training and modeling expertise. Together, we built a next-generation model for general-purpose manipulation - adapted to industry requirements and integrated into mimic's full-stack deployment system. FLUX-mimic is a video-action model built on the FLUX 3 backbone. Decoding the learned world model Our thesis is that FLUX 3 has to learn an internal representation of the world to be able to generate videos. FLUX-mimic follows through on this thesis and decodes actions from the learned world representation of the FLUX backbone. This approach, pioneered in mimic-video, trains a lightweight action decoder on top of intermediate features extracted from the video prediction path of FLUX. Architecture overview how FLUX-mimic is built on top of FLUX 3 The success of this approach depends on two related but different aspects: the quality of the world model learned by FLUX and the quality of the feature representation of this world model. The quality of the world model is directly related to the generation quality: if a model does not understand how the world behaves, it cannot simulate it. However, even the best world model does not help an action decoder if it is inaccessible: if the feature space keeps the causal relationships between modalities entangled nonlinearly, understanding those relationships from the feature representation remains as difficult as understanding them from the raw inputs - representation quality matters. Generation quality and representation quality have long been studied and approached in isolation from each other. Generative approaches result in high-quality world models that enable simulations and they exhibit scaling laws for predictable returns on compute investments. However, compared to more specialized approaches for representation learning they produce less disentangled representations, which puts a ceiling on their usefulness for tasks that require world understanding. As generative models themselves rely on their own features, this divergence in their representation quality seems counter-intuitive. Improved representations within generative models should improve the quality of their world model and make them more usable for downstream tasks. In our work, Self-Flow , we demonstrated how to unify generation and representation learning in a single framework and observed exactly this reciprocal improvement: the world model improved - as measured by generation quality across video, image and audio - and its representation quality improved - as measured by success rate for robot control tasks in simulation. Self-Flow vs. Flow Matching (FM). Left: generation error (Fréchet distance) per modality, each normalized to FM = 100 (lower is better). Right: success rate on manipulation tasks averaged over four task groups through finetuning (higher is better). Scaling the world model Scaling laws remain true with Self-Flow, and FLUX 3 is the application of that: the scaled-up version of Self-Flow. It is trained on tens of millions of hours of general video content to learn world dynamics as broadly as possible from day one, and on hundreds of thousands of hours of video content focused on human and robot manipulation tasks to be ready as a backbone for visual intelligence. This scaling is what translates the success of Self-Flow from the lab to reality. mimic deployed FLUX-mimic in real factory use cases spanning the daily reality