넷플릭스, 영상 내 객체와 물리적 상호작용까지 완벽히 삭제하는 비디오 AI 'VOID' 공개

VOID: Video Object and Interaction Deletion Saman Motamed 1,2 , William Harvey 1 , Benjamin Klein 1 , Luc Van Gool 2 , Zhuoning Yuan 1 , Ta-Ying Cheng 1 1 Netflix 2 INSAIT, Sofia University "St. Kliment Ohridski" VOID removes objects from videos along with all interactions they induce on the scene — not just secondary effects like shadows and reflections, but physical interactions like objects falling when a person is removed. It is built on top of CogVideoX and fine-tuned for video inpainting with interaction-aware mask conditioning. Example: If a person holding a guitar is removed, VOID also removes the person's effect on the guitar — causing it to fall naturally. teaser-with-name.mp4 🤖 Models VOID uses two transformer checkpoints, trained sequentially. You can run inference with Pass 1 alone or chain both passes for higher temporal consistency. Model Description HuggingFace VOID Pass 1 Base inpainting model Download VOID Pass 2 Warped-noise refinement model Download Place checkpoints anywhere and pass the path via --config.video_model.transformer_path (Pass 1) or --model_checkpoint (Pass 2). ▶️ Quick Start The fastest way to try VOID is the included notebook — it handles setup, downloads the models, runs inference on a sample video, and displays the result: Note: Requires a GPU with 40GB+ VRAM (e.g., A100). For more control over the pipeline (custom videos, Pass 2 refinement, mask generation), see the full setup and instructions below. ⚙️ Setup pip install -r requirements.txt Stage 1 of the mask pipeline uses Gemini via the Google AI API. Set your API key: export GEMINI_API_KEY=your_key_here Also install SAM2 separately (required for mask generation): git clone https://github.com/facebookresearch/sam2.git cd sam2 && pip install -e . Download the pretrained base inpainting model from HuggingFace: hf download alibaba-pai/CogVideoX-Fun-V1.5-5b-InP \ --local-dir ./CogVideoX-Fun-V1.5-5b-InP The inference and training scripts expect it at ./CogVideoX-Fun-V1.5-5b-InP relative to the repo root by default. If ffmpeg is not available on your system, you can use the binary bundled with imageio-ffmpeg : ln -sf $( python -c " import imageio_ffmpeg; print(imageio_ffmpeg.get_ffmpeg_exe()) " ) ~ /.local/bin/ffmpeg 📁 Expected directory structure After cloning the repo and downloading all assets, your directory should look like this: VOID/ ├── config/ ├── datasets/ │ └── void_train_data.json ├── inference/ ├── sample/ # included sample sequences for inference ├── scripts/ ├── videox_fun/ ├── VLM-MASK-REASONER/ ├── README.md ├── requirements.txt │ ├── CogVideoX-Fun-V1.5-5b-InP/ # hf download alibaba-pai/CogVideoX-Fun-V1.5-5b-InP ├── void_pass1.safetensors # download from huggingface.co/void-model (see Models above) ├── void_pass2.safetensors # download from huggingface.co/void-model (see Models above) ├── training_data/ # generated via data_generation/ pipeline (see Training section) └── data_generation/ # data generation code (HUMOTO + Kubric pipelines) 📂 Input Format Each video sequence lives in its own folder under a root data directory: data_rootdir/ └── my-video/ ├── input_video.mp4 # source video ├── quadmask_0.mp4 # quadmask (4-value mask video, see below) └── prompt.json # {"bg": "background description"} The prompt.json contains a single "bg" key describing the scene after the object has been removed — i.e. what you want the background to look like. Do not describe the object being removed; describe what remains. { "bg" : " A table with a cup on it. " } // ✅ describes the clean background { "bg" : " A person being removed from scene. " } // ❌ don't describe the removal A few examples from the included samples: Sequence Removed object bg prompt lime the glass "A lime falls on the table." moving_ball the rubber duckie "A ball rolls off the table." pillow the kettlebell being placed on the pillow "Two pillows are on the table." The quadmask encodes four semantic regions per pixel: Value Meaning 0 Primary object to remove 63 Overlap of primary + affected regions 127 Affected region (interactions: falling objects, displaced items, etc.) 255 Background (keep) 🚀 Pipeline 🎭 Stage 1 — Generate Masks The VLM-MASK-REASONER/ pipeline generates quadmasks from raw videos using SAM2 segmentation and a VLM (Gemini) for reasoning about interaction-affected regions. 🖱️ Step 0 — Select points (GUI) python VLM-MASK-REASONER/point_selector_gui.py Load a JSON config listing your videos and instructions, then click on the objects to remove. Saves a *_points.json with the selected points. Config format: { "videos" : [ { "video_path" : " path/to/video.mp4 " , "output_dir" : " path/to/output/folder " , "instruction" : " remove the person " } ] } ⚡ Steps 1–4 — Run the full pipeline After saving the points config, run all remaining stages automatically: bash VLM-MASK-REASONER/run_pipeline.sh my_config_points.json Optional flags: bash VLM-MASK-REASONER/run_pipeline.sh my_config_points.json \ --sam2-checkpoint path/to/sam2_hiera_large.pt \ --device cuda This runs the following stages in order: Stage Script Output 1 — SAM2 segmentation stage1_sam2_segmentation.py black_mask.mp4 2 — VLM analysis stage2_vlm_analysis.py vlm_analysis.json 3 — Grey mask generation stage3a_generate_grey_masks_v2.py grey_mask.mp4 4 — Combine into quadmask stage4_combine_masks.py quadmask_0.mp4 The final quadmask_0.mp4 in each video's output_dir is ready to use for inference. 🎬 Stage 2 — Inference VOID inference runs in two passes. Pass 1 is sufficient for most videos; Pass 2 adds a warped-noise refinement step for better temporal consistency on longer clips. ✨ Pass 1 — Base inference python inference/cogvideox_fun/predict_v2v.py \ --config config/quadmask_cogvideox.py \ --config.data.data_rootdir= " path/to/data_rootdir " \ --config.experiment.run_seqs= " my-video " \ --config.experiment.save_path= " path/to/output " \ --config.video_model.model_name= " path/to/CogVideoX-Fun-V1.5-5b-InP " \ --config.video_model.transformer_path= " path/to/void_pass1.safetensors " To run multiple sequences at once, pass a comma-separated list: --config.experiment.run_seqs= " video1,video2,video3 " Key config options: Flag Default Description --config.data.sample_size 384x672 Output resolution (HxW) --config.data.max_video_length 197 Max frames to process --config.video_model.temporal_window_size 85 Temporal window for multidiffusion --config.video_model.num_inference_steps 50 Denoising steps --config.video_model.guidance_scale 1.0 Classifier-free guidance scale --config.system.gpu_memory_mode model_cpu_offload_and_qfloat8 Memory mode ( model_full_load , model_cpu_offload , sequential_cpu_offload ) The output is saved as <save_path>/<sequence_name>.mp4 , along with a *_tuple.mp4 side-by-side comparison. 🔁 Pass 2 — Warped noise refinement Uses optical flow-warped latents from the Pass 1 output to initialize a second inference pass, improving temporal consistency. Single video: python inference/cogvideox_fun/inference_with_pass1_warped_noise.py \ --video_name my-video \ --data_rootdir path/to/data_rootdir \ --pass1_dir path/to/pass1_outputs \ --output_dir path/to/pass2_outputs \ --model_checkpoint path/to/void_pass2.safetensors \ --model_name path/to/CogVideoX-Fun-V1.5-5b-InP Batch: Edit the video list and paths in inference/pass_2_refine.sh , then run: bash inference/pass_2_refine.sh Key arguments: Argument Default Description --pass1_dir — Directory containing Pass 1 output videos --output_dir ./inference_with_warped_noise Where to save Pass 2 results --warped_noise_cache_dir ./pass1_warped_noise_cache Cache for precomputed warped latents --temporal_window_size 85 Temporal window size --height / --width 384 / 672 Output resolution --guidance_scale 6.0 CFG scale --num_inference_steps 50 Denoising steps --use_quadmask True Use quadmask conditioning ✏️ Stage 3 — Manual Mask Refinement (Optional) If the auto-generated quadmask does not accurately capture the object or its interaction region, use the included GUI editor to r