마이크로소프트 VibeVoice 실전 튜토리얼

Editors Pick Agentic AI Technology Artificial Intelligence Language Model Audio Language Model Staff TTS Tutorials Uncategorized Voice AI In this tutorial, we explore Microsoft VibeVoice in Colab and build a complete hands-on workflow for both speech recognition and real-time speech synthesis. We set up the environment from scratch, install the required dependencies, verify support for the latest VibeVoice models, and then walk through advanced capabilities such as speaker-aware transcription, context-guided ASR, batch audio processing, expressive text-to-speech generation, and an end-to-end speech-to-speech pipeline. As we work through the tutorial, we interact with practical examples, test different voice presets, generate long-form audio, launch a Gradio interface, and understand how to adapt the system for our own files and experiments. Copy Code Copied Use a different Browser !pip uninstall -y transformers -q !pip install -q git+https://github.com/huggingface/transformers.git !pip install -q torch torchaudio accelerate soundfile librosa scipy numpy !pip install -q huggingface_hub ipywidgets gradio einops !pip install -q flash-attn --no-build-isolation 2>/dev/null || echo "flash-attn optional" !git clone -q --depth 1 https://github.com/microsoft/VibeVoice.git /content/VibeVoice 2>/dev/null || echo "Already cloned" !pip install -q -e /content/VibeVoice print("="*70) print("IMPORTANT: If this is your first run, restart the runtime now!") print("Go to: Runtime -> Restart runtime, then run from CELL 2.") print("="*70) import torch import numpy as np import soundfile as sf import warnings import sys from IPython.display import Audio, display warnings.filterwarnings('ignore') sys.path.insert(0, '/content/VibeVoice') import transformers print(f"Transformers version: {transformers.__version__}") try: from transformers import VibeVoiceAsrForConditionalGeneration print("VibeVoice ASR: Available") except ImportError: print("ERROR: VibeVoice not available. Please restart runtime and run Cell 1 again.") raise SAMPLE_PODCAST = "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav" SAMPLE_GERMAN = "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav" print("Setup complete!") We prepare the complete Google Colab environment for VibeVoice by installing and updating all the required packages. We clone the official VibeVoice repository, configure the runtime, and verify that the special ASR support is available in the installed Transformers version. We also import the core libraries and define sample audio sources, making our tutorial ready for the later transcription and speech generation steps. Copy Code Copied Use a different Browser from transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration print("Loading VibeVoice ASR model (7B parameters)...") print("First run downloads ~14GB - please wait...") asr_processor = AutoProcessor.from_pretrained("microsoft/VibeVoice-ASR-HF") asr_model = VibeVoiceAsrForConditionalGeneration.from_pretrained( "microsoft/VibeVoice-ASR-HF", device_map="auto", torch_dtype=torch.float16, ) print(f"ASR Model loaded on {asr_model.device}") def transcribe(audio_path, context=None, output_format="parsed"): inputs = asr_processor.apply_transcription_request( audio=audio_path, prompt=context, ).to(asr_model.device, asr_model.dtype) output_ids = asr_model.generate(**inputs) generated_ids = output_ids[:, inputs["input_ids"].shape[1]:] result = asr_processor.decode(generated_ids, return_format=output_format)[0] return result print("="*70) print("ASR DEMO: Podcast Transcription with Speaker Diarization") print("="*70) print("\nPlaying sample audio:") display(Audio(SAMPLE_PODCAST)) print("\nTranscribing with speaker identification...") result = transcribe(SAMPLE_PODCAST, output_format="parsed") print("\nTRANSCRIPTION RESULTS:") print("-"*70) for segment in result: speaker = segment['Speaker'] start = segment['Start'] end = segment['End'] content = segment['Content'] print(f"\n[Speaker {speaker}] {start:.2f}s - {end:.2f}s") print(f" {content}") print("\n" + "="*70) print("ASR DEMO: Context-Aware Transcription") print("="*70) print("\nComparing transcription WITH and WITHOUT context hotwords:") print("-"*70) result_no_ctx = transcribe(SAMPLE_GERMAN, context=None, output_format="transcription_only") print(f"\nWITHOUT context: {result_no_ctx}") result_with_ctx = transcribe(SAMPLE_GERMAN, context="About VibeVoice", output_format="transcription_only") print(f"WITH context: {result_with_ctx}") print("\nNotice how 'VibeVoice' is recognized correctly when context is provided!") We load the VibeVoice ASR model and processor to convert speech into text. We define a reusable transcription function that enables inference with optional context and multiple output formats. We then test the model on sample audio to observe speaker diarization and compare the improvements in recognition quality from context-aware transcription. Copy Code Copied Use a different Browser print("\n" + "="*70) print("ASR DEMO: Batch Processing") print("="*70) audio_batch = [SAMPLE_GERMAN, SAMPLE_PODCAST] prompts_batch = ["About VibeVoice", None] inputs = asr_processor.apply_transcription_request( audio=audio_batch, prompt=prompts_batch ).to(asr_model.device, asr_model.dtype) output_ids = asr_model.generate(**inputs) generated_ids = output_ids[:, inputs["input_ids"].shape[1]:] transcriptions = asr_processor.decode(generated_ids, return_format="transcription_only") print("\nBatch transcription results:") print("-"*70) for i, trans in enumerate(transcriptions): preview = trans[:150] + "..." if len(trans) > 150 else trans print(f"\nAudio {i+1}: {preview}") from transformers import AutoModelForCausalLM from vibevoice.modular.modular_vibevoice_text_tokenizer import VibeVoiceTextTokenizerFast print("\n" + "="*70) print("Loading VibeVoice Realtime TTS model (0.5B parameters)...") print("="*70) tts_model = AutoModelForCausalLM.from_pretrained( "microsoft/VibeVoice-Realtime-0.5B", trust_remote_code=True, torch_dtype=torch.float16, ).to("cuda" if torch.cuda.is_available() else "cpu") tts_tokenizer = VibeVoiceTextTokenizerFast.from_pretrained("microsoft/VibeVoice-Realtime-0.5B") tts_model.set_ddpm_inference_steps(20) print(f"TTS Model loaded on {next(tts_model.parameters()).device}") VOICES = ["Carter", "Grace", "Emma", "Davis"] def synthesize(text, voice="Grace", cfg_scale=3.0, steps=20, save_path=None): tts_model.set_ddpm_inference_steps(steps) input_ids = tts_tokenizer(text, return_tensors="pt").input_ids.to(tts_model.device) output = tts_model.generate( inputs=input_ids, tokenizer=tts_tokenizer, cfg_scale=cfg_scale, return_speech=True, show_progress_bar=True, speaker_name=voice, ) audio = output.audio.squeeze().cpu().numpy() sample_rate = 24000 if save_path: sf.write(save_path, audio, sample_rate) print(f"Saved to: {save_path}") return audio, sample_rate We expand the ASR workflow by processing multiple audio files together in batch mode. We then switch to the text-to-speech side of the tutorial by loading the VibeVoice real-time TTS model and its tokenizer. We also define the speech synthesis helper function and voice presets to generate natural audio from text in the next stages. Copy Code Copied Use a different Browser print("\n" + "="*70) print("TTS DEMO: Basic Speech Synthesis") print("="*70) demo_texts = [ ("Hello! Welcome to VibeVoice, Microsoft's open-source voice AI.", "Grace"), ("This model generates natural, expressive speech in real-time.", "Carter"), ("You can choose from multiple voice presets for different styles.", "Emma"), ] for text, voice in demo_texts: print(f"\nText: {text}") print(f"Voice: {voice}") audio, sr = synthesize(text, voice=voice) print(f"Duration: {len(audio)/sr:.2f} seconds") display(Audio(audio, rate=sr)) print("\n" + "="*70) print("TTS DEMO: Compare All Voice Presets") print("="*70) comparison_text = "VibeVoice produces remark