텍스트-음성 변환 모델을 50ms 미만 응답으로 만든 방법

TL;DR Our Qwen3-TTS 1.7B CustomVoice implementation achieves 10 requests per second (RPS) and sub-50 ms p95 time-to-first-audio (TTFA) while maintaining real-time playback on a single NVIDIA H100 SXM. × We compare five implementations: ours, vLLM-Omni, SGLang-Omni △ , VoxServe, and M*, under Poisson open-loop traffic. After tuning each implementation for low-latency streaming, ours is the only one to achieve sub-50 ms p95 TTFA . We maintain sub-50 ms p95 TTFA through 10 RPS and keep it below 100 ms even at 20 RPS . Our system produces approximately 630 characters per second at 10 RPS. At $4.29 per hour for a 1× H100 SXM instance, this translates to ~$2 per 1M characters at full utilization 1 . For comparison , ElevenLabs V3 is $100 / 1M and Cartesia Sonic 3.5 is $49 / 1M at a higher TTFA . We open source the implementation and benchmark . Our methodology is explained below. Defining “Real-time” TTS Let’s start by discussing what a real-time TTS server needs to achieve. We think it’s a four-part problem: Low Audible TTFA: Time from request dispatch to the first audible sample must be low. Zero underruns: Once playback starts, the client must not run out of buffered audio. Capacity: 1 and 2 must hold as RPS increases. Non-malformed output: Speech must be intelligible. We choose Qwen3-TTS CustomVoice 1.7B because it is one of the most popular TTS models with a permissive license. Based on the above definition, we target low p95 audible TTFA with zero underruns while maintaining high RPS on a single NVIDIA H100 SXM. All benchmarks run for five minutes under Poisson open-loop traffic to approximate real workloads, following Fireworks AI’s LLM benchmark . Each engine receives the complete text in a single HTTP request, while audio output remains streamed. We detect audible TTFA, reconstruct playback from received PCM, and evaluate the completed audio using Deepgram STT. How Do Other Engines Perform? The table below shows the upstream/default result at 1 RPS for each engine. We only apply changes for compatibility in this run. Engine p95 audible TTFA p95 leading silence Requests with underruns vLLM-Omni 277.883 ms 90 ms 100% SGLang-Omni △ 1,140.69 ms 80 ms 0% VoxServe 315.064 ms 30 ms 0% M* 1,159.956 ms 90 ms 0% These defaults have substantial room for improvement. We tune each serving engine for its own latency, continuity, quality, and capacity requirements. 1. Remove leading silence The first PCM returned by a model can contain tens of milliseconds of silence before the first sustained sound. This gap pushes audible TTFA back like so: We add a dynamic trim. It detects sustained speech from short RMS windows, removes samples before onset, and streams the remaining audio normally. This change improves TTFA by ~80ms but does not make model inference itself faster. 2. Tune frame accumulation We also tune how many codec frames are collected before decoding and releasing an audio chunk. Smaller initial chunks reduce TTFA, but provide less playback headroom and create more frequent decoder work. Larger chunks are easier to batch and make continuous playback safer, but delay the first audible output. A useful configuration therefore starts with a small chunk and increases the chunk size for later output. The exact knobs differ by engine: vLLM-Omni exposes settings such as codec_chunk_frames and codec_chunk_ramp ; the other engines provide equivalent chunk or stride controls. We iterate over these values to find the config that best matches: low p95 TTFA, zero underruns and stable behavior as load increases. Performance after tuning existing serving engines The following table shows the selected no-underrun profile for each engine after leading-silence and frame-accumulation tuning. Engine p95 TTFA (1 RPS) p95 TTFA (6 RPS) vLLM-Omni 56.815 ms 93.451 ms SGLang-Omni △ 120.879 ms 273.700 ms VoxServe 49.3 ms 363.2 ms M* 104.035 ms 179.501 ms VoxServe reaches sub-50 ms p95 TTFA at 1 RPS, while the other three engines do not. By around 6 RPS, every engine is at roughly 100 ms p95 TTFA or higher 2 . How We Optimized Qwen3-TTS We first need to understand Qwen3-TTS architecture. It is a 3-part model performing hierarchical multi-codebook generation. The Talker predicts the first codebook token for each audio frame, the Code Predictor generates the remaining 15 codebook tokens, and the causal Codec converts codebook tokens into waveform samples. Each module has its own compute profile, batching behavior, and latency requirements. Rather than optimizing each module in isolation, we focus on a broader question: how should a serving system coordinate these heterogeneous tasks? 1. Bringing three modules under one scheduler Most Qwen3-TTS serving implementations are split into two stages: the Talker and Code Predictor run together, while the Codec runs separately. This separation enables token generation and waveform decoding to overlap across requests. We take this a step further. We expose the Talker, Code Predictor, and Codec as three independently schedulable tasks. The key is not merely splitting them into parts, but bringing all three onto a shared scheduling surface managed by one scheduler. This design draws inspiration from M* ( arXiv ). With this setup, the scheduler can decide whether to run the Talker, advance the Code Predictor, or prioritize a Codec job that is approaching its playback deadline. It can also batch requests waiting for the same module. Instead of following a fixed execution order, we can rearrange work according to urgency. Combining the Talker and Code Predictor may appear more efficient because it removes an intermediate boundary. However, the combined operation can become a non-preemptible unit of work that blocks more urgent Code Predictor or Codec jobs. Keeping the modules separate creates shorter units of work and gives the scheduler more opportunities to interleave requests. 2. Scheduling around the needs of speech streaming Speech streaming has two distinct notions of urgency. Before the first chunk of audio arrives, every millisecond increases TTFA, so we need to prioritize this path. But once playback begins, the goal changes: the next chunk only needs to arrive before the current audio finishes playing. Producing it earlier provides no user-visible benefit. Thus, we give high priority to requests that have not produced their first audio, while established streams become urgent only as they approach a playback deadline. Running every urgent request alone would destroy batching efficiency. Instead, our scheduler selects an urgent request as an anchor and fills the rest of the batch with compatible work. This helps the critical request meet its deadline while making effective use of the GPU. This policy works especially well because all three modules share a scheduling surface, allowing the scheduler to choose both the request and the pipeline stage to advance. 3. Exploiting the regular structure of the Code Predictor The Code Predictor is an autoregressive transformer, but its execution is unusually regular. It always performs a fixed number of steps (15) per frame to fill the remaining audio codebooks. We exploit its fixed structure to preallocate its KV cache and capture the entire frame-generation loop as a single CUDA graph. We also use a Triton attention kernel specialized for its short, bounded context. By replacing a host-driven sequence with a fixed GPU program, we lower latency and simplify the execution system. 4. Rebuilding the Codec around cached state The Qwen3-TTS Codec is made up of Transformers and CNNs. Generating the next audio chunk depends on both the Transformer context and convolutional state from previous chunks. A naive implementation reprocesses the full frame history on every update, repeatedly decoding old audio as the utterance grows. To avoid this, we use a state-cache-based Codec. Each request retains the Transformer context and convolutional state needed by the next