OpenAI의 WebRTC 문제점

published 5/6/2026 OpenAI’s WebRTC Problem OpenAI posted a technical blog a few days ago. This blog post triggered me more than it should have. I urge to slap my meaty fingers on the keyboard. You should NOT copy OpenAI. I don’t think you should use WebRTC for voice AI. WebRTC is the problem. Me Like 6 years ago I wrote a WebRTC SFU at Twitch. Originally we used Pion (Go) just like OpenAI, but forked after benchmarking revealed that it was too slow. I ended up rewriting every protocol, because of course I did! Just a year ago, I was at Discord and I rewrote the WebRTC SFU in Rust. Because of course I did! You’re probably noticing a trend. Fun Fact : WebRTC consists of ~45 RFCs dating back to the early 2000s. And some de-facto standards that are technically drafts (ex. TWCC, REMB). Not a fun fact when you have to implement them all. You should consider me a Certified WebRTC Expert . Which is why I never, never want to use WebRTC again. Product Fit I’m going to cheat a little bit and start with the hot takes before they get cold. Don’t worry, we’ll get right back to talking about the OpenAI blog post and load balancing, I promise. WebRTC is a poor fit for Voice AI. But that seems counter-intuitive? WebRTC is for conferencing, and that involves speaking? And robots can speak, right? WebRTC is too aggressive Let’s say I pull up my OpenAI app on my phone. I say hi to Scarlett Johansson Sky and then I utter: should I walk or drive to the car wash? WebRTC is designed to degrade and drop my prompt during poor network conditions. wtf my dude WebRTC aggressively drops audio packets to keep latency low. If you’ve ever heard distorted audio on a conference call, that’s WebRTC baybee. The idea is that conference calls depend on rapid back-and-forth, so pausing to wait for audio is unacceptable. …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate. After all, I’m paying good money to boil the ocean, and a garbage prompt means a garbage response. It’s not like LLMs are particularly responsive anyway. But I’m not allowed to wait . It’s impossible to even retransmit a WebRTC audio packet within a browser; we tried at Discord. The implementation is hard-coded for real-time latency or else . Yes, Voice AI agents will eventually get the latency down to the conversational range. But reducing latency has trade-offs . I’m not even sure that purposely degrading audio prompts will ever be worth it. TTS is faster than real-time You speak into the microphone, it gets sent to one of OpenAI’s billion servers, and then a GPU pretends to talk to you via text-to-speech. Neato. Let’s say it takes 2s of GPUs to generate 8s of audio. In an ideal world, we would stream the audio as it’s being generated (over 2s) and the client would start playing it back (over 8s). That way, if there’s a network blip, some audio is buffered locally. The user might not even notice the network blip. But nope, WebRTC has no buffering and renders based on arrival time . Like seriously, timestamps are just suggestions. It’s even more annoying when video enters the picture. To compensate for this, OpenAI has to make sure packets arrive exactly when they should be rendered. They need to add a sleep in front of every audio packet before sending it . But if there’s network congestion, oops we lost that audio packet and it’ll never be retransmitted. OpenAI is literally introducing artificial latency, and then aggressively dropping packets to “keep latency low”. It’s the equivalent of screen sharing a YouTube video instead of buffering it. The quality will be degraded . Fun fact : WebRTC actually adds latency. It’s not much, but WebRTC has a dynamic jitter buffer that can be sized anywhere from 20ms to 200ms (for audio). This is meant to smooth out network jitter, but none of this is needed if you transfer faster than real-time. Ports Ports Ports Okay but let’s talk about the technical meat of the OpenAI article. We’re no longer on a boat , but let’s talk about ports. When you host a TCP server, you open a port (ex. 443 for HTTPS) and listen for incoming connections. The TCP client will randomly select an ephemeral port to use, and the connection is identified by the source/destination IP/ports. For example, a connection might be identified as 123.45.67.89:54321 -> 192.168.1.2:443 . But there’s a minor problem… client addresses can change. When your phone switches from WiFi to cellular, oops your IP changes. NATs can also arbitrarily change your source IP/port because of course they can. Whenever this happens, bye bye connection , it’s time to dial a new one. And that means an expensive TCP + TLS handshake which takes at least 2-3 RTTs. The users definitely notice the network hiccup when you’re live streaming. WebRTC tried to solve this issue but made things worse. Seriously . A WebRTC implementation is supposed to allocate an ephemeral port for each connection. That way, a WebRTC session can identified by the destination IP/port only; the source is irrelevant. If the source IP/port changes, oh hey that’s still Bob because the destination port is the same. But as OpenAI corroborates, this causes issues at scale because… Servers only have a limited number of ports available. Firewalls love to block ephemeral ports. Kubernetes lul You could probably abuse IPv6 to work around this, but IDK I never tried. Twitch didn’t even support IPv6… Hacks by Necessity So most services end up ignoring the WebRTC specifications. Because of course they do. We mux multiple connections onto a single port instead. At Twitch I literally hosted my WebRTC server on UDP:443 . That’s supposed to be the HTTPS/QUIC port, but lying meant we could get past more firewalls. Like the Amazon corporate network, which blocked all but ~30 ports. Discord uses ports 50000-50032, one for each CPU core. As a result it gets blocked on more corporate networks. But like, if you’re on a Discord voice call on the Amazon corporate network, you probably won’t be there much longer anyway. HOWEVER, HUGE PROBLEM . WebRTC is actually a bunch of standards in a trenchcoat, and 5 of those go over UDP directly. It’s not hard to figure out which protocol a packet is using, but we need to figure out how to route each packet. STUN : We can choose a unique ufrag and route on it. SRTP/SRTCP : The browser chooses a random ssrc (u32)… which we can usually route based on. DTLS : Uh oh. We pray that RFC9146 gets widespread support. TURN : IDK I’ve never implemented it. So OpenAI only uses STUN: No protocol termination: Relay parses only STUN headers/ufrag; it uses cached state for subsequent DTLS, RTP, and RTCP, keeping packets opaque. It’s a positive way of saying: We really hope the user’s source IP/port never changes, because we broke that functionality. While it’s impressive load balancing anything at OpenAI scale, their custom load balancing is a hack. But a necessary hack, because the core protocol is at fault. Fun fact : Browsers can randomly generate the same ssrc . If there is a collision, and no source IP/port mapping is available, Discord attempts to decrypt the packet with each possible decryption key. If the key worked, hey we identified the connection! Round Trips and U The OpenAI blog post starts with 3 requirements, one of them is: Fast connection setup so a user can start speaking as soon as a session begins lol It takes a minimum of 8* round trips (RTT) to establish a WebRTC connection. While we try to run CDN edge nodes close enough to every user to minimize RTT, it adds up. Signaling server (ex. WHIP ): 1 for TCP 1 for TLS 1.3 1 for HTTP Media server: 1 for ICE (with server) 2 for DTLS 1.2 2 for SCTP * It’s complicated to compute, because some protocols can be pipelined to avoid 0.5 RTT. Kinda like half an A-Press . All of this nonsense is because WebRTC needs to support P2P. It doesn’t matter if you have a server with a static IP address, you still need to do this dance. It’s extra depressing when the signali