우버의 재시도 폭풍(Retry Storm) 방지 기법

September 17, 2026 How Uber Protects Against Retry Storms DM Deepanshu Mehndiratta Senior Staff Engineer AS Alok Srivastava Principal Engineer VD Vibhor Dhingra Sr Software Engineer 1+ Share this article Facebook Linkedin X social Link Introduction Retry storms historically impact business operations and brand trust. While retry configuration tuning and retry budgets provide meaningful mitigation at the service level, they’re manually configured and lack visibility into cross-service amplification caused by deep dependency chains and fan-out patterns. As a result, it can be difficult to shield infrastructure against the domino effect triggered by a single service outage deeper in the stack. A key reason is that retry behavior today isn’t context-aware. While we can control how many retries occur, we can’t precisely control when they occur. This stems from the challenge of reliably distinguishing between errors generated by a service and those merely propagated through it. As a result, retries are applied uniformly rather than conditionally. This approach works for transient or low-rate failures. However, during moderate or severe degradation, it becomes counterproductive. Aggressively retrying against an already struggling service increases load, accelerates failure, and amplifies retry traffic across upstream dependencies. What begins as a localized outage can quickly escalate into a stack-wide incident—ultimately degrading, or in the worst case, completely breaking, the end user experience. One might argue that error codes from downstream services could be translated upstream to provide context for retries. While theoretically possible, this approach doesn't scale at Uber due to large fan-in and fan-out, evolving call flows, and the need for frequent adaptive changes. Therefore, we developed a context-aware mechanism in shared infrastructure to handle errors more efficiently. This blog explains the mechanism. Background Consider a simple call chain as shown in Figure 1, where the total number of requests arriving at Node A is Ƞ. By deduction, all nodes B, C, D, E, F, and G serve Ƞ requests in the steady state (when no node errors out). Figure 1: Call-chain with 1:1 fan-out, where a node calls its downstream exactly once for any incoming request. If service D starts erroring out and each service is configured to retry once (1 regular attempt and another attempt if the downstream fails), let’s look at the total number of requests served by each node. Figure 2: Call chain where a service errors out. Node A B C D E F G Depth 0 1 2 3 4 5 6 Requests Served Ƞ 2 × Ƞ 4 × Ƞ 8 × Ƞ 8 × Ƞ 8 × Ƞ 8 × Ƞ This can be distilled down to a simple formula, assuming the number of retries R is the same at every hop. ɗ denotes the depth of the node in the call chain, the number of requests served by the node if the node creates or passes through an error: Rɗ × Ƞ Retry Budgets We can optimize this by introducing retry budgets. Let’s assume the same retry budget at every hop represented by B. The new formula becomes: (1+B)ɗ × Ƞ Now, let’s try to see the number of requests served with a retry budget of 10%: Node A B C D E F G Depth 0 1 2 3 4 5 6 Requests Served Ƞ 1.1 × Ƞ 1.21 × Ƞ 1.33 × Ƞ 1.33 × Ƞ 1.33 × Ƞ 1.33 × Ƞ In the above example, the error originates at Node D, and if we limit the retry to only between Node D and Node C , and restrict entirely Node A and Node B from retrying on this error, we can guarantee a similar availability of the call-path without overburdening Node D, Node E, Node F , and Node G. Error Ownership Consider the same example of retry budgets while restricting retries between the edge from Node C to Node D , where the error originates. Node A B C D E F G Depth 0 1 2 3 4 5 6 Requests Served Ƞ Ƞ Ƞ 1.1 × Ƞ 1.1 × Ƞ 1.1 × Ƞ 1.1 × Ƞ Here, we clamp down the total number of requests served by all nodes from D till the leaf node G to just 10% over baseline, while allowing at least once retry for up to 10% of errors when they’re first returned. But what about the availability of Node D as seen by Node C ? Let’s run some numbers for various availability scenarios, and try to calculate ‌availability after retry. Base Availability % Base Error Rate % Retry Budget Error Rate after Retries % Availability after Retries % 99.9 0.1 10% 0.0001 99.9999 99 1 10% 0.01 99.99 95 5 10% 0.25 99.75 90 10 10% 1 99 80 20 10% 12 88 70 30 10% 23 77 As shown in the table above, for availability drops up to 10% in the callee node, even a single retry is helpful in getting the perceived availability by the caller node up to 99%. Beyond this, perceived availability drops significantly as a good chunk of requests are never retried due to the retry budget in place. This calculation assumes the errors from the callee are independent and that retries will lead to recovery. However, in many real-world scenarios like service overload, bad database hosts, database overload, or sharding issues, the probability of retries remains high even with retries. This contradicts the idea that retries to callee always increase perceived availability to the caller. It’s also this intuition that forms the basis of error ownership. During periods of high error rates from a service, the errors are less likely to be randomized, and wouldn’t benefit from a higher number of retries, and instead might be responsible for further degradation. Architecture The solution is about establishing error ownership, which can be explained using the symptom versus cause analogy. If a service calls N outbounds for fulfilling a request, and if an outbound error-out causes it to return an error, then the error returned by that service is only a symptom. Simultaneously, in the context of the service, the cause is the incoming error from its downstream. However, if no outbound of the service errors out while fulfilling the request and it still returns an error, the service is the cause of the returned error, and is the owner. In the next section, we discuss some possible solutions that can leverage this. Simple Correlation Claiming Error Ownership We use the Service Dependency Analysis Solution to correlate an inbound failure with an outbound failure and use the ruleset shown in Figure 3 for making or refuting error claims. Figure 3: Decision logic for claiming error ownership. Retrying with Error Ownership The caller uses the logic shown in Figure 4 to determine if it should retry the request. Figure 4: Decision logic for allowing retries. While the scenario of missing error claim headers is an uncooperative environment, it could occur because the downstream service doesn’t have the service dependency analysis solution, and is unable to correlate outbound and inbound errors. Or, the downstream service has missing context propagation, resulting in an incomplete correlation between outbound and inbound errors. Here, the first node to see a missing error claim from a downstream unclaims the error, limiting the impact radius of the retry disturbance (it’s no longer a storm), while still allowing sufficient retries to the error-returning service. Decision Matrix Callee Error Caller Error Callee Error Claim Caller Should Retry (Retry Middleware) Caller Propagated Error Claim No Yes NA NA Claim Yes Yes Missing Yes Unclaim Yes Yes Claimed Yes Unclaim Yes Yes Unclaimed No Unclaim Figure 5: 3 nodes used to demonstrate caller errors. When the edges A -> B and B -> C are fail-close, and the Node C returns an internal error that it’s claimed, Node B upon seeing the claimed error from C should retry to C. However, if the retry fails, it’d propagate the error to Node A, but while returning the error it must unclaim it. Node A upon seeing the error from B and the unclaimed error header shouldn’t retry the request to Node B. Figure 6: Decision logic for Error claim propagation. Coincidental Errors and Why We Need Service Dependency Analysis The decision matrix above covers cases where the downstream call fails or there’s an internal server er