C++ 비대칭 펜스(Asymmetric Fences)의 내부 원리

Asymmetric Fences? While browsing through Folly’s synchronization primitives to study up on concurrency concepts, one familiar-looking name in the codebase caught my attention: Asymmetric Thread Fence. I already knew what std::atomic_thread_fence is meant to achieve, but what does this construct do? And what exactly is asymmetric about it? That question led me down an interesting rabbit hole: understanding the details of asymmetric thread fences and what they are actually doing underneath the hood (at least on Linux). We’ll start with the C++ proposal for this construct, then work our way down into the implementation details. C++ P1202R0 Introduction We’ll walk through the mechanics in a later section, but for now it suffices to understand what these fences are trying to achieve. Referring to the proposal , let’s look at the overview: Some types of concurrent algorithms can be split into a common path and an uncommon path , both of which require fences (or other operations with non-relaxed memory orders) for correctness. On many platforms, it’s possible to speed up the common path by adding an even stronger fence type (stronger than memory_order_seq_cst ) down the uncommon path. These facilities are being used in an increasing number of concurrency libraries. We propose standardizing these asymmetric fences, and incorporating them into the memory model. Essentially, a concurrent algorithm may originally require a fence on both sides. However, if one path is significantly more common than the other, we may optimize for the common path by using a lighter fence there, while shifting the heavier synchronization cost onto the uncommon path. This can result in an overall performance improvement if we ensure the performance gain from the common path is much more than the overhead from the heavier fence of the uncommon path. This pattern is more common than it may first appear. If you browse through Folly’s codebase, you can find several uses of it: folly/synchronization/HazptrDomain.h : Hazard Pointers folly/synchronization/detail/ThreadCachedReaders.h : RCU folly/executors/ThreadPoolExecutor.cpp : Thread Pool ExWcutor Let’s begin with a modified example from the paper, known as Dekker’s example: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 atomic_int x { 0 }, y { 0 }; int r1 , r2 ; // Path A: x . store ( 1 , memory_order_relaxed ); atomic_thread_fence ( memory_order_seq_cst ); r1 = y . load ( memory_order_relaxed ); // Path B: y . store ( 1 , memory_order_relaxed ); atomic_thread_fence ( memory_order_seq_cst ); r2 = x . load ( memory_order_relaxed ); // Happens after both paths are completed. assert ( ! ( r1 == 0 && r2 == 0 )); // Never fails. We know the assertion cannot fail. Let’s examine why under the C++ memory model: Constraint $(1) \rightarrow (2)$ and $(1) \rightarrow (5)$ arise from the synchronizes-with relationship between thread construction and the start of the thread function [thread.thread.constr]/6 . This also establishes the modification order of both X and Y via the write-write coherence guarantees described in [intro.races]/15 . Constraint $(4) \rightarrow (5)$ and $(7) \rightarrow (2)$ are coherence-orderings that arise from the requirements in [atomics.order]/3.3 . For example, $(4)$ and $(5)$ are not the same atomic read-modify-write operation, and $(4)$ reads the value stored by $(1)$, and $(1)$ precedes $(5)$ in the modification order of Y . From [atomics.order]/4.4 , memory_order​::​seq_cst fence $A$ happens-before $(4)$, and $(5)$ happens-before memory_order​::​seq_cst fence $B$, therefore $(3)$ must precede $(6)$ in the single total order $S$ on all memory_order​::​seq_cst operations. By symmetric reasoning, $(6)$ must also precede $(3)$ in the same total order $S$. This is impossible, hence a contradiction. Therefore the execution where r1 == 0 && r2 == 0 is forbidden. Now, returning to asymmetric thread fences. The key idea is that in some concurrent algorithms, we care about optimizing a common path at the expense of an uncommon path . If we are willing to let the slow path absorb stronger synchronization costs, we can restructure the code using asymmetric fences. The paper illustrates this with the following transformation: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 // Globals: atomic_int x { 0 }, y { 0 }; int r1 , r2 ; // Fast path: x . store ( 1 , memory_order_relaxed ); asymmetric_thread_fence_light (); // Like atomic_signal_fence; // only inhibits compiler reordering r1 = y . load ( memory_order_relaxed ); // Slow path: y . store ( 1 , memory_order_relaxed ); asymmetric_thread_fence_heavy (); // Stronger than atomic_thread_fence(memory_order_seq_cst); r2 = x . load ( memory_order_relaxed ); // Happens after both fast path and slow path are complete. assert ( ! ( r1 == 0 && r2 == 0 )); // Never fails. That’s all we need to know about them for now. Let’s now move on to how these asymmetric fences are implemented. Asymmetric Light Fence - Compiler Barrier Let’s start with the simpler primitive: asymmetric_thread_fence_light . In Folly’s implementation, this is implemented using asm volatile("" : : : "memory") on GCC/Clang or _ReadWriteBarrier() on MSVC, with std::atomic_thread_fence(order) used as a fallback. As a side note, asm declarations are only conditionally supported in C++ according to [dcl.asm]/1 , so we will refer to GCC’s documentation, specifically the extended asm section for the exact semantics here. 1 2 3 4 asm asm-qualifiers ( AssemblerTemplate : OutputOperands [ : InputOperands [ : Clobbers ] ]) Here, we only care about the clobber list, specifically the special "memory" clobber argument: The "memory" clobber tells the compiler that the assembly code performs memory reads or writes to items other than those listed in the input and output operands (for example, accessing the memory pointed to by one of the input parameters). To ensure memory contains correct values, GCC may need to flush specific register values to memory before executing the asm. Further, the compiler does not assume that any values read from memory before an asm remain unchanged after that asm; it reloads them as needed. Using the "memory" clobber effectively forms a read/write memory barrier for the compiler . Note that this clobber does not prevent the processor from doing speculative reads past the asm statement . To prevent that, you need processor-specific fence instructions . This means that the compiler must preserve the ordering of memory reads and writes with respect to asm volatile("" : : : "memory") , effectively making it a compiler barrier . However, this places constraints only on instruction reordering performed by the compiler, not on the processor itself. The CPU may still execute memory operations out-of-order at runtime. As an additional note, we can also examine what the volatile qualifier does here: GCC’s optimizers sometimes discard asm statements if they determine there is no need for the output variables. Also, the optimizers may move code out of loops if they believe that the code will always return the same result (i.e. none of its input values change between calls). Using the volatile qualifier disables these optimizations . asm statements that have no output operands and asm goto statements, are implicitly volatile . Here, volatile simply prevents unwanted compiler optimizations on the asm statement itself. Strictly speaking, it is not necessary in this case since an asm statement without output operands is already implicitly volatile. Asymmetric Heavy Fence - membarrier() The key synchronization mechanism behind asymmetric fences lies in asymmetric_thread_fence_heavy . There are several implementation strategies, but we will focus on the usual Linux path: relying on the membarrier() syscall (Another alternative is to force a TLB shootdown, though I am not familiar enough with that approach to discuss it in detail). As before, std::atomic_thread_fence(order) is used as a fallback. Let’s take a look at the documentation: