애플 M3 뉴럴 엔진에서 50 GB/s 되찾기

Getting 50 GB/s Back Out of the ANE Aug 10, 2026 (3162 words) Introduction An RTL performance erratum in the Apple M3 Neural Engine throttles DRAM weight streaming throughput down to 17–19 GB/s from the nominal 45–60 GB/s, whenever the total weight size is an integer multiple of 1 MiB, which currently affects 7 of ANEMLL ’s 15 models. Avoiding the problematic path in the kernel DMA engine's speculative prefetch ring increased Llama 3.2 1B token throughput from 10.0 to 24.3 tokens/s (DRAM usage from 24.7 to 60.0 GB/s), and Qwen3-8B from 1.36 to 2.97 tokens/s (DRAM usage from 22.4 to 48.7 GB/s). llama-split4-benchmark.html ' style="display:block;width:1e3px;height:764px;border:0;transform-origin:top left"> Discovery I was profiling the neural engine's DRAM weight streaming throughput (GB/s) for single token decode: \[ X[1,D] \times W[D,N] = Y[1,N]. \] At \(N=4096\), I noticed that \(D=1536\) ran nearly 3× faster than \(D=2048\), the default used in Llama 3.2. STATIC (pure KernelDMA) median µs per replica, N=4096: D 576 768 1024 1280 1536 2048 rep a 150.4 196.8 238.1 293.8 310.9 997.6 rep b 157.8 190.2 250.1 288.3 326.8 995.0 rep c 148.7 189.6 249.4 275.2 316.5 995.4 Sweeping the D around the neighborhood of D = 2048: Huh? At D=2048, throughput was 16.93 GB/s. At D=2016, throughput was 44.5 GB/s, meaning 44.505062 − 16.930761 = 27.574301 GB/s (61.96% lower). A 27.57 GB/s drop , from 44.5 down to 16.93 GB/s. Note that the sweep data was collected on an M3 Air, repeated across 40 runs, under the same thermal/load conditions in a single run. I also ensured that the ANE register file's DMA size and address were the only variables being changed: D=2044 D=2048 D=2052 TD+0x004 estimated cycles 0x 000001e a 0x 000001e b 0x 000001e c TD+0x078 core 1 base 0x 00 0ff8 00 0x 00 1000 00 0x 00 1008 00 TD+0x07c core 2 base 0x 00 1ff 000 0x 00 200 000 0x 00 201 000 ... TD+0x0b0 core 15 base 0x 00 ef88 00 0x 00 f000 00 0x 00 f078 00 TD+0x0b4–0x0f0 core sizes ×16 0x 00 0ff8 00 0x 00 1000 00 0x 00 1008 00 TD+0x134 Common.Cin 0x 00000 7fc 0x 00000 800 0x 00000 804 TD+0x1f0 L2 source stride 0x 0000 7fc 0 0x 0000 800 0 0x 0000 804 0 TD+0x1f4 unknown stride mirror 0x 0000 7fc 0 0x 0000 800 0 0x 0000 804 0 TD+0x214 L2 result base 0x 0000 8fc 0 0x 0000 900 0 0x 0000 905 0 So then I sweep across the whole aperture of D: KernelDMA DRAM D-size sweep (interactive) KernelDMA DRAM D-size sweep 2,747 raw observations &middot; 67 D values &times; 41 randomized rounds &middot; Apple M3 all timed observations median trend ' style="display:block;width:1e3px;height:720px;border:0;transform-origin:top left"> That was a good idea, because I'm seeing a resonance at D = 2048. Never thought I’d do an FFT of throughput (GB/s) against tensor dimension (D), but here it is: Apparently the memory controller's throughput has a dominant harmonic with wavelength 2048 in tensor-dimension space. And sadly, it's a dip :(. All multiples of D = 2048 are similarly capped at a fixed bandwidth floor of 17-19 GB/s: At multiples of D = 2048, throughput sharply drops from the nominal 45–60 GB/s down to 17–19 GB/s, and recovers to nominal just ~256 lines away. This is not an RTL correctness bug, as kernel DMA still completes the transfer correctly. But the requests around 2048 are being forced into a separate, credit-starved issue regime, choking throughput by an unreasonable 28–43 GB/s (worst case 60→17), at transfer sizes that are, unfortunately, very common. Hypothesis 1 - DRAM spatial correlation Are the 16 cores aliasing onto the same DRAM bank at power-of-two strides? DRAM is a parallel data interface: DRAM bandwidth is the number of DQ (data) pins times the data rate per pin, \[ \text{DRAM BW} = N \times R = 128\ \text{bit} \times 6.4\ \text{GT/s} = 102.4\ \text{GB/s} \] M3's LPDDR-6400's 102.4 GB/s checks out with the advertised 100 GB/s. Sustained DRAM bandwidth is strictly that DQ utilization, and every GB/s short of the 102.4 GB/s DRAM ceiling means every extra cycle that the DQ line sat idle. DRAM TLDR: DRAM memory controller uses parallel accesses to stream bits through the high speed DQ pins; a large DRAM array is divided into banks and bandwidth (roughly) depends on spreading parallel requests across banks. Parallelism buys throughput if the resources are independent. If parallel requesters go for the same resource, their requests will serialize back-to-back and effectively be throttled at the single rate. A throttled floor at ~17–19 GB/s (while their immediate neighbors run at 45–60 GB/s), could be explained by collapse happening at pow-2 boundaries. It's also good to start low level: additional AXI requests can't do anything if they're requesting the same physical bank. Core Contention The neural engine has several avenues of parallelism, the first class being core-level parallelism. ANE has 16 cores in parallel. Cores divide work by partitioning a buffer evenly across \(N\) cores, and mutually agreeing to work on a different slice. We know the cores are assigned to fetch a different slice of the weight buffer, but ANE still has 16 cores all requesting their slice from DRAM in parallel, on the same cycle. If each core fetches their own slice from DRAM, then streaming latency should take the same amount of time whether one core or all 16 cores are enabled, because their requests should be serviced in parallel. However, if there is reduced bandwidth due to any core contention, then reducing the number of cores could ironically increase throughput, for the throttled D=2048 case. Sweeping the number of active cores for D=2048 and D=2016: Latency is constant from 1 to 16 active cores for both D=2016 and D=2048, meaning the throttling is present even at the core=1. The problem exists at the per-core level, the problem is replicated across cores. Address Contention Even after ruling out core-level contention, I still suspected some DRAM contention due to the power-of-two period. A power-of-two stride like 2048 adds \(2^k\) at each rotation, meaning the lower bits \([0..k-1]\) are constant. DRAM hashes the physical address so that strided access patterns get spatially decorrelated across different banks, so a hash collapsing the lower bits, or aliasing the upper \(2^k\) bit, could explain the pow2 periodicity. To test if DRAM spatial correlation is the issue, we scramble the address that the weights are fetched from. The address was randomly scrambled and spread across the whole ~64 MiB IOVA arena (59.90 MiB span), so it was scrambled both in-page and out-of-page. To rule out thermal drift on the fanless M3 Air, baseline and scrambled samples were interleaved run-to-run, so any thermal ramp hits both conditions equally. KernelDMA D=2048 dense vs scatter KernelDMA D=2048 throughput: every run, in order 287 interleaved rounds &middot; dense &amp; scatter measured back-to-back each round &middot; 16&nbsp;MiB per transfer &middot; hover any point. ' style="display:block;width:1e3px;height:620px;border:0;transform-origin:top left"> Median throughput of the baseline was 31.37 GB/s, and median throughput of the randomly scrambled addresses was 32.29 GB/s. The random scramble had a marginally higher sustained throughput of 1 GB/s average, suggesting that we may have attacked some spatial correlation through scrambling in this run, but (1) this is not proven across all cases (2) scrambling cannot recover the ~+200% throughput drop needed to explain the collapse. Hypothesis 2 - RTL integer wraparound Recall that the collapse repeated at every integer multiple of D = 2048: Q: What repeats at exact power-of-two integer boundaries? A: Integer overflows in fixed-width digital logic. module line_counter ( input wire clk, input wire reset, input wire advance, output reg [ 13 : 0 ] line_count ); always @( posedge clk) begin if (reset) line_count <= 14'h0000 ; else if (advance) line_count <= line_count + 1 'b1 ; // wrap at 0x3fff + 1 -> 0x0000 end endmodule Kernel Dimension \[ X[1,D] \times W[D,N] = Y[1,N]. \] Where \(D\)