다항식 오토인코더, 트랜스포머 임베딩 압축에서 PCA 능가

02 Blog The most direct way to compress an embedding (other than quantization) is to fit PCA on the corpus and keep the top-d eigenvectors. It works, but PCA is a linear projection, and neural-network embeddings on the sphere are structurally nonlinear — the well-known cone effect in transformers. Some of the variance lives in a nonlinear tail that a linear decoder can’t reach. This post is about a closed-form way to add a quadratic decoder on top of PCA, to capture part of that nonlinear tail. The encoder stays as plain PCA. The decoder is a degree-2 polynomial lift plus Ridge OLS (ordinary linear regression with L2 regularization), also closed-form. No SGD, no epochs, no hyperparameter search. One np.linalg.solve over corpus statistics. The construction itself isn’t mine. “PCA encoder + quadratic decoder + least-squares fit” appears in the dynamical-systems literature under the name quadratic manifold (see Jain 2017 , Geelen-Willcox 2022 / 2023 , Schwerdtner-Peherstorfer 2024+ — more in §9 ). I only stumbled onto these papers after running my experiments and believing the construction was new. The polynomial lift doesn’t come up much in modern ML conversations, and this post is a note about a useful trick from an adjacent discipline carrying over to retrieval. The concrete result. BEIR/FiQA, mxbai-embed-large-v1 (1024d), per-vector budget 512 bytes. Metric is NDCG@10 (Normalized Discounted Cumulative Gain over the top-10, the standard retrieval ranking-quality measure; range [0, 1], higher is better): method NDCG@10 Δ vs raw 1024d raw 1024d (4096 bytes) 0.4525 — PCA top-256 0.4168 -3.58 p.p. poly-AE 256d 0.4441 -0.85 p.p. matryoshka top-256 0.4039 -4.86 p.p. PCA already gives 4× per-vector memory compression at -3.58 p.p. NDCG. A quadratic decoder on top of PCA pulls another +2.73 p.p. — closing almost the entire gap to raw, at the same byte budget. Matryoshka is in the table as another familiar baseline (here it drops more than PCA — a known side observation, not the central claim of this post; see §3 footnote and §4 ). Measured on four models (nomic-v1.5, mxbai-large, bge-base, e5-base). Poly-AE over PCA: +1 to +4.4 p.p. at d=128, +0.03 to +2.7 p.p. at d=256. Full table in §3 . Full implementation — ~150 lines of numpy, MIT, repository github.com/IvanPleshkov/poly-autoencoder . The BEIR eval script is beir_eval.py in the same repo. Reproduces in 30–40 minutes on an M-series MacBook (10–15 min to encode the corpus, ~15 min for the Ridge solve at d=256). Contents: §1 what we’re comparing §2 setup §3 headline table §4 where the quadratic decoder helps §5 why linear projection loses §6 method §7 small-corpus caveat §8 residual compression §9 where the method came from §10 limits §11 what to try next 1. What we’re comparing Four lines in every measurement: raw — the full embedding, no compression. Quality ceiling and the most bytes per vector. The “expensive” baseline that gives a fair ceiling. matryoshka — embedding[:d] plus L2 normalization. On models trained with a matryoshka loss (nomic, mxbai in our sample), this is a valid matryoshka vector. On models without matryoshka training (bge-base, e5-base) it’s a test of what happens when you naively slice a non-MRL model — the scenario users of the bge family, e5, and custom-fine-tuned embeddings actually fall into. PCA — top-d eigenvectors of the corpus covariance. Vectors live in d-dimensional PCA coordinates. poly-AE — our method. Encode with PCA into p ∈ ℝ^d , decode with a quadratic polynomial back to a full D-dimensional V̂ , retrieve on V̂ . At a fixed d , all four methods store 2d bytes per vector (fp16 coordinates). 2. Experimental setup BEIR is the standard set of retrieval datasets (SciFact, FiQA, NFCorpus, TREC-COVID, etc.). The metric is NDCG@10. Corpus and queries are encoded with the chosen model, top-10 are retrieved by cosine similarity, and NDCG is computed against the labeled qrels. PCA and poly-AE are fit transductively : statistics are computed on the corpus we want to compress. Queries never participate in the fit — they hit a fixed encoder/decoder at inference time. This matches a production deployment: an index operator computes PCA + Ridge once on their data and then serves queries. For the main runs we use FiQA — 57K documents, 648 queries, 1706 qrels. 3. The headline table — four models NDCG@10 on FiQA at budgets of 256 fp16 (512 bytes/vector) and 128 fp16 (256 bytes): Model D d raw matryoshka† PCA poly-AE poly over matryoshka poly vs raw nomic-embed-text-v1.5 768 128 0.3746 0.3190 0.3273 0.3380 +1.90 p.p. -3.65 p.p. nomic-embed-text-v1.5 768 256 0.3746 0.3508 0.3670 0.3673 +1.65 p.p. -0.73 p.p. mxbai-embed-large-v1 1024 128 0.4525 0.3503 0.3689 0.4129 +6.26 p.p. -3.97 p.p. mxbai-embed-large-v1 1024 256 0.4525 0.4039 0.4168 0.4441 +4.02 p.p. -0.85 p.p. bge-base-en-v1.5* 768 128 0.4062 0.2914 0.3266 0.3654 +7.40 p.p. -4.09 p.p. bge-base-en-v1.5* 768 256 0.4062 0.3574 0.3688 0.3958 +3.84 p.p. -1.05 p.p. e5-base-v2* 768 128 0.3987 0.2498 0.3065 0.3317 +8.18 p.p. -6.70 p.p. e5-base-v2* 768 256 0.3987 0.3333 0.3618 0.3852 +5.19 p.p. -1.35 p.p. † The “matryoshka” column is embedding[:d] plus L2 normalization. On nomic and mxbai it’s a valid matryoshka vector. On models marked with * (bge-base, e5-base) the model wasn’t trained for matryoshka, and the slice here is a test of what happens when you naively slice a non-MRL model. This is the scenario that users of the bge family, e5, and custom-fine-tuned embeddings actually fall into — and we measure it here honestly. What this shows: Poly-AE is consistently ahead of PCA across all four models. Lift: +1 to +4.4 p.p. NDCG at d=128, +0.03 to +2.7 p.p. at d=256. Where the quadratic decoder actually helps and at what d — discussed in §4 . At d=256 , poly-AE loses 0.7–1.4 p.p. NDCG vs raw 768/1024 on all four models. 4× per-vector memory compression for less than 1.5 p.p. lost — the main number of the post. On non-matryoshka-trained models, the matryoshka column drops more than PCA — up to -15 p.p. NDCG at d=128. This is a side observation: the post compares PCA and poly-AE, not PCA vs matryoshka. If the matryoshka numbers in the table look surprising, there’s a short pointer in §4 . 4. Where the quadratic decoder actually helps PCA is the linear baseline. The quadratic decoder adds the nonlinear piece that the linear one can’t reach (mechanics in §5 ). How much does that actually help on retrieval, and at what d ? Poly-AE lift over PCA, by model and d : Model poly over PCA, d=128 poly over PCA, d=256 nomic-v1.5 +1.07 p.p. +0.03 p.p. mxbai-large +4.40 p.p. +2.73 p.p. bge-base +3.88 p.p. +2.70 p.p. e5-base-v2 +2.52 p.p. +2.34 p.p. The picture: At d=128 (8× compression) poly is consistently 1–4 p.p. ahead of PCA. This is the regime where the linear decoder starts dropping noticeable variance into the nonlinear tail, and the quadratic correction pulls it back. Sweet spot for the method. At d=256 (4× compression) the gap is uneven. On mxbai/bge/e5 — a stable +2.3–2.7 p.p. On nomic — close to zero (+0.03). Likely reason: nomic was carefully trained with multi-slice contrastive loss, its latent is more isotropic, and at d=256 the linear projection already takes most of what’s there. On non-MRL models the nonlinear tail is bigger → poly helps more. More anisotropy → bigger lift. The stronger the cone effect, the more variance lives in the nonlinear tail that PCA can’t reach but poly can. That’s the geometry §5 unpacks. Side: where matryoshka sits in the table In §3 you can see that on non-MRL models (bge, e5) the matryoshka column drops more than PCA — i.e. on a random non-MRL-trained model, naive slicing works worse than a corpus-side linear projection. This is a known result; the “MRL vs PCA on retrieval” question has been discussed independently of this post — see Matryoshka-Adaptor 2024 , SMEC “Rethinking MRL” 2025 , CoRECT 2025 , and the YouTube video literally titled «Is PCA enough?» . This post compares PCA and poly-AE; matryoshka is in