Sparse Attention · Part II

The object at the center of the selector systems in this part is the hard top-\(k\) mask. Write it as \(\operatorname{topkmask}(\mathbf z) = \arg\max_{\mathbf y\in C_k}\langle\mathbf z,\mathbf y\rangle\), the maximizer of a linear function over the capped simplex \(C_k\), whose vertices are the \(\binom nk\) hard \(k\)-subsets, so the argmax lands on the vertex selecting the \(k\) largest scores. Part I §4.3 stated the trouble in one line. Away from ties this map is locally constant, so its Jacobian is identically zero; at ties it jumps. Zero almost everywhere is worse than noisy, because the loss could be reduced by selecting differently and the gradient says nothing about how.

A sizable research area manufactures a gradient anyway, by relaxing or randomizing the selection; surveying it is not this post’s job. What matters here is what deployed systems do about the dead gradient. Some avoid learnable scorers altogether. The two frontier-scale systems examined in §3 train one by distilling the softmax attention it replaces, with the selector cut out of the graph, and explain why. Either way, one system-level problem remains for everybody: given scores, find the threshold fast, at context lengths where moving memory, not doing arithmetic, is the scarce resource (§4). That second problem is where this post earns the word “systems,” and where the false dichotomy from the introduction finally dissolves.

Here is the fact that triggered this series. Open the proceedings of a recent conference and “sparse attention” is everywhere, with dozens of papers per cycle. Read them closely and most are not about sparse models at all. They take a model that was pretrained with dense softmax attention and, at inference time, skip computation or evict state. That is one family. The other family makes sparsity part of the model itself, present during training, so that training and inference compute the same thing. The line between the two determines what each method can and cannot achieve, and the papers themselves blur it often enough that this section draws it once, carefully, before anything else.

Training-free methods need no retraining, which is their great practical appeal, and the best of them are excellent engineering. But they inherit a hard constraint. The model they run on assigns strictly positive weight to every token (Part I §2.2), so any sparse shortcut is an approximation of a dense distribution. Whenever it keeps a strict subset, it discards some probability mass \(\delta > 0\). The output error that mass induces is at most \(2B\delta\) for values bounded by \(B\), a tight bound, and it is generically nonzero whenever \(\delta>0\) (Duarte et al., 2026, Props. 1–2). Larger allowances shrink the floor; no allowance short of the full context makes it provably zero. Every training-free method in this post is a way of controlling that floor, not of removing it: FFD’s top-δ test discards only mass below \(e^{-\delta}\) of the peak, BLASST’s \(\ln\lambda\) rule carries an explicit error bound, SALE calibrates per-head thresholds, Twilight prunes to a top-\(p\) core (§4.4). Good engineering, honest bounds, same ceiling.

Trainable methods make sparsity native. Block-native systems like NSA, MoBA and InfLLM-V2 learn which blocks to attend to; DSA and MSA train token- and block-level selectors at frontier scale (§3); and the entmax line trains attention that is sparse at the level of individual tokens, with exact zeros. Because the model is genuinely sparse, inference stops being an approximation problem and becomes support recovery, and it can be exact. The superset guarantee of §4.1 is the formal version, and it has no training-free analogue: restrict a softmax model to any subset and the answer changes, restrict an entmax model to a superset of its support and it does not.

One honest wrinkle. A thin middle band retrofits trained components onto a frozen or lightly tuned backbone: SeerAttention distills block gates from the model’s own attention maps (Gao et al., 2025), DuoAttention learns which heads deserve a full cache (Xiao et al., 2024), and Dynamic Memory Compression retrofit-trains per-head eviction decisions (Nawrot et al., 2024). They buy accuracy with a little training, but the target they are trained to imitate is still the dense softmax model, so for the question that organizes this post they sit with the training-free family, under the same ceiling. The frontier systems of §3 are a different case again, trained selectors inside natively sparse models, distilled for reasons of their own.

This line is why the post keeps the families apart to the end. Section 3 covers the trained systems and what they say about gradients. Section 4 covers the kernels, where the two families meet, because bracketing a threshold is the same trick whether the threshold guards an approximation or recovers an exact support; the difference lives in the last column of the lanes table, in what “exact” means. Section 5 lays the whole territory out in one table, family by family.

Two frontier-scale systems now run sparse attention with a learned selector: DeepSeek-V3.2 and MiniMax’s MSA. Neither differentiates through selection. Both train the selector by distilling the softmax attention it replaces, with the selector explicitly cut out of the computational graph. Unusually, and to their credit, both papers say exactly why. This section quotes them because the wording is the evidence.

3.1 DeepSeek-V3.2, in its own words

DSA scores every prefix token with a “lightning indexer,” \(I_{t,s} = \sum_{j} w^I_{t,j}\cdot\mathrm{ReLU}(\mathbf q^I_{t,j}\cdot\mathbf k^I_s)\), a few FP8 heads with ReLU chosen “for throughput consideration,” and keeps the top \(k=2048\) tokens per query (DeepSeek-AI, 2025). Training is two-stage. A dense warm-up freezes everything except the indexer and trains it by KL divergence toward the L1-normalized, head-summed softmax attention distribution: 1,000 steps × 16 sequences × 128K tokens = 2.1B tokens. Then the sparse stage switches selection on and continues the KL on the selected set only, while the main model trains on language modeling: 15,000 steps × 480 × 128K = 943.7B tokens. And the sentence this series has been circling:

“It is worth noting that we detach the indexer input from the computational graph for separate optimization. The training signal of the indexer is from only \(\mathcal L_I\), while the optimization of the main model is according to only the language modeling loss.”

— DeepSeek-AI (2025), §2.1, on DSA’s sparse training stage

Read it plainly. No task-loss gradient reaches the selector directly; the task influences it only indirectly by moving the teacher it imitates. The indexer also scores all \(L\) prefix tokens for each query, \(O(L^2)\) per layer over the sequence, which is why §4’s kernel literature exists. The released inference code carries the indexer and its top-\(k\) selection in kernel.py of deepseek-ai/DeepSeek-V3.2-Exp.

3.2 MiniMax MSA, in its own words, with the ablation

MSA is the block-level counterpart on GQA. Two added projection matrices score tokens, max-pool to blocks, take top-\(k\) blocks per query and GQA group (the local block always included), and run exact block-sparse softmax over the selection; the selected-attention cost per query is fixed as length grows (Lai et al., 2026) (released kernels in sparse.py of MiniMax-AI/MSA). Their §3.2 opens with the cleanest statement of the problem in the systems literature:

“The top-\(k\) selection in Equation 7 is non-differentiable, so the language-modeling loss cannot train the index Q/K projections \(W_q^{\text{idx}}, W_k^{\text{idx}}\) directly. We therefore train the Index Branch with a KL alignment loss and use three mechanisms to stabilise sparse training: Gradient Detach, Indexer Warmup, and a forced Local Block.”

— Lai et al. (2026), §3.2

The teacher is the probability-level average of the per-head softmax over the selected tokens, with stop-gradient on the teacher and on the indexer’s input, so the KL updates only the two index projections. What makes MSA especially useful here is that they ablated the alternatives (10B-parameter pilot, their App. B). Give the indexer an additive attention output so the LM loss trains it (“LM loss only”) and it “performs poorly on long-context retrieval: without an objective on the top-\(k\) selection itself, the indexer receives little direct pressure to select relevant blocks.” Use “KL loss only” and short-context ability drops (the removed output path was doing work). And un-detach the KL and two failure modes appear: gradient spikes propagating into the backbone, “LM-loss divergence within a few hundred steps,” and, more insidious:

“We attribute this regression to a self-distillation effect: the backbone can lower the KL loss by simplifying the Main Branch attention distribution, rather than by improving the Index Branch.”

— Lai et al. (2026), App. B.3

The student, given influence over the teacher, dumbs the teacher down. Their final recipe keeps the KL and the detach, then drops the indexer’s value head after warm-up. At scale, it trains a natively multimodal, 109B-parameter model on 3T tokens to match its GQA baseline. At 1M context, the paper reports 28.4× less per-token attention compute, 14.2× faster prefill, and 7.6× faster decoding on H800 GPUs.

3.3 The recipe predates them

Detached distillation from softmax attention was not invented at the frontier. SeerAttention trains block-level “attention gates” by KL toward the 2D-max-pooled attention map of the frozen model: “only requires training the gate parameters,” with the observation that “jointly training the gate and model from scratch, as in MoE, is costly and difficult” (Gao et al., 2025). DuoAttention learns per-head gate values (full attention vs. streaming) with a distillation loss on synthetic data, model frozen, and binarizes them at deployment (Xiao et al., 2024). The other systems dodge learning a scorer at all. NSA reuses the softmax scores of its compressed-attention branch as block importance, with no auxiliary loss. Its discussion of alternative selection strategies says explicitly that with a separate learned scorer, “since the selection operation is non-differentiable, importance score computation based on neural networks relies on auxiliary loss” (Yuan et al., 2025); InfLLM-V2 goes parameter-free precisely because a learned compressor in its position “would not receive gradients” (Zhao et al., 2026) (CUDA kernels at OpenBMB/infllmv2_cuda_impl); MoBA’s gate is mean-pooling plus top-\(k\), no parameters to train (Lu et al., 2025) (its fused kernel is moba_efficient.py in MoonshotAI/MoBA). Across the deployed long-context systems surveyed here, the pattern is consistent. None passes a task gradient through its selector. They imitate softmax attention (DSA, MSA, SeerAttention), reuse it (NSA, InfLLM-V2), or hard-code around it (MoBA).

3.4 What detaching buys, what it costs, and an open question

What it buys is legible in MSA’s ablations: stability (no KL spikes into the backbone), locality (each layer’s selector gets a clean, well-shaped supervision signal), and immunity from the self-distillation collapse, because the teacher cannot be gamed if the student cannot touch it. What it costs is equally legible. The selection is trained to reproduce softmax attention, not directly to serve the task. No task gradient reaches the selector; the only route from the language-modeling objective is indirect, through changes to the teacher distribution. The trained-in adaptivity results of Part I §4.1 sit on the other side of exactly this line.

And one more thing follows from Part I that I have not seen stated, so let me state it carefully as a question rather than a claim. The distillation target in both systems is a softmax attention distribution. Part I’s dispersion bound says that, if the logit spread remains bounded, that distribution’s entropy climbs toward \(\log n\) as context grows. At 128K and beyond, the teacher is, provably, drifting toward the very dispersion that motivated sparsity in the first place. DSA’s two training stages run on 128K-token sequences. What does it cost to train a selector to imitate a flattening target? Does the imitation inherit the dispersion, does the top-\(k\) truncation mask it, does it not matter at all? I do not know, and I have found no measurement of it in public work.

I will not pretend neutrality here. I find the detached recipe simultaneously completely reasonable and deeply unsatisfying. Reasonable, because MSA’s ablation is real evidence that the naive alternatives are worse, and because in a 3T-token training run for the 109B-parameter MSA model, stability is not a nicety. Unsatisfying, because “the loss cannot train the projections directly” is a statement about hard top-\(k\), not about sparse allocation in general. α-entmax is a counterexample. It produces exact zeros with an exact, cheap Jacobian (Part I §4.3), and nothing of that kind appears in either paper’s experiments. The honest summary is that differentiable selection at frontier scale is not refuted; it is untested. That is the open problem, and I will leave it exactly there.

Gradient or no gradient, everyone still has to find the threshold, whether it is a top-\(k\) cutoff over a million scores inside a fused kernel or an entmax normalizer during training. In the last two years, three communities with almost disjoint bibliographies (serving-kernel engineers, the softmax-threshold line, and the entmax line) have converged on one recipe: compute a coarse, cheap statistic of the score distribution; use it to bracket the threshold; act conservatively outside the bracket; refine only inside it. This section documents the convergence, because I know of nowhere else it is laid out side by side.

First, the paradigm shift that makes any of this worth doing, because it is where §1’s false dichotomy dissolves. The old objection to unstructured sparsity was arithmetic: deciding whether a token matters means computing its score, a \(d\)-dimensional dot product, and the output contribution it might save is another \(d\)-dimensional multiply-accumulate, so the test costs the same order of compute as the work it hopes to skip. When compute was the bottleneck, input-dependent sparsity could not pay for itself. Tensor cores inverted the ledger. A pre-tensor-core P100 SXM2 could execute roughly 29 half-precision FLOPs in the time it took to move one byte through HBM (21.2 TFLOPS against 732 GB/s (NVIDIA, 2016)); an H100 SXM executes roughly 295 (≈989 dense bf16 TFLOPS against 3.35 TB/s (NVIDIA, 2022)), an order of magnitude more compute per byte. This widened gap makes HBM traffic decisive in many long-context attention regimes, which is FlashAttention’s founding observation: “a missing principle is making attention algorithms IO-aware,” i.e., reducing “the number of memory reads/writes between GPU high bandwidth memory (HBM) and GPU on-chip SRAM” (Dao et al., 2022). In that regime the economics of sparsity flip. An operator may happily spend extra FLOPs (α-entmax’s root-finding iterations cost more arithmetic than softmax’s single normalization) provided those FLOPs decide memory reads and writes, because in those regimes skipped bytes, not skipped multiplications, are where wall-clock time lives. Every kernel in this section is that sentence, implemented three different ways.

Figure 2 separates the three data flows. Score computation does not disappear. What changes is whether tensor-sized intermediates cross the HBM boundary, and whether later passes have to touch value blocks that are already known to contribute zero.

4.1 The pattern, and why conservative errors preserve correctness

The recipe works because of two small results that make one-sided errors harmless. The first is special to exact sparsity:

Conservative supersets are exact (Treviso et al., 2022, Prop. 1; Duarte et al., 2026, Prop. 2). Let \(\mathbf p = \alpha\text{-entmax}(\mathbf z)\) with support \(S\), and let \(M \supseteq S\) be any candidate set. Then \(\alpha\text{-entmax}(\mathbf z|_M) = \mathbf p\): restricting attention to any superset of the support reproduces the full result exactly. Recall is the only thing that matters; precision only buys speed.

Proof

The normalization \(\sum_{i\in M}[(\alpha-1)z_i-\tau]_+^{1/(\alpha-1)} = 1\) is solved by the full-problem \(\tau^\star\), since coordinates in \(M\setminus S\) contribute zero at \(\tau^\star\); by uniqueness of the root, the restricted problem returns the same \(\tau^\star\), hence the same weights. ∎

This is the property that first licensed predicting entmax supports before computing any scores (Treviso et al., 2022), and it is what makes exact sparse decoding a support-recovery problem rather than an approximation (Duarte et al., 2026). The second is a triviality worth stating because an entire kernel literature leans on it:

A subsample’s \(k\)-th statistic is a one-sided bound. For any \(T\subseteq[n]\) with \(|T|\ge k\): the \(k\)-th largest score within \(T\) is \(\le\) the \(k\)-th largest score overall. So a threshold estimated from any subsample errs only toward keeping too much; it can never cause a false rejection of a true top-\(k\) element.

Proof

Let \(a\) be the \(k\)-th largest in \(T\). At least \(k\) elements of \(T\) are \(\ge a\); those elements also lie in \([n]\), so \([n]\) has at least \(k\) elements \(\ge a\), whence the global \(k\)-th largest is \(\ge a\). ∎

One more shared move deserves a name before the tour: anchor at the maximum. The entmax threshold always lives in a unit-length window below the top score, \(m - 1 \le \tau^\star \le m - n^{1-\alpha}\) with \(m = (\alpha{-}1)\max_i z_i\) (Peters et al., 2019), and the softmax-side thresholds of §4.4 are all of the form “within \(\delta\) of the (running) max.” The max is the one statistic every streaming kernel computes anyway; every lane builds its bracket on it.

4.2 The entmax lane: AdaSplash-2 up close

Training-time entmax attention needs \(\tau\) for every query row, inside the FlashAttention tiling, without materializing \(n\times n\) scores. AdaSplash did it with a safeguarded Halley-bisection hybrid (cubic local convergence, bisection fallback whenever a step exits the bracket), cutting ~23 bisection iterations to ~3, a 15× solver speedup (2.38 ms vs. 36.67 ms for standard-Gaussian rows of length 8192, averaged over 1000 runs on one H100), and making α-entmax training competitive with FlashAttention-2 (Gonçalves et al., 2025). AdaSplash-2 then attacked the initialization: shift scores so the max is 1, bin \([0,1]\) into \(B\) equal bins in SRAM, and solve the normalization on the histogram instead of the scores (Gonçalves et al., 2026). Figure 3 shows the kernel’s anatomy.

The estimate this produces is not merely close. It is provably conservative.

The histogram bracket (Gonçalves et al., 2026, Prop. 1). Let \(\tau^\star\) solve the exact normalization \(f(\tau)=0\) and \(\tau_h\) solve the histogram version \(f_h(\tau)=0\), where each score is replaced by the left edge of its bin of width \(h=1/B\). Then

$$\tau^\star - h \;<\; \tau_h \;\le\; \tau^\star .$$

The estimate is conservative, never above the truth, so the support it induces is a superset of the true support (the superset guarantee then applies), and the error is at most \(1/B\).

Proof sketch

Left-edge binning lowers every score by less than \(h\). At the exact root this gives \(f_h(\tau^\star) \le f(\tau^\star)=0<f_h(\tau^\star-h)\). Continuity and strict decrease then place the histogram root in \((\tau^\star-h,\tau^\star]\). ∎

The histogram itself uses a packing trick. With \(B = 8\), each local accumulator packs eight 8-bit counters into one uint64. A query row maintains \(B_c\) such accumulators, one per column position in a query-key block; shift-and-add updates them without atomics, and a final shift-and-mask reduction produces the row histogram. From \(\tau_h\), the refinement solver (Halley for \(\alpha\le1.5\), Newton for \(\alpha\in(1.5,2]\), dispatched by the smoothness dial of Part I §4.3) typically converges in 1–2 iterations. Figure 4 draws the bracket.

Solving for \(\tau\) is half the kernel. The other half is spending the sparsity it certifies. While the weights are computed, the kernel also builds a binary block mask \(\mathcal M \in \{0,1\}^{T_r\times T_c}\) marking which score blocks contain any nonzero attention weight, bit-packed with 32 column-blocks per int32, so the whole mask costs \(O(T_r \times T_c)\) bits of memory. The output pass and the entire backward pass then never touch the full grid. A population count (popc) tells each row how many blocks survive, and find-next-set (fns) jumps straight from one surviving block to the next, giving \(O(\|\mathcal M\|_0)\) traversal instead of \(O(T_r \times T_c)\) (Gonçalves et al., 2026). A skipped block’s values are never read from HBM in the output pass, and its gradients are never written in backward. That is exactly where the economics of §4’s opening cash out. Figure 5 walks the trick on a toy grid.

The interactive α-entmax to AdaSplash explainer lets you step through the threshold solver, histogram bracket, sparse Jacobian, and bitmask traversal.

All of this is public code, and short. The fused Triton kernels live in adasplash_v2.py of deep-spin/adasplash (pip install adasplash); the first-generation block-mask variant is adasplash_block_mask.py in the same package.

On random Gaussian inputs, with sparsity controlled through query variance, the AdaSplash-2 kernels are over 2× faster than both FlashAttention-2 implementations at high block sparsity (causal, bf16, head dim 64, 4K–128K, one A6000). The same work trains a 1-billion-parameter entmax language model for 50B tokens at 4K context, then extends it to 32K with 10B ProLong tokens. At decode time, EntmaxKV plays a related game against the KV cache. Per-page coordinate-wise min/max key boxes give a deterministic upper bound on every in-page score; paired with a conservative threshold estimate, that policy provably cannot miss a support token. Its Gaussian-aware selector trades this worst-case guarantee for adaptivity, reaching support recall 0.9977 at 16K context and 3.36×/5.43× over full-cache softmax/entmax decoding at 1M tokens (A6000, batch 8) (Duarte et al., 2026).

4.3 The serving lane

GPU top-\(k\) selection has been a bucket game since long before LLMs: bucketSelect and radixSelect (histogram the values by range or by digits, find the bucket containing the \(k\)-th element, recurse into it) date to 2012 (Alabi, Blanchard, Gordon & Steinbach, 2012), with bitonic-select database variants (Shanbhag, Pirk & Madden, 2018), a comprehensive modern study and the adaptive AIR top-\(k\) (Zhang, Naruse, Li & Wang, 2023), and RadiK’s scalable radix select, which also documents that the peaked score distributions of LLM inference are adversarial for radix digits, fixed by a random shift (Li et al., 2024). What the sparse-attention wave added is fusion with the scorer, and the numbers explain the urgency: profiling GLM-5.2 at 1M-token prefill across eight B200s, the DSA scoring-plus-selection kernel is 83.7% of prefill runtime; attention over the 2048 selected tokens is 4.5% (Yin et al., 2026).

LiteTopK is one recent instance of the fused design, and it puts the full bracketing pattern in one kernel. It samples the previous chunk’s top-\(3k\) scores to estimate the range (temporal coherence supplies the sample; §4.1’s one-sided bound supplies the safety), builds an equal-width histogram, identifies the threshold bin, folds the score→bin map into the existing FFMA chain so gating costs no extra instructions, lets idle warps update the histogram lock-free (a stale threshold is “merely looser,” conservative again), and runs exact selection only inside the threshold bin. Measured (1M context, chunk 8192, \(k=2048\), B200, inputs from GLM-5.2 activations on Wikipedia): 146.6 → 41.9 ms against the official DSA kernel (3.5×), 1.24× over the Blackwell-optimized vLLM kernel, 1.22× end-to-end prefill (Yin et al., 2026). A small ecosystem has grown around the same problem. HISA prunes the scan hierarchically. It mean-pools the scorer’s keys per block of size \(B\), keeps the top-\(m\) blocks (forcing the first and last), and re-scores tokens only inside the \(\le mB\) survivors. It is training-free, requires \(mB\ge k\), and cuts per-query scan cost from \(O(L)\) to \(O(L/B + mB)\), with up to 3.75× indexer-kernel speedup at 64K (one A100, TileLang, query length 1024, \(k=2048\), block size 128, 8K candidate pool) at near-parity on LongBench for DeepSeek-V3.2 and GLM-5, while its block-only ablation on DeepSeek-V3.2 collapses to 0.00 on mid-depth needle retrieval at 128K, a sharp warning that block granularity alone may not be enough (Xu, Meng et al., 2026). StreamIndex removes the memory cliff instead, with a chunked partition–merge top-\(k\) that never materializes the score tensor, taking the same scorer from out-of-memory at 65K (on one H200) to running at 1M with 6.21 GB peak, with set-level parity verified empirically at small lengths, minimum recall 0.998 at 16K, on synthetic inputs; “our contribution is engineering,” the authors say, and the honesty is part of why I cite it (Jaber & Jaber, 2026). IndexCache skips the work wholesale: adjacent layers share 70–100% of their selections, so let most layers reuse the previous scorer’s indices, for 1.82× prefill at 200K on a 30B DSA-style model in SGLang on an H100 node (Bai et al., 2026). Fused small-\(k\) KNN kernels exist in classical-ML tooling as well. LiteTopK reports that FlashLib’s kernel degrades sharply once \(k\) reaches the hundreds (Yang et al., 2026), while BBC brings the threshold-bucket collector to large-\(k\) CPU retrieval (Yin et al., 2026b). Different papers, one skeleton: statistic, bracket, conservative gate, refine.

4.4 The softmax-threshold lane

A parallel line asks the top-\(p\)-flavored question (how much mass am I discarding?) and answers it with a threshold anchored at the max, because under softmax, distance-from-max in logit space is relative mass: \(s_{ij} \ge m_i - \delta \iff p_{ij}/p_{i,\max} \ge e^{-\delta}\). Faster-than-Flash decoding (FFD) builds its entire kernel around this identity. A 2-bit quantized “thumbnail” of the keys is scanned to test the top-δ condition against a pseudo-max computed only from sink and local tokens (no global reduction), and blocks that pass are recomputed with an 8-bit residual to near-FP16 accuracy. Setting δ = 5 “guarantees that we discard only tokens whose contribution is less than \(e^{-5}\approx 0.67\%\) of the peak attention mass”; those are their words. Notice the quiet adaptivity. The number of survivors floats with the sharpness of each row. Measured: layer-averaged kernel speedups of 7.33×/6.18× (δ = 5/7; RTX 4090, batch 1) with a peak of 11.63×, and end-to-end decoding throughput up to 2.37× over FlashAttention-2 at 16K on an RTX 4090 (1.96× on H100); on RULER at 32K the dense Llama-3.1-8B-Instruct baseline stays near parity (e.g. variable tracking 98.4 vs. 99.6), with measured layer sparsity of 82%/73% (Liu et al., 2026). BLASST is the same idea embedded in FlashAttention’s own loop: skip a key block whenever its local max trails the running max by more than \(\ln\lambda\), with a one-parameter calibration \(\lambda = a/L\) and an explicit error bound \(\|\mathbf y-\hat{\mathbf y}\| \le |S|\,B_c\,\lambda\,V_{\max}\); 1.52× prefill at 71.9% sparsity against FlashAttention-3 BF16 on B200 (Yuan et al., 2026). SALE runs the scan in 4-bit with per-head calibrated thresholds on relative attention scores (≥3.36× prefill beyond 64K on Llama-3.1-8B, RTX 4090s) (Ji et al., 2026). Twilight wraps a base selector and prunes its conservative candidate set to the top-\(p\) core. On LongBench it removes up to 98% of the tokens over-selected by the base selector, while LLaMA-3.1-8B-Instruct stays within <1% of full attention (Lin et al., 2025). And the lane now has a compute-bound frontier. With MLA making decoding arithmetic-bound, TileSparse applies the threshold to compute tiles rather than tokens, where each KV page either is fully pruned or gets enough query heads to stay above the hardware’s roofline ridge, with an AutoTuner choosing tiered patterns. On DeepSeek-V3-0324, under a 256-token-equivalent allowance, it reports 36.4% higher average RULER accuracy than ArkVale. At 99% of full-attention accuracy, it uses 18.3% less attention work than that K-only baseline (Wang et al., 2026). The δ-thresholds here are the structural softmax mirror of the entmax bracket in §4.2; both are thresholds pegged to the max, and the difference is that entmax’s threshold is the operator itself, while softmax’s is an approximation knob with an error bound.

4.5 Three lanes, one trick, and what stays open

And what stays open, stated as open. The first window. LiteTopK’s bracket leans on the previous chunk; the first chunk, and any distribution shift between chunks, need a fallback that the paper leaves as engineering. Exactness accounting. The lanes mean different things by “exact”: bit-exact selection (LiteTopK), set parity verified empirically (StreamIndex), “nearly the same set” (HISA), bounded discarded mass (FFD, BLASST), exact-given-recall (§4.1’s superset guarantee), and the benchmarks do not overlap enough to weigh these semantics against each other. Prefill vs. decode. The lanes specialize (LiteTopK and SALE to prefill; FFD, EntmaxKV, TileSparse to decode), and the Sparse Frontier’s finding that the two phases tolerate sparsity differently (Nawrot et al., 2026) suggests the specialization is real rather than incidental. And the lanes have not met. As published, the serving kernels bracket for hard selection and carry no gradients, while the fused streaming lane (§4.2) differentiates exactly but controls only total mass, with the support size an outcome rather than a control. Each lane currently has something the others lack. Whether they stay three literatures is not for a blog post to decide; the bibliographies, at least, should start intersecting.

One table for the whole zoo, grouped by the family line of §2. Read across for what each method does; the column that matters most is vs. softmax, which records whether a method approximates a dense softmax model (a permanent error floor, §2) or computes a genuinely sparse model, at block granularity (block-native) or with individual-token exact zeros (\(\delta = 0\)). Method names link to the papers. The table scrolls sideways.

Three doses of discipline before reading the table too triumphantly. The Sparse Frontier evaluates the training-free family across a four-axis taxonomy and finds that it genuinely shifts the accuracy–efficiency frontier, larger-but-sparse beating smaller-but-dense at equal cost, while warning that safe sparsity levels are task-dependent and that “fixed-budget methods in production are suboptimal” (Nawrot et al., 2026). SCBench stress-tests the KV lifecycle, multi-turn and shared-context serving, and finds that methods holding sub-\(O(n)\) memory collapse there while \(O(n)\)-memory dynamic sparsity holds up (Li et al., 2025). And passkey-style benchmarks sit in the easiest quadrant of the long-context difficulty map, low dispersion and low scope, so headline needle numbers systematically flatter every row of the table (Goldman et al., 2024). Our own corner owes a disclosure too. The largest public entmax pretraining I found by mid-2026 is the 1-billion-parameter AdaSplash-2 run, pretrained for 50B tokens at 4K context and then extended to 32K with 10B ProLong tokens (Gonçalves et al., 2026); frontier-scale entmax pretraining has not been shown, a gap to state rather than paper over.

The live disagreements that will shape the next few years, compressed to one more table:

Table 2. The tensions, stated as camps. Most rows are genuinely open; the last is mathematics.

Read down the right columns and a single position emerges, the one this series has been arguing for. Sparsity should be a property of the model, trained in, exact where possible, and co-designed with the kernel. That is precisely the object Part I built from sparsemax and α-entmax, and §3–§4 are what it takes to train and run it. The efficiency era did not replace the principled lineage; it caught up to it.

The two posts compress to four sentences. Under bounded score spread, softmax attention must flatten at long context, a theorem, and exact zeros are the exit that changes the mathematics rather than the constants (Part I §3). Top-\(k\), top-\(p\), and sparse prediction maps do not share one objective, but they share a computational skeleton, and the sparse maps carry exact gradients where hard mask membership has none (Part I §§4–5). Among the frontier-scale systems surveyed here, the hard selectors do not use that task gradient; they detach their selectors and imitate the softmax attention they replace, for reasons they state and one open question they do not address (§3). Whatever the operator, threshold search repeatedly takes the form “bracket, then refine,” and it pays in the regimes where bytes, not FLOPs, are the scarce resource (§4).

Where the evidence runs out, I have tried to say so in place. The teacher-dispersion question has no measurement, and the lanes have no shared benchmark. Those are the edges of what is known, as best I can draw them, which was the point of writing this down.