Blog
Fragments of code, concepts, and curiosity.
Why Sparse, Why Adaptive, and the Classical Problem Behind Thresholds
A math-first essay on what is actually known about sparse attention. Softmax and entmax derived from one regularized family, with every derivation and proof a click away; softmax must disperse at long context (a theorem, proved in the post); a fixed top-k slices a continuum the trained models don't mark; and every thresholding method in use today, from top-k and top-p to entmax, reduces to a classical resource-allocation problem that optimization solved decades ago.
How Sparse Attention Runs Fast, and Why the Frontier Labs Don’t Differentiate Through It
The systems half. What DeepSeek-V3.2 and MiniMax say, in their own words, about training a selector they detach from the graph; why unstructured sparsity is fast on modern GPUs, where compute has outgrown memory; how three communities converged on bracketing the threshold with coarse statistics; and AdaSplash-2 up close, from SRAM histograms to the bit-packed block masks that skip provably-zero work, with pointers into the code.
SLURM in the Wild: A Practical Guide for Academic Labs
A complete guide from basic concepts to production deployment — covering multi-node setup, GPU scheduling, advanced monitoring, and hard-learned lessons from scaling a research lab from 2 to 30+ users across heterogeneous hardware.