Nous Research's Lighthouse Attention delivers 1.4–1.7x pretraining speedup at long context

Lighthouse Attention is a training-only, selection-based hierarchical attention that wraps scaled dot-product attention during pretraining and is removed afterward. Unlike NSA/HISA which pool only K/V, Lighthouse pools Q, K, and V symmetrically across a multi-resolution pyramid.