AI 译本 · Equalang
新上架PDF · 2026年11月翻译 · 公式优化
加载中…
加载中…
arXiv · 2026
长上下文线性注意力的克罗内克 Delta 规则
Anamaria-Roberta Hartl, Levente Zólyomi, Michael List, Günter Klambauer et al.
We propose KronLSTM, a linear attention architecture based on the Kronecker Delta Rule, a novel update rule for associative matrix memories. KronLSTM extends the xLSTM family with a memory-revision mechanism that corrects targeted associations while modifying the memory only along a single direction in matrix space. This minimally invasive memory revision reduces interference with previously stored information compared with delta-rule updates, making it particularly well suited for long-context language modeling. The resulting memory dynamics can be mathematically described as a Markov chain that converges to a steady-state distribution, while the expected norm of the memory matrix remains uniformly bounded throughout the entire input sequence. Hence, KronLSTM ensures stable memory dynamics without requiring additional normalization and stabilization mechanisms for the memory state. Experiments show that KronLSTM is competitive with, and often outperforms, existing linear attention architectures, including models of the (Gated) DeltaNet family, with particularly strong performance on long context benchmarks. These results establish the Kronecker Delta Rule as an effective and analytically tractable foundation for stable associative memory in recurrent linear attention architectures. Our code (https://github.com/NX-AI/KronLSTM) and models (https://huggingface.co/collections/NX-AI/kronlstm) are open-source.
我们提出了 KronLSTM，这是一种基于克罗内克 Delta 规则的线性注意力架构。克罗内克 Delta 规则是一种用于联想矩阵记忆的新型更新规则。KronLSTM 将一种记忆修订机制引入 xLSTM 系列：该机制能够校正特定关联，同时仅沿矩阵空间中的单一方向修改记忆。与 Delta 规则更新相比，这种对记忆的最小化干预能减少对先前存储信息的干扰，因此特别适用于长上下文语言建模。其记忆动态可用一个收敛至稳态分布的马尔可夫链进行数学描述，同时，在整个输入序列中，记忆矩阵范数的期望值始终有界。因此，KronLSTM 无需额外的归一化和稳定化机制来处理记忆状态，也能确保记忆动态保持稳定。实验表明，KronLSTM 的性能可与现有线性注意力架构媲美，且往往优于这些架构，包括（门控）DeltaNet 系列模型；它在长上下文基准测试中表现尤为出色。这些结果表明，克罗内克 Delta 规则可作为循环线性注意力架构中稳定联想记忆的一种有效且易于解析的基础。我们的代码（https://github.com/NX-AI/KronLSTM）和模型（https://huggingface.co/collections/NX-AI/kronlstm）均已开源。
PDF · 2026年11月翻译 · 公式优化