Rethinking Key-Value Relationships in Linear Attention重新审视线性注意力中的 key-value 关系

I. Background: Linear Attention as Test-Time Regression一、背景：把线性注意力看成测试时回归

Linear attention mechanisms maintain a state matrix $S_t \in \mathbb{R}^{d \times d}$ that evolves recurrently. Three update rules represent the evolution of this paradigm:线性注意力维护一个按递归方式演化的状态矩阵 $S_t \in \mathbb{R}^{d \times d}$。下面三条更新规则代表了这一范式的演进：

Recent work frames these as online regression: each model tries to minimize $\sum_{i=1}^{t} \|v_i - S_t k_i\|^2$. DeltaNet's advantage comes from the projection term $(I - k_t k_t^T)$, which removes outdated information before updating.近期工作把它们统一看作在线回归：每个模型都在最小化 $\sum_{i=1}^{t} \|v_i - S_t k_i\|^2$。DeltaNet 的优势来自投影项 $(I - k_t k_t^T)$，它在更新前先把过时的信息清掉。

Our question: How do these models perform when the key-value relationship varies? Can DeltaNet handle nonlinear relationships like $v = k^4$ or $v = \sin(10\pi k)$?我们的问题是：当 key-value 之间的关系改变时，这些模型表现如何？DeltaNet 能不能处理 $v = k^4$ 或 $v = \sin(10\pi k)$ 这类非线性关系？

II. Experimental Setup二、实验设置

Configuration配置

Most experiments used 1000-token sequences with 64-dimensional keys, queries, and values—matching standard transformer configurations. For scaling analysis, we tested up to 32,000 tokens. All models operated in single-head mode to isolate the core mechanism behavior.多数实验用 1000 token 的序列，key、query、value 都是 64 维——与标准 transformer 的配置一致。做 scaling 分析时最长测到 32,000 token。所有模型都跑单头模式，以便隔离出核心机制本身的行为。

Test Scenarios测试场景

We generated synthetic key-value pairs with controlled relationships:我们生成了关系可控的合成 key-value 对：

• Identity: $v = k$ (perfect linear)恒等：$v = k$（完全线性）
• Quadratic: $v = k \odot k$ (element-wise square)二次：$v = k \odot k$（逐元素平方）
• Cubic: $v = k^3$ (higher-degree polynomial)三次：$v = k^3$（更高次多项式）
• Quartic: $v = k^4$ (very high degree)四次：$v = k^4$（次数很高）
• Exponential: $v = \exp(k)$ (unbounded growth)指数：$v = \exp(k)$（无界增长）
• Low-frequency sine: $v = \sin(\pi k)$低频正弦：$v = \sin(\pi k)$
• High-frequency sine: $v = \sin(10\pi k)$ (rapid oscillations)高频正弦：$v = \sin(10\pi k)$（快速振荡）

A Note on Normalization关于归一化的一点说明

Before diving into results, it's important to clarify when normalization happens in practice. In real transformer architectures: input embeddings are projected through learned weight matrices $W_q$, $W_k$, and $W_v$, then normalization is applied before the attention computation. This means normalization operates on $q = \text{norm}(xW_q)$ and similar for keys and values.在看结果之前，先说清楚实际中归一化发生在什么位置。在真实的 transformer 架构里：输入 embedding 先经过学到的权重矩阵 $W_q$、$W_k$、$W_v$ 投影，然后在注意力计算之前做归一化。也就是说归一化作用在 $q = \text{norm}(xW_q)$ 上，key 和 value 同理。

This detail matters because the nonlinearity we're testing (like $v = k^2$) happens conceptually before projection, and normalization smooths the results afterward. We tested both scenarios: applying normalization after the nonlinearity (matching real architectures) and skipping it entirely to see the raw effect.这个细节很重要，因为我们要测的非线性（比如 $v = k^2$）在概念上发生在投影之前，而归一化会在之后把结果抹平。两种情形我们都测了：在非线性之后做归一化（对应真实架构），以及完全不做归一化，看原始效果。

III. The Normalization Effect: Stability vs. Information三、归一化效应：稳定性与信息量的取舍

We first tested all relationships under standard normalized conditions. The results were strikingly uniform.我们先在标准归一化条件下测了所有关系。结果一致得惊人。

With normalization enabled, DeltaNet achieves MSE between 0.028 and 0.030 across every relationship we tested. No systematic degradation appears as mathematical nonlinearity increases. This was unexpected. Standard intuition suggests that $k^4$ or $e^k$ should challenge linear approximation more than simple identity, but the measurements show otherwise.开启归一化后，DeltaNet 在我们测的每一种关系上 MSE 都落在 0.028 到 0.030 之间。随着数学上的非线性增强，并没有出现系统性的退化。这出乎意料：按常规直觉，$k^4$ 或 $e^k$ 应该比简单的恒等关系更难被线性逼近，但测量结果并非如此。

The explanation lies in what normalization removes. By forcing all value vectors to unit length ($\|v\| = 1$), normalization strips away magnitude information and leaves only directional relationships. Whether values come from $k^2$ or $e^k$, after normalization they're all unit vectors positioned at various angles from the keys. The state matrix learns this angular mapping, which proves remarkably consistent across different generating functions.原因在于归一化拿走了什么。把所有 value 向量强行拉到单位长度（$\|v\| = 1$）之后，幅值信息被剥掉了，只剩下方向关系。不管 value 来自 $k^2$ 还是 $e^k$，归一化之后都是与 key 成某个夹角的单位向量。状态矩阵学的是这个角度映射，而它在不同生成函数之间出奇地一致。

This uniformity has practical benefits for training stability, but it also means we lose information about the original relationship structure. To understand what normalization hides, we needed to test without it.这种一致性对训练稳定性很有好处，但也意味着我们丢掉了原始关系结构的信息。要弄清归一化藏起了什么，就得把它去掉再测一遍。

IV. Removing Normalization: The Polynomial Paradox四、去掉归一化：多项式悖论

Without normalization, the landscape changes completely:不做归一化，整个图景完全变了：

The Polynomial Mystery: Why Easier Without Normalization?多项式之谜：为什么不归一化反而更容易？

The most surprising finding deserves deeper examination. Intuitively, we'd expect higher-degree polynomials to be harder for linear models. Yet our measurements show the opposite: quadratic achieves 30× lower error than normalized, cubic 100×, and quartic 3000×. What's happening?最意外的这个发现值得细看。按直觉，次数越高的多项式对线性模型应该越难。但测量结果恰恰相反：二次的误差比归一化时低 30×，三次低 100×，四次低 3000×。这是怎么回事？

The answer lies in magnitude shrinkage. When keys are normalized to unit vectors, most elements fall in the range $[-0.5, 0.5]$ due to high-dimensional geometry. Squaring these values doesn't amplify them—it crushes them toward zero. A key element $k_i = 0.5$ raised to successive powers produces $k^2 = 0.25$, $k^3 = 0.125$, $k^4 = 0.0625$.答案在于幅值收缩。key 被归一化成单位向量后，受高维几何的影响，大多数元素落在 $[-0.5, 0.5]$ 区间里。对这些数取平方不会放大它们——而是把它们压向 0。一个 $k_i = 0.5$ 的元素，逐次取幂得到 $k^2 = 0.25$、$k^3 = 0.125$、$k^4 = 0.0625$。

We measured this effect across 1000 random normalized keys. Identity vectors have mean magnitude 1.0 and variance 0.0156. Quadratic drops to magnitude 0.21 with variance 0.00046. Cubic plummets to magnitude 0.056 and variance 0.000051. By quartic, we're at magnitude 0.017 with variance 0.000005—the vectors are nearly constant.我们在 1000 个随机归一化 key 上测了这个效应。恒等向量的平均幅值为 1.0，方差 0.0156。二次降到幅值 0.21、方差 0.00046。三次直坠到幅值 0.056、方差 0.000051。到四次时，幅值只剩 0.017、方差 0.000005——向量几乎成了常数。

For linear regression, this is a dream scenario. When the true relationship is $k^2$ or $k^3$, magnitude shrinkage compresses values so tightly near zero that a nearly-flat straight line captures most of the signal. The mathematical nonlinearity remains, but the signal variance that regression must capture becomes trivial.对线性回归来说，这是理想局面。真实关系是 $k^2$ 或 $k^3$ 时，幅值收缩把取值压得离 0 极近，一条几乎水平的直线就能抓住大部分信号。数学上的非线性仍然在，但回归需要拟合的信号方差已经小到可以忽略。

Contrast this with exponential and sine functions. Exponentials cause magnitude explosion: $e^k$ creates large dynamic range with MSE jumping to 1.92. High-frequency sine ($\sin(10\pi k)$) oscillates rapidly, producing MSE of 0.96. These functions maintain or amplify variance, giving linear regression real difficulty.指数和正弦则相反。指数会造成幅值爆炸：$e^k$ 带来很大的动态范围，MSE 跳到 1.92。高频正弦（$\sin(10\pi k)$）快速振荡，MSE 为 0.96。这两类函数会保持甚至放大方差，给线性回归带来真正的困难。

V. Normalization as Architectural Choice五、把归一化当成架构选择

Stepping back, these experiments reveal normalization as a fundamental design decision rather than mere numerical convenience. The trade-off is clear: stability and consistency versus information preservation.退一步看，这些实验说明归一化是一个根本性的设计决策，而不只是数值上的便利。取舍很清楚：稳定性与一致性，换取的是信息的丢失。

With normalization, we get uniform behavior across diverse relationships. This consistency makes training stable and ensures predictable behavior across different data distributions. Without normalization, we confront the raw characteristics of each relationship. Exponentials blow up to MSE above 1.9; polynomials collapse toward zero through magnitude shrinkage.有归一化时，各种关系的行为高度一致。这种一致性让训练更稳，也让不同数据分布下的行为可预测。没有归一化时，我们直面每种关系的原始特性：指数关系 MSE 冲到 1.9 以上，多项式则因幅值收缩塌向 0。

Modern transformers universally adopt normalization precisely because the stability benefits outweigh the information loss for their use cases. Our measurements quantify both sides of this trade-off explicitly.现代 transformer 普遍采用归一化，正是因为在它们的使用场景下，稳定性的收益压过了信息的损失。我们的测量把这笔账的两边都量化了出来。

VI. Optimal Linear Approximation: DeltaNet vs RLS六、最优线性近似：DeltaNet 对比 RLS

Recursive Least Squares (RLS) represents the theoretical optimum for linear approximation—it explicitly maintains $(K^T K)^{-1}$ to compute the globally best linear fit at every step. This comes at $O(td^2 + d^3)$ computational cost compared to DeltaNet's $O(d^2)$. Does the extra computation pay off?递归最小二乘（RLS）代表线性近似的理论最优——它显式维护 $(K^T K)^{-1}$，在每一步都算出全局最优的线性拟合。代价是 $O(td^2 + d^3)$ 的计算量，而 DeltaNet 只要 $O(d^2)$。多出来的计算值得吗？

Finding: Under normalization, DeltaNet and RLS perform nearly identically (gap < 5%). The expensive $O(d^3)$ matrix inversion in RLS provides minimal benefit when values are normalized.结论：在归一化条件下，DeltaNet 与 RLS 表现几乎相同（差距小于 5%）。value 归一化之后，RLS 那个昂贵的 $O(d^3)$ 矩阵求逆几乎没带来收益。

VII. What We Didn't Find七、我们没有找到的东西

We started expecting to identify fundamental limitations of DeltaNet when confronted with extreme nonlinearities. Surely quartic polynomials, exponentials, or high-frequency periodic functions would expose breaking points.一开始我们以为，在极端非线性面前能找出 DeltaNet 的根本局限。四次多项式、指数函数、高频周期函数，总该暴露出某个崩溃点吧。

Under standard normalized conditions, no such breaking points emerged. DeltaNet maintained MSE around 0.030 across every relationship tested. The anticipated "nonlinear ceiling" simply didn't materialize in realistic settings.在标准归一化条件下，这样的崩溃点并没有出现。DeltaNet 在所有被测关系上都维持在 0.030 左右。我们预期的那道「非线性天花板」，在现实设定下根本没有出现。

This outcome is genuine insight into normalization's power. By projecting all relationships onto the unit sphere, normalization makes them equally tractable to linear approximation from a directional perspective. The wild diversity observed without normalization gets compressed into uniform angular variation that linear models handle consistently.这个结果本身就是对归一化威力的真实认识。把所有关系投影到单位球面上之后，从方向的角度看，它们对线性近似的难度是一样的。不归一化时观察到的巨大差异，被压缩成了线性模型能够稳定处理的、均匀的角度变化。

VIII. Multi-Query Retrieval Capacity八、多查询检索容量

After processing 1000 tokens to build the final state $S_{\text{final}}$, we randomly selected past keys and measured reconstruction accuracy for their associated values. The test scaled from single-pair retrieval up to 50 simultaneous queries.在处理完 1000 个 token 得到最终状态 $S_{\text{final}}$ 之后，我们随机挑出过去的 key，测量对应 value 的重建精度。测试从单对检索一直做到同时 50 个查询。

For identity relationships ($v = k$), both DeltaNet and RLS achieve remarkable results. Retrieval error stays below $10^{-6}$ regardless of how many pairs are queried simultaneously. The state becomes a perfect compressed representation.在恒等关系（$v = k$）下，DeltaNet 和 RLS 的结果都非常好。无论同时查询多少对，检索误差都在 $10^{-6}$ 以下。此时状态矩阵就是一份完美的压缩表示。

Quadratic relationships ($v = k^2$) show graceful degradation. DeltaNet maintains MSE around 0.001 whether retrieving 1 pair or 50. Exponential relationships reveal real capacity limits—single-pair retrieval starts at MSE 0.004, but jumps to 2.9 when querying 10 historical pairs simultaneously.二次关系（$v = k^2$）退化得很平缓：不管检索 1 对还是 50 对，DeltaNet 的 MSE 都在 0.001 附近。指数关系则暴露出真正的容量上限——单对检索的 MSE 是 0.004，同时查询 10 对历史记录时跳到 2.9。

IX. Short Convolution as Nonlinearity Regulator九、短卷积作为非线性调节器

The exponential relationship's poor performance suggested exploring architectural modifications. We tested 1D convolution with kernel sizes 3, 5, and 7, applied to both keys and values before DeltaNet processing.指数关系上的糟糕表现提示我们去试试架构层面的改动。我们在 DeltaNet 处理之前，对 key 和 value 都加了一维卷积，核大小分别取 3、5、7。

Short convolution acts as a nonlinearity regulator—smoothing extreme relationships while preserving structure for well-behaved ones.短卷积起到了非线性调节器的作用——把极端关系抹平，同时保留温和关系原有的结构。

X. Scaling Analysis: From 500 to 32,000 Tokens十、规模分析：从 500 到 32,000 token

The divergence is stark. Vanilla linear attention starts reasonably at 500 tokens but degrades exponentially. By 32,000 tokens, it completely breaks down with MSE exceeding 1300. DeltaNet and RLS maintain MSE around 0.031 regardless of sequence length. The delta rule's projection term $(I - k_t k_t^T)$ actively removes error accumulation before each update.差距非常明显。原始线性注意力在 500 token 时还算正常，之后呈指数退化，到 32,000 token 时彻底崩掉，MSE 超过 1300。DeltaNet 和 RLS 则不管序列多长都维持在 0.031 左右。delta 规则的投影项 $(I - k_t k_t^T)$ 会在每次更新前主动清掉累积的误差。

XI. Information-Theoretic Analysis十一、信息论分析

Linear attention compresses a growing history $K_t, V_t$ (size $2td$) into a fixed state $S_t$ (size $d^2$). For our setup (t=1000, d=64): original 128,000 scalars compressed to 4,096 — a 3.2% compression ratio (96.8% reduction). We quantify information loss using mutual information.线性注意力把不断增长的历史 $K_t, V_t$（规模 $2td$）压缩进一个固定大小的状态 $S_t$（规模 $d^2$）。在我们的设定下（t=1000，d=64）：原本 128,000 个标量被压成 4,096 个 — 压缩比 3.2%（减少 96.8%）。我们用互信息来量化信息损失。

XII. Conclusion十二、结论

Testing linear attention mechanisms across diverse key-value relationships revealed patterns that weren't obvious from theory alone. Normalization emerged not as a numerical convenience but as fundamentally reshaping what these models learn. By converting all vectors to unit length, we shift from tracking magnitude and direction to purely angular relationships. This trade-off brings remarkable stability—DeltaNet maintains consistent performance whether values come from $k^2$ or $e^k$—but at the cost of losing information about scale.把线性注意力放在各种 key-value 关系上测一遍，暴露出一些单靠理论看不出来的规律。归一化并不是一个数值上的便利手段，而是从根本上改变了这些模型学到的东西。把所有向量拉成单位长度之后，我们从「同时跟踪幅值和方向」变成了「只跟踪角度」。这笔交易换来了出色的稳定性——不管 value 来自 $k^2$ 还是 $e^k$，DeltaNet 的表现都一致——代价是丢掉了关于尺度的信息。

The polynomial mystery taught us to look beyond mathematical nonlinearity to actual signal characteristics. Higher-degree polynomials paradoxically become easier because they shrink variance toward zero. When quartic values cluster with variance 0.000005, almost any linear function fits well.多项式之谜教会我们：不要只盯着数学上的非线性，要看信号本身的特性。高次多项式反而更容易，是因为它把方差压向了 0。当四次关系的取值聚成方差 0.000005 的一团时，几乎任何线性函数都能拟合得不错。

Scaling tests extending to 32,000 tokens exposed the fundamental instability of vanilla linear attention. MSE explodes from 0.4 to 1306 as correlations accumulate. DeltaNet's delta rule projection fixes this by explicitly removing error before each update, resulting in flat scaling: 0.031 MSE whether processing 500 or 32,000 tokens.把规模推到 32,000 token 的测试，暴露了原始线性注意力的根本不稳定：随着相关性累积，MSE 从 0.4 爆到 1306。DeltaNet 的 delta 规则投影在每次更新前显式清除误差，因此扩展曲线是平的：不管处理 500 还是 32,000 个 token，MSE 都是 0.031。

Information-theoretic analysis provided the clearest framework for understanding capacity limits. DeltaNet retains 99.6% of mutual information for identity relationships—nearly perfect compression. For exponentials, only 1.2% survives. Linear models can't losslessly compress nonlinear relationships, and some relationships resist compression far more than others.信息论分析提供了理解容量上限最清晰的框架。DeltaNet 在恒等关系下保留 99.6% 的互信息——几乎是无损压缩；在指数关系下只剩 1.2%。线性模型无法无损地压缩非线性关系，而且有些关系对压缩的抵抗远强于另一些。

Short convolution provided an unexpected path forward for specific failure cases. Adding kernel-size-7 convolution to exponential relationships reduced MSE 64-fold (1.92 → 0.03). Local smoothing helps tame magnitude explosions. Interestingly, the same convolution slightly hurt polynomials, suggesting the solution is relationship-specific rather than universally beneficial.对某些具体的失败情形，短卷积给出了一条意外的出路。给指数关系加上核大小为 7 的卷积，MSE 降低了 64 倍（1.92 → 0.03）。局部平滑有助于压住幅值爆炸。有意思的是，同样的卷积对多项式略有害处，说明这个方案是「看关系下菜」的，而不是普遍有益。

For practitioners deploying these systems: normalization isn't optional—it provides the stability that makes diverse data tractable. DeltaNet's delta rule offers genuine advantages over vanilla linear attention, especially at longer contexts. For researchers: testing under realistic conditions (normalization, appropriate sequence lengths) matters more than theoretical worst-case analysis.对要把这些系统部署上线的人：归一化不是可选项——正是它带来的稳定性，让多样的数据变得可处理。DeltaNet 的 delta 规则相对原始线性注意力有实实在在的优势，长上下文下尤其明显。对研究者：在现实条件下测试（带归一化、用合适的序列长度），比理论上的最坏情况分析更重要。

@misc{deng2026linearattention,
  author       = {Chunyuan Deng},
  title        = {Rethinking Key-Value Relationships in Linear Attention},
  year         = {2026},
  url          = {https://charlesdddd.github.io/blog/linear-attention-key-value.html}
}