A probe is a measurement attached to a training run. A large run emits tens of thousands of them, and the hard part is knowing which ten to watch. So every metric here carries a priority as well as a definition, an aggregation rule, and a default cadence.探针（probe）是挂在训练上的一次测量。一次大规模训练会吐出成千上万个探针，难的从来不是多采几个，而是知道该盯住哪十个。所以这里的每个指标除了定义、聚合方式和默认采样频率，还带一个优先级。
| Level级别 | Meaning含义 | Typical cadence典型频率 |
|---|---|---|
| P0 | Alert. Wired to a pager. Roughly ten metrics; if one moves, the run stops or someone wakes up.报警。直接接到 on-call。大约十个指标；其中任何一个动了，要么停训练，要么把人叫起来。 | Every step每步 |
| P1 | Diagnose. Opened when a P0 moves. Usually per-layer or per-tensor. Nobody watches these continuously.诊断。P0 动了才打开。通常是按层、按张量的视图，没人需要一直盯着。 | Every 100 steps每 100 步 |
| P2 | Forensics. Per-parameter, per-expert, per-domain detail, for one investigation.取证。按参数、按 expert、按数据域的细节，只为某一次具体排查而开。 | On demand按需 |
1. First screen第一屏
If you build one page, build this one: nine P0 signals, all per-step except the last.如果只做一个看板页，就做这一页：九个 P0 信号，除最后一个外都是每步采。
Convergence, and whether one worker or shard carries the damage.收敛情况，以及是不是某个 worker 或数据分片在单独扛雷。
A domain or modality regressing under a flat global curve.全局曲线看着平稳时，某个数据域或模态已经在退化。
The earliest numerical warning; it moves before the loss.最早的数值告警，通常比 loss 先动。
Loss and gradient spiking on the same step.loss 和梯度在同一步一起尖刺。
Stalls, replays, stragglers, communication regressions.卡顿、重放、慢节点、通信退化。
Headroom before the first out-of-memory failure.距离第一张卡 OOM 还剩多少余量。
Routing collapse and the compute imbalance behind it.路由塌缩，以及它带来的算力不均。
Overflow and corruption, before they reach the loss.溢出和数值损坏，在它们传到 loss 之前。
Whether compute is still buying quality. Every 0.5–2%.算力是否还在换来质量。每 0.5–2% 进度跑一次。
Escalation order. Job state → loss → gradient norm and spikes → step time → memory → expert balance. Open the layer-level views only after that ordering fails to explain the symptom.排查顺序。任务状态 → loss → 梯度范数与尖峰 → 单步耗时 → 显存 → 专家均衡。只有这条线索解释不了现象时，才去打开按层的视图。
2. Metric index指标总表
The default monitoring surface. Definitions and aggregation rules follow in the sections below.默认的监控面。定义与聚合规则见后面各节。
| Metric指标 | What it catches能发现什么 | Cadence频率 | Priority优先级 |
|---|---|---|---|
| Loss损失 | |||
| Global mean $L$全局平均损失 $L$ | Convergence; the number every comparison rests on收敛；所有对比都建立在这个数上 | Every step每步 | P0 |
| Worst-rank mean $L_{\max}$最差 rank 均值 $L_{\max}$ | A bad worker or data shard hidden inside $L$被 $L$ 平均掉的坏 worker 或坏数据分片 | Every step每步 | P0 |
| Loss by source and modality按数据源与模态的损失 | A domain or modality regressing under a flat global curve全局曲线平稳时，正在退化的数据域或模态 | Every 1–10 steps每 1–10 步 | P0 |
| Gradients and stability梯度与稳定性 | |||
| Global norm $G_2$全局梯度范数 $G_2$ | Numerical stability; moves before the loss数值稳定性；通常比 loss 先动 | Every step每步 | P0 |
| Clip rate裁剪率 | A clip threshold that no longer matches the run裁剪阈值已经跟不上这次训练 | Every step每步 | P0 |
| Co-spike count共同尖峰计数 | Loss and gradient spiking together: real instabilityloss 与梯度同时尖刺：真正的不稳定 | Every step每步 | P0 |
| Robust spike score $z_t$稳健尖峰分数 $z_t$ | Spikes, without assuming a Gaussian不假设高斯分布也能抓到尖峰 | Every step每步 | P1 |
| Mean and max $|g|$$|g|$ 的均值与最大值 | Typical scale versus one exploding element典型量级，对比单个元素的爆炸 | Every step每步 | P1 |
| Zero fraction $Z_g$零元素比例 $Z_g$ | Dead paths, underflow, parameters getting no gradient死路径、下溢、拿不到梯度的参数 | Every step每步 | P1 |
| Skip and replay counters跳过与重放计数 | Steps dropped by the spike guard; nondeterministic replays被尖峰保护丢掉的步；不可复现的重放 | Every step每步 | P1 |
| System系统 | |||
| Step time, throughput单步耗时、吞吐 | Stalls, replays, slow shards卡顿、重放、慢分片 | Every step每步 | P0 |
| Peak memory, max over ranks显存峰值，跨 rank 取最大 | Headroom before the first out-of-memory failure距离第一张卡 OOM 还剩多少余量 | Every step每步 | P0 |
| Compute rate $\Phi$算力速率 $\Phi$ | Hardware efficiency against a fixed FLOP estimator固定 FLOP 估计口径下的硬件效率 | Every step每步 | P0 |
| Straggler score $S$慢节点分数 $S$ | One rank setting the pace for the job整个任务被某一个 rank 拖住 | Every step每步 | P1 |
| Exposed collective wait暴露的集合通信等待 | Communication that computation is not hiding没有被计算掩盖掉的通信 | Every 10 steps每 10 步 | P1 |
| Device and host telemetry设备与主机遥测 | Thermal, allocator, and network causes温度、显存分配器、网络层面的原因 | Every step每步 | P1 |
| Tensors and optimizer张量与优化器 | |||
| Absolute maximum绝对值最大 | Overflow, non-finite values, silent corruption溢出、非有限值、静默损坏 | Every 100 steps每 100 步 | P0 |
| RMS and absolute meanRMS 与绝对值均值 | Scale drift in activations, gradients, updates激活、梯度、更新量的尺度漂移 | Every 100 steps每 100 步 | P1 |
| Zero fraction零元素比例 | Sparsity, underflow, inactive paths稀疏、下溢、失活路径 | Every 100 steps每 100 步 | P1 |
| Update ratio $\rho$更新比例 $\rho$ | Updates too small to matter, or large enough to damage更新小到没用，或大到有害 | Every 100 steps每 100 步 | P1 |
| Excess kurtosis超额峰度 | Heavy tails; low-precision risk重尾；低精度风险 | Every 100 steps每 100 步 | P2 |
| Optimizer chain, per parameter按参数的优化器链路 | Which stage introduced a scale or direction change是哪一级改变了尺度或方向 | On demand按需 | P2 |
| Experts专家 | |||
| MaxVio, imbalance ratio $I$MaxVio 与不均衡比 $I$ | Routing collapse and wasted expert compute路由塌缩，以及被浪费的 expert 算力 | Every step每步 | P0 |
| Shard imbalance分片不均衡 | A hot expert-parallel group throttling the step某个 EP 组过热，拖慢整步 | Every step每步 | P0 |
| Normalized entropy $H_{\mathrm{norm}}$归一化熵 $H_{\mathrm{norm}}$ | Routing sharpening onto a few experts路由向少数几个 expert 收紧 | Every 100 steps每 100 步 | P1 |
| Per-expert load, router scores单 expert 负载与路由分数 | Which expert, and by how much具体是哪个 expert，偏了多少 | On demand按需 | P2 |
| Residual paths残差路径 | |||
| Relative branch scale $R_b$分支相对尺度 $R_b$ | A branch too weak to matter, or one that dominates分支弱到无关紧要，或强到压过主干 | Every 100 steps每 100 步 | P0 |
| Block importance $B$, angle $A$块重要性 $B$ 与夹角 $A$ | Layers that stopped changing the stream已经不再改变残差流的层 | Every 100 steps每 100 步 | P1 |
| Bfloat16 no-op fractionbfloat16 空更新比例 | Residual updates lost to rounding被舍入吃掉的残差更新 | Every 100 steps每 100 步 | P1 |
| Gate saturation fraction门控饱和比例 | Gated branches closing permanently门控分支被永久关死 | Every 100 steps每 100 步 | P1 |
| Modality模态 | |||
| Token share $\pi^{(\mu)}$模态 token 占比 $\pi^{(\mu)}$ | Data-mix and packing changes数据配比与打包方式的变化 | Every 1–10 steps每 1–10 步 | P0 |
| Media scale ratio $\gamma$媒体特征尺度比 $\gamma$ | Media embeddings the backbone ignores, or that swamp it主干会忽略、或反过来淹没主干的媒体 embedding | Every 100 steps每 100 步 | P0 |
| Encoder gradient share $S_{\mathrm{enc}}$编码器梯度占比 $S_{\mathrm{enc}}$ | An encoder that stopped learning已经不再学习的编码器 | Every 100 steps每 100 步 | P1 |
| Evaluation评测 | |||
| Held-out NLL, bits per byte留出集 NLL 与 bits per byte | Generalization and the scaling trend泛化能力与 scaling 趋势 | Every 0.5–2%每 0.5–2% 进度 | P0 |
| Choice accuracy选择题准确率 | Capability, on a frozen harness固定评测框架下的能力 | Key checkpoints关键 checkpoint | P0 |
| Greedy generation贪心生成 | Capability that only free generation exposes只有自由生成才暴露出来的能力 | Key checkpoints关键 checkpoint | P0 |
| Media gain媒体增益 | Whether media inputs carry information at all媒体输入到底有没有带来信息 | Key checkpoints关键 checkpoint | P0 |
| Sampled generation, IoU采样生成与 IoU | Coverage under sampling; grounding quality采样下的覆盖率；grounding 质量 | Final checkpoints最终 checkpoint | P1 |
| Averaged-weight evaluation权重平均评测 | Mid-run noise removed from the comparison去掉训练中途的噪声再做对比 | Every evaluation每次评测 | P1 |
Cadence rule. Cheap alerting metrics run every step, tensor scans every 100 steps, evaluation every 0.5–2% of training. An anomaly triggers denser probes or a replay.频率原则。便宜的报警指标每步都算，张量扫描每 100 步，评测每 0.5–2% 训练进度。一旦出现异常，就临时加密探针或重放这一步。
Comparison axis. If batch size changes, compare runs by cumulative tokens or estimated floating-point operations, never by step.对比横轴。batch size 一旦变过，两次训练就要按累计 token 数或估计的浮点运算量来比，绝不能按 step 比。
3. Loss损失
Let $R$ be the number of GPU ranks — one training process each. Rank $r$ contains $T_r$ token positions. For token $t$ on rank $r$, $\ell_{r,t}$ is its negative log-likelihood and $m_{r,t}$ is 1 for a valid target and 0 otherwise.设 $R$ 为 GPU rank 数，每个 rank 是一个训练进程。rank $r$ 上有 $T_r$ 个 token 位置。对 rank $r$ 上的 token $t$，$\ell_{r,t}$ 是它的负对数似然，$m_{r,t}$ 在它是有效目标时取 1，否则取 0。
Global mean loss $L$.全局平均损失 $L$。P0 Sum valid-token loss and valid-token count over all ranks, then divide.先在所有 rank 上分别求有效 token 的 loss 之和与 token 计数，再相除。
Use $L$ for convergence and scaling comparisons.收敛判断和 scaling 对比都用 $L$。
Local mean loss $L_r$.单 rank 平均损失 $L_r$。P1 The valid-token mean on rank $r$.rank $r$ 上有效 token 的平均。
Omit a rank with no valid target; do not record it as zero.没有有效目标的 rank 直接跳过，不要记成 0。
Worst-rank loss $L_{\max}$.最差 rank 损失 $L_{\max}$。P0 The maximum local mean over ranks.在所有 rank 的局部均值里取最大。
This is a maximum over GPU ranks, not over tokens or examples. It catches a bad worker or an unusual shard that $L$ averages away.这是对 GPU rank 取最大，不是对 token 或样本取最大。它能抓到被 $L$ 平均掉的坏 worker 或异常数据分片。
Loss by source and modality.按数据源与模态的损失。P0 Let $\mu$ index a stable data source or modality, and $m^{(\mu)}_{r,t}$ be 1 when token $t$ on rank $r$ is a valid target belonging to $\mu$. Report the group loss with its token share.设 $\mu$ 表示一个稳定的数据源或模态，当 rank $r$ 上的 token $t$ 是属于 $\mu$ 的有效目标时 $m^{(\mu)}_{r,t}$ 取 1。报这一组的 loss 时要同时报它的 token 占比。
A group holding a few percent of the tokens can regress for thousands of steps without moving $L$. Compare each group against its own history: absolute levels are not comparable across sources, whose token entropies differ.只占几个百分点 token 的一组，可以连续退化几千步而完全不影响 $L$。每一组只跟自己的历史比：不同数据源的 token 熵不同，绝对数值之间没有可比性。
Aggregation: do not average $L_r$ when valid-token counts differ. Reduce the loss sum and the token count separately, then divide.聚合：各 rank 有效 token 数不同的时候，不要直接平均 $L_r$。要把 loss 之和与 token 计数分别 reduce，再相除。
4. Gradients and spikes梯度与尖峰
Let $g=(g_1,\ldots,g_P)$ contain all $P$ gradient elements. Measure after gradient accumulation and loss-scale removal, but before global clipping.设 $g=(g_1,\ldots,g_P)$ 包含全部 $P$ 个梯度元素。测量点在梯度累积完成、去掉 loss scale 之后，但在全局裁剪之前。
Global gradient norm $G_2$.全局梯度范数 $G_2$。P0 The Euclidean norm over all gradient elements.所有梯度元素上的欧氏范数。
The primary stability signal, and usually the first thing to move in a failing run.最主要的稳定性信号，一次要崩的训练里通常是它先动。
Clip rate.裁剪率。P0 With clip threshold $c$ over a window of $W$ recent steps:取裁剪阈值 $c$，在最近 $W$ 步的窗口上：
A low steady rate is normal. A sustained rise means clipping has become an unplanned learning-rate schedule.维持在一个低值是正常的。持续上升则说明裁剪已经变成了一条你没有设计过的学习率调度。
Mean absolute gradient $G_{\mathrm{mean}}$.梯度绝对值均值 $G_{\mathrm{mean}}$。P1 The average absolute gradient element.所有梯度元素绝对值的平均。
Tracks typical element scale, and is less sensitive to one outlier than $G_{\max}$.反映典型的元素量级，比 $G_{\max}$ 更不容易被单个离群值带偏。
Maximum absolute gradient $G_{\max}$.梯度绝对值最大 $G_{\max}$。P1 The largest absolute gradient element.绝对值最大的那个梯度元素。
Detects localized explosion that $G_{\mathrm{mean}}$ dilutes away.能抓到被 $G_{\mathrm{mean}}$ 稀释掉的局部爆炸。
Gradient zero fraction $Z_g$.梯度零元素比例 $Z_g$。P1 Exact zeros divided by the element count.严格等于 0 的元素数除以元素总数。
The indicator $\mathbf{1}[\mathcal{C}]$ is 1 when $\mathcal{C}$ holds and 0 otherwise. A rise means sparse paths, underflow, or parameters receiving no gradient. Report embeddings and expert parameters separately, where sparsity is expected and a change in it still matters.示性函数 $\mathbf{1}[\mathcal{C}]$ 在条件 $\mathcal{C}$ 成立时为 1，否则为 0。它上升意味着稀疏路径、下溢，或者有参数完全拿不到梯度。embedding 和 expert 参数要单独报：这两处本来就该稀疏，但它的变化仍然有意义。
Timing: record whether each gradient metric is scaled or unscaled, accumulated or per-microbatch, and before or after clipping. Norms measured at different points in the optimizer are not comparable.测量时点：每个梯度指标都要记清楚它是否带 scale、是累积后还是单个 microbatch、在裁剪前还是裁剪后。优化器里不同位置量出来的范数之间没有可比性。
Spike detection尖峰检测
For a positive scalar $x_t$ at step $t$, define $u_t=\log(x_t+\varepsilon)$ with numerical floor $\varepsilon>0$ and natural $\log$. Let $W_t$ contain earlier $u$-values.对第 $t$ 步的正标量 $x_t$，定义 $u_t=\log(x_t+\varepsilon)$，其中 $\varepsilon>0$ 是数值下限，$\log$ 取自然对数。$W_t$ 是此前若干个 $u$ 值组成的窗口。
Rolling median $q_t$ and deviation $D_t$.滑动中位数 $q_t$ 与偏差 $D_t$。P1 A local baseline and a spread estimate that a spike cannot inflate.一个局部基线，加一个不会被尖峰撑大的离散度估计。
Robust spike score $z_t$.稳健尖峰分数 $z_t$。P1 Normalize the current deviation by $D_t$.用 $D_t$ 对当前偏差做归一化。
The factor 0.6745 calibrates the score for normally distributed log-values. Apply the detector separately to $L$ and $G_2$.系数 0.6745 是按对数值服从正态分布来校准的。对 $L$ 和 $G_2$ 要各跑一套检测器。
Co-spike indicator $C_t$.共同尖峰指示 $C_t$。P0 Mark a step only when loss and gradient norm both spike.只有 loss 和梯度范数同时尖刺时才标记这一步。
This is the one worth paging on. Track cumulative count and cumulative magnitude, not just the instantaneous flag.值得接报警的就是这一个。要记累计次数和累计幅度，而不只是当前这一步的标志位。
Skip and replay counters.跳过与重放计数。P1 Count skipped updates, replayed steps, and replays that did not reproduce the original loss.分别统计被跳过的更新、被重放的步，以及重放后没有复现出原 loss 的次数。
The last one is the valuable counter: a replay that returns a different loss on identical inputs points at hardware, not data.最后一个才是真正有价值的：同样的输入重放出不同的 loss，问题在硬件，不在数据。
Thresholds: calibrate the window length and the $z_t$ threshold on stable runs. Do not tune them on a single failure.阈值：窗口长度和 $z_t$ 阈值要在稳定的训练上校准，不要拿一次故障去调。
5. System health系统健康
Let $\tau_r$ be the local step time on rank $r$, and $\tau=\max_r\tau_r$ the global wall time of the step.设 $\tau_r$ 为 rank $r$ 上的本地单步耗时，$\tau=\max_r\tau_r$ 为这一步的全局墙钟时间。
Token throughput $\operatorname{TPS}$.token 吞吐 $\operatorname{TPS}$。P0 With $N_{\mathrm{tok}}$ the global valid tokens in one step:记 $N_{\mathrm{tok}}$ 为一步内全局的有效 token 数：
Include data loading, forward, backward, optimizer, and synchronization in $\tau$.$\tau$ 要把数据加载、前向、反向、优化器和同步全算进去。
Compute rate $\Phi$.算力速率 $\Phi$。P0 With $F_{\mathrm{step}}$ estimated operations per step and $D$ devices:记 $F_{\mathrm{step}}$ 为每步估计的运算量，$D$ 为设备数：
Keep the FLOP estimator fixed across compared runs; changing it changes the metric without changing the hardware.参与对比的训练要用同一套 FLOP 估计口径；换了口径，硬件没变指标也会变。
Straggler score $S$.慢节点分数 $S$。P1 Global step time over median local step time.全局单步耗时除以本地单步耗时的中位数。
$S\approx1$ when ranks progress together. It rises when one slow rank sets the pace.各 rank 步调一致时 $S\approx1$。一旦被某个慢 rank 卡住节奏，它就会上升。
Peak memory.显存峰值。P0 Maximum allocated or reserved bytes in the interval, reduced with a maximum over ranks.统计区间内 allocated 或 reserved 的最大字节数，再跨 rank 取最大。
The max over ranks identifies the device closest to failure. Reset peak counters only at documented boundaries, or adjacent points describe different windows.跨 rank 取最大，才能定位到最接近挂掉的那张卡。峰值计数器只能在明确约定的边界上清零，否则相邻两个点描述的根本是不同的窗口。
Allocated and reserved memory.已分配与已预留显存。P1 Allocated bytes are held by live tensors; reserved bytes are held by the allocator, including cached blocks.allocated 是活跃张量占用的字节；reserved 是分配器持有的字节，含缓存块。
A widening reserved-minus-allocated gap means fragmentation or an oversized cache, not a larger model.reserved 减 allocated 的差越拉越大，说明是碎片化或缓存过大，而不是模型变大了。
Exposed collective wait $W_{\mathrm{comm}}$.暴露的集合通信等待 $W_{\mathrm{comm}}$。P1 With $\tau_{\mathrm{wait}}$ the critical-path collective time not overlapped with computation:记 $\tau_{\mathrm{wait}}$ 为关键路径上没有和计算重叠的集合通信时间：
Report collective count and payload alongside it, to separate many small calls from one bandwidth-limited large call.同时报集合通信的次数和数据量，才能区分「很多次小调用」和「一次被带宽卡住的大调用」。
Device and host telemetry.设备与主机遥测。P1 Utilization, power, temperature, clock, process memory, CPU pressure, page faults, network traffic.利用率、功耗、温度、频率、进程内存、CPU 压力、缺页、网络流量。
These separate model regressions from host, thermal, allocator, and network failures — which is most of what a throughput drop turns out to be.这些信号用来把模型自身的退化，和主机、散热、分配器、网络的故障区分开——吞吐掉下来，绝大多数时候是后者。
Timing: place device events around the real asynchronous wait and read them at an existing synchronization boundary. Adding host synchronization changes the overlap you are measuring.计时：device event 要卡在真正的异步等待两侧，并且在已有的同步点上读取。额外加一次主机同步，会直接改变你想测的那个重叠关系。
Node counters: sample each network interface once per node. Reading it from every local rank multiplies the reported traffic.节点级计数器：每个网卡每节点只采一次。每个本地 rank 都读一遍，报出来的流量会翻好几倍。
6. Tensor probes张量探针
Let $x=(x_1,\ldots,x_N)$ be a flattened tensor with $N$ elements. One pass over memory can produce all of the statistics below at once.设 $x=(x_1,\ldots,x_N)$ 是一个展平后含 $N$ 个元素的张量。下面这些统计量可以在一次内存遍历里一起算出来。
Absolute maximum.绝对值最大。P0 The largest absolute element.绝对值最大的那个元素。
The fastest indicator of an outlier, overflow, or local explosion. At the model output it is P0 in its own right: it moves before the loss.发现离群值、溢出和局部爆炸最快的指标。放在模型输出端，它本身就是 P0：它比 loss 先动。
Root mean square.均方根 RMS。P1 The square root of the mean squared element value.元素平方均值再开方。
RMS measures energy scale and stays comparable across tensor sizes. Store the square mean $M_2(x)=\frac1N\sum_j x_j^2$ instead: ranks reduce the squared sum and element count separately, and RMS follows.RMS 衡量能量尺度，而且在不同大小的张量之间可比。落盘时存平方均值 $M_2(x)=\frac1N\sum_j x_j^2$ 更好：各 rank 分别 reduce 平方和与元素数，RMS 由它推出来即可。
Absolute mean.绝对值均值。P1 The average absolute element value.所有元素绝对值的平均。
Typical scale without sign cancellation. It ignores a few large elements, so its gap from RMS is itself informative.不受正负抵消影响的典型量级。它对少数大元素不敏感，所以它和 RMS 之间的差本身就是信息。
Zero fraction $Z_x$.零元素比例 $Z_x$。P1 Exact zeros divided by $N$.严格为 0 的元素数除以 $N$。
Measures sparsity, underflow, and inactive paths.衡量稀疏程度、下溢和失活路径。
Excess kurtosis $\kappa$.超额峰度 $\kappa$。P2 With empirical mean $\mu$, normalize the fourth central moment by the squared variance.取经验均值 $\mu$，用方差的平方去归一化四阶中心矩。
Large positive $\kappa$ means heavy tails, a low-precision risk. Undefined at zero variance. Worth tracking on weights, rarely on activations.$\kappa$ 明显为正说明分布重尾，是低精度下的风险。方差为 0 时该量无定义。值得在权重上跟踪，激活上一般不必。
Observation points. Embeddings, logits, query/key/value, normalized query and key, attention output, feed-forward activations, residual streams, expert inputs and outputs, parameter gradients, applied updates. Name forward activations and backward gradients distinctly — the pair is what makes the scan diagnostic.观测点。embedding、logits、query/key/value、归一化后的 query 与 key、attention 输出、前馈激活、残差流、expert 的输入与输出、参数梯度、实际施加的更新。前向激活和反向梯度要用不同的名字——成对出现，这套扫描才有诊断力。
Reading an explosion. Walk the backward absolute maximum inward from the output: model output, each block's pre-clip tensors, then attention and feed-forward internals. The first layer that jumps is the one to investigate.怎么定位爆炸。沿反向的绝对值最大从输出往里走：模型输出、每个 block 裁剪前的张量，再到 attention 和前馈内部。第一个出现跳变的层，就是要查的那层。
Distributed reduction: aggregate sums and element counts before computing means, and aggregate maxima with a maximum reduction. Never average local maxima, or ratios whose denominators differ.分布式归约：先聚合求和项与元素计数再算均值，最大值要用 max 归约。绝不要去平均各 rank 的局部最大值，也不要平均分母不同的比值。
Relative update size相对更新幅度
Relative update size $\rho$.相对更新幅度 $\rho$。P1 For parameter tensor $w$ and applied update $\Delta w$:对参数张量 $w$ 和实际施加的更新 $\Delta w$：
A persistent fall toward zero means ineffective updates; a sudden rise means an update large relative to the parameter scale. The ratio is comparable across layers and runs; the raw update norm is not.持续趋近于 0 说明更新已经不起作用；突然抬高说明更新相对参数尺度过大。这个比值在不同层、不同训练之间可比，而更新的绝对范数不可比。
Optimizer chain.优化器链路。P2 Record the raw gradient, normalized gradient, preconditioned direction, and applied update as separate tensors.把原始梯度、归一化后的梯度、预条件后的方向、最终施加的更新当成四个独立张量分别记录。
This locates the stage that introduces a scale or direction change. Too many series for a standing dashboard; turn it on for an investigation.用来定位是哪一级引入了尺度或方向的变化。曲线太多，不适合常驻看板，排查时再打开。
Naming: specify whether an optimizer tensor is before or after normalization, preconditioning, weight decay, learning-rate scaling, and clipping.命名：优化器里的张量要写清楚它在归一化、预条件、weight decay、学习率缩放和裁剪之前还是之后。
7. Expert balance专家负载均衡
Let $E$ be the number of experts and $c_i$ the valid tokens sent to expert $i$, with mean load $\bar c=\frac1E\sum_i c_i$.设 $E$ 为 expert 数，$c_i$ 为路由到 expert $i$ 的有效 token 数，平均负载 $\bar c=\frac1E\sum_i c_i$。
Imbalance ratio $I$ and maximum violation.不均衡比 $I$ 与最大违反度。P0 Compare the busiest expert with the mean.把最忙的 expert 和平均值做比较。
Perfect balance gives $I=1$ and $\operatorname{MaxVio}=0$; $\operatorname{MaxVio}=0.25$ means the busiest expert takes 25% more tokens than the mean. MaxVio is the form to put on a dashboard, because it is zero when nothing is wrong.完全均衡时 $I=1$、$\operatorname{MaxVio}=0$；$\operatorname{MaxVio}=0.25$ 表示最忙的 expert 比平均多拿 25% 的 token。上看板用 MaxVio 更合适：没问题的时候它就是 0。
Shard imbalance.分片不均衡。P0 The same ratio applied to expert-parallel shard loads.把同一个比值用在 expert 并行的分片负载上。
Expert-level balance does not imply balanced communication groups, and it is the shard load that sets the step time.单个 expert 均衡了，不等于通信组也均衡；真正决定单步耗时的是分片负载。
Routing entropy $H$ and $H_{\mathrm{norm}}$.路由熵 $H$ 与 $H_{\mathrm{norm}}$。P1 With normalized loads $p_i=c_i/\sum_j c_j$:取归一化负载 $p_i=c_i/\sum_j c_j$：
$H_{\mathrm{norm}}=0$ is complete concentration, 1 is uniform routing. A sharp drop means the router is collapsing onto a few experts, usually before $I$ makes it obvious.$H_{\mathrm{norm}}=0$ 表示完全集中，1 表示均匀路由。它骤降说明 router 正在塌缩到少数几个 expert 上，通常比 $I$ 更早看出来。
Per-expert load and router scores.单 expert 负载与路由分数。P2 Per-expert token fractions, routing bias terms, top-$k$ margin distributions.每个 expert 的 token 占比、路由 bias 项、top-$k$ 的 margin 分布。
These answer which expert once the aggregate has already said that something is wrong.聚合指标已经告诉你「出问题了」之后，这些才用来回答「是哪个 expert」。
Masking: exclude padding and invalid tokens, and skip ratios entirely when no valid token reaches the layer. An empty shard is not a measured zero.掩码：要排除 padding 和无效 token；这一层一个有效 token 都没收到时，比值就整个跳过不记。空分片不是一个测出来的 0。
8. Residual paths残差路径
Let $s\in\mathbb{R}^{d}$ be one token's residual-stream vector, $b\in\mathbb{R}^{d}$ the branch output added to it, and $a=s+b$ the updated stream.设 $s\in\mathbb{R}^{d}$ 是某个 token 的残差流向量，$b\in\mathbb{R}^{d}$ 是加到它上面的分支输出，$a=s+b$ 是更新后的残差流。
Relative branch scale $R_b$.分支相对尺度 $R_b$。P0 Branch norm $N_b=\lVert b\rVert_2$ over stream norm $N_s=\lVert s\rVert_2$.分支范数 $N_b=\lVert b\rVert_2$ 除以残差流范数 $N_s=\lVert s\rVert_2$。
Small means a branch too weak to matter; large means a branch that overwrites the stream. Plotted across depth, it is the clearest single view of whether a deep model uses its depth. Report $N_s$ and $N_b$ too: the ratio cannot distinguish a shrinking branch from a growing stream.值小说明分支弱到无关紧要，值大说明分支直接盖掉了主干。沿深度画出来，它是判断一个深模型有没有用上自己深度的最清楚的一张图。$N_s$ 和 $N_b$ 也要各自报：光看比值分不清是分支在缩小，还是残差流在增大。
For vectors $u$ and $v$, cosine similarity is对向量 $u$ 与 $v$，余弦相似度定义为
Block importance $B$ and angular distance $A$.块重要性 $B$ 与角距离 $A$。P1 The directional change from $s$ to $a$, raw and normalized.从 $s$ 到 $a$ 的方向变化，分别给原始值和归一化值。
$B=0$ when the update preserves direction, and $A$ lies in $[0,1]$. A block whose $B$ decays toward zero has stopped contributing, whatever its branch norm says.更新不改变方向时 $B=0$，$A$ 落在 $[0,1]$ 内。$B$ 衰减到 0 的 block 已经不再有贡献了，不管它的分支范数看起来多大。
BF16 no-op fraction $F_{\mathrm{noop}}$.BF16 空更新比例 $F_{\mathrm{noop}}$。P1 With $Q_{\mathrm{BF16}}(\cdot)$ rounding to bfloat16, count nonzero branch updates that rounded addition throws away.记 $Q_{\mathrm{BF16}}(\cdot)$ 为舍入到 bfloat16，统计那些非零、却在舍入相加后完全没留下痕迹的分支更新。
A large value means residual updates are being swallowed by precision. It grows as the stream norm grows, so watch it with $N_s$.值偏大说明残差更新正在被精度吃掉。它会随残差流范数一起变大，所以要和 $N_s$ 一起看。
Gate saturation fraction.门控饱和比例。P1 For a branch scaled by a learned gate, let $\beta_j=\sigma(g_j/T)$ with logistic $\sigma$, gate $g_j$, temperature $T$. Report channels below a threshold $\theta$.对带可学习门控的分支，记 $\beta_j=\sigma(g_j/T)$，其中 $\sigma$ 是 logistic 函数，$g_j$ 是门控参数，$T$ 是温度。统计低于阈值 $\theta$ 的通道比例。
Track a few thresholds, say $\theta\in\{0.05,0.1,0.25\}$. A branch whose gates have all closed contributes nothing and rarely reopens.取几个阈值一起跟踪，例如 $\theta\in\{0.05,0.1,0.25\}$。门全关死的分支不再有任何贡献，而且几乎不会自己打开。
Aggregation: compute these per valid token, then report the mean and upper quantiles. Flattening all tokens and features together hides the token-level outliers you are looking for.聚合：这些量要逐个有效 token 算，再报均值和高分位。把所有 token 和特征拍平在一起，恰好会把你要找的 token 级离群点藏掉。
9. Modality paths模态路径
A multimodal run adds one failure mode: a modality degrading, or never contributing, while the aggregate curves look healthy. Three probes cover most of it; the fourth check — whether media inputs carry information at all — is an evaluation, in Section 10.多模态训练多出一种失败方式：聚合曲线一切正常，而某个模态正在退化，或者从头到尾就没起过作用。下面三个探针能覆盖大部分情况；第四项检查——媒体输入到底有没有带来信息——属于评测，放在第 10 节。
Media scale ratio $\gamma$.媒体特征尺度比 $\gamma$。P0 Let $e^{(\mathrm{media})}$ be the embeddings an encoder and its projector hand to the backbone, and $e^{(\mathrm{text})}$ the text embeddings entering at the same point.记 $e^{(\mathrm{media})}$ 为编码器与投影层交给语言主干的 embedding，$e^{(\mathrm{text})}$ 为在同一位置进入主干的文本 embedding。
Near 1, the two streams enter at the same scale. Persistently small, and the backbone can ignore media tokens at no cost to the loss — so it will. Large, and media tokens dominate the early layers. Both are projector problems more often than encoder problems.接近 1 说明两路特征以同一尺度进入主干。长期偏小，主干可以零代价地忽略媒体 token——那它就会真的忽略。偏大，媒体 token 会主导前几层。这两种情况多半是投影层的问题，而不是编码器的问题。
Variable media length: samples carry different numbers of tiles or frames, so weight per-modality statistics by token count rather than averaging over samples. A rank that received no media has no measurement, not a zero.媒体长度可变：不同样本带的 tile 或帧数不同，所以按模态统计时要用 token 数加权，而不是对样本求平均。没收到媒体的 rank 是「没有测量值」，不是 0。
10. Evaluation评测
Let $M$ be the number of valid held-out tokens and $\ell_k$ the negative log-likelihood of token $k$.设 $M$ 为留出集中有效 token 的数量，$\ell_k$ 为第 $k$ 个 token 的负对数似然。
Negative log-likelihood and perplexity.负对数似然与困惑度。P0 Average held-out token losses, and exponentiate.对留出集 token 的 loss 取平均，再取指数。
Perplexity adds no information over NLL, and is comparable only when tokenization and loss normalization are fixed.困惑度相对 NLL 没有增加任何信息，而且只有在分词方式和 loss 归一化都固定时才可比。
Bits per byte.每字节比特数 BPB。P0 With $N_{\mathrm{byte}}$ the source bytes covered by the held-out tokens:记 $N_{\mathrm{byte}}$ 为这些留出 token 所覆盖的原始字节数：
The right default for comparing runs: insensitive to tokenizer vocabulary and segmentation, which token-level perplexity is not.跨训练对比时的默认选择：它对分词器的词表和切分方式不敏感，而 token 级困惑度做不到这一点。
Choice accuracy.选择题准确率。P0 Over $Q$ questions with prediction $\hat a_q$ and reference $a_q$:共 $Q$ 道题，预测为 $\hat a_q$，参考答案为 $a_q$：
For multiple choice, select $\hat a_q$ by length-normalized option NLL. Freeze prompt construction and answer normalization for the life of the run.多选题用长度归一化后的选项 NLL 来选 $\hat a_q$。prompt 的拼法和答案归一化方式，在整个训练周期内都不要再改。
Greedy generation.贪心生成。P0 Decode at temperature 0 and score after task-specific post-processing: exact match, pass@1, or a symbolic checker.temperature 取 0 解码，按任务做完后处理再打分：exact match、pass@1，或者符号判定器。
The cheapest evaluation that exposes failures likelihood scoring cannot see. Worth running on every key checkpoint, not only at the end.在所有能暴露「似然打分看不见的失败」的评测里，它最便宜。值得在每个关键 checkpoint 上跑，而不是只在最后跑。
Media gain $\Delta_{\mathrm{media}}$.媒体增益 $\Delta_{\mathrm{media}}$。P0 The same higher-is-better score computed with and without the media input.同一个「越高越好」的指标，分别在有媒体输入和去掉媒体输入的条件下各算一次。
The only metric that proves a multimodal model uses its media input rather than answering from text priors. A capable model with $\Delta_{\mathrm{media}}\approx0$ is a unimodal model with extra cost.这是唯一能证明多模态模型真的在用媒体输入、而不是靠文本先验作答的指标。一个能力不错但 $\Delta_{\mathrm{media}}\approx0$ 的模型，本质上是一个成本更高的单模态模型。
Pass at $k$.pass@$k$。P1 For one problem with $n$ samples, $c$ of them passing, and $k\le n$:对某道题采样 $n$ 次，其中 $c$ 次通过，取 $k\le n$：
If $n-c<k$, the estimate is 1. Pass@1 evaluates one attempt; larger $k$ measures coverage under repeated sampling.当 $n-c<k$ 时估计值为 1。pass@1 衡量一次作答的水平，$k$ 更大时衡量的是重复采样下的覆盖率。
Intersection over union.交并比 IoU。P1 For predicted region $\mathcal{P}$ and reference $\mathcal{G}$:对预测区域 $\mathcal{P}$ 与参考区域 $\mathcal{G}$：
0 for no overlap, 1 for identical regions. Promote to P0 for a run whose target capabilities are grounding-heavy.完全不重叠为 0，完全重合为 1。如果这次训练的目标能力以 grounding 为主，就把它提到 P0。
Averaged-weight evaluation.权重平均评测。P1 Evaluate a running average of the weights alongside the raw ones.在评测原始权重的同时，也评测一份权重的滑动平均。
Mid-run comparisons on raw weights are noisy enough to reverse a ranking between two recipes. Decide once which weight stream is the reference for decisions.训练中途拿原始权重做对比，噪声足以把两个方案的排序整个颠倒。一次性定好哪一路权重是决策依据，然后一直用它。
Comparability: version the dataset, tokenizer, prompt, decoding parameters, post-processing, and judge. A metric change is uninterpretable if its definition also changed.可比性：数据集、分词器、prompt、解码参数、后处理和判分器都要有版本号。定义跟着一起变了的指标，涨跌都没法解释。
Extrapolation: when a scaling fit predicts a larger run, report a bound outside the fitted range rather than a point estimate.外推：用 scaling 拟合去预测更大规模时，拟合区间之外应该给区间上下界，而不是一个点估计。
11. Common mistakes常见错误
Every one of these produces a plausible-looking curve, which is what makes them expensive.下面每一条都会画出一条看起来挺合理的曲线——这正是它们代价高的原因。
| Mistake错误 | What goes wrong会出什么问题 |
|---|---|
| Everything is P0什么都是 P0 | If every panel can page you, none of them does如果每块面板都能把你叫醒，那就等于没有一块能 |
| Wrong reduction group归约组用错 | Data, tensor, pipeline, context, and expert-parallel groups have different semanticsdata、tensor、pipeline、context、expert 并行组的语义各不相同 |
| Unclear maximum没说清最大值是对谁取的 | A maximum over tensor elements, tokens, examples, or GPU ranks gives four different metrics under one name对张量元素、token、样本、GPU rank 取最大，是同一个名字下的四个不同指标 |
| Ambiguous timing测量时点含糊 | Gradients scaled, accumulated, clipped, or already applied are four different quantities带 scale 的、累积后的、裁剪后的、已施加的梯度，是四个不同的量 |
| Duplicate recomputation重复计算 | Activation checkpointing may run a forward probe a second time, double-counting it激活重算会让前向探针跑第二遍，导致重复计数 |
| Empty shards空分片 | Absence of data recorded as a measured zero drags every average toward zero把「没有数据」记成测量值 0，会把所有平均值往下拽 |
| Padding counted as tokens把 padding 当成 token | Distorts loss, routing, and modality statistics in proportion to how much you padpadding 越多，loss、路由和模态统计失真越严重 |
| Over-smoothing过度平滑 | A smoothed curve with no raw curve behind it hides exactly the short spikes you are watching for平滑曲线背后不留原始曲线，藏掉的恰好是你要找的短尖峰 |
| High-cardinality labels高基数标签 | Arbitrary parameter names and sample identifiers overwhelm online metric stores任意的参数名和样本 ID 会把在线指标存储撑爆 |
| Sensitive artifacts敏感产物 | Raw samples and token-level traces must be opt-in, access-controlled, and short-lived原始样本和 token 级 trace 必须默认关闭、有权限控制、短期保留 |
12. Keeping probes cheap控制探针开销
- Register once. Choose tensor locations while building the model.只注册一次。在搭模型的时候就把观测点定好。
- Gate early. On an unsampled step, skip the scan itself, not just the write.尽早判断。不采样的那些步，要跳过扫描本身，而不只是跳过写盘。
- Scan once. Compute every local statistic for a tensor in one pass over memory.只扫一遍。一个张量的所有局部统计量，在一次内存遍历里全部算完。
- Reduce in batches. Group values by reduction type and issue one collective per group.批量归约。按归约类型分组，每组只发一次集合通信。
- Move only summaries. Keep full tensors on the device.只搬摘要。完整张量留在设备上。
- Write asynchronously. Never put storage latency on the training path.异步落盘。绝不要把存储延迟放到训练主路径上。
Let $K$ be the sampling interval in steps. Retain every $K$-th sample, every anomaly step, the warm-up and final steps, and 50–100 steps on each side of an anomaly. Small runs can use $K=1$; long runs use $K$ between 100 and 200 for expensive views, while cheap detectors keep running every step.设 $K$ 为采样间隔（单位：步）。保留每第 $K$ 步的样本、所有异常步、warm-up 与最后若干步，以及异常点前后各 50–100 步。小规模训练可以直接 $K=1$；长训练里昂贵的视图取 $K$ 在 100 到 200 之间，同时便宜的检测器仍然每步都跑。
A detector cannot recover expensive measurements from before the event that triggered it. If that context matters, keep a short in-memory ring buffer, or replay the affected batch under denser probes.检测器没法把触发之前那些昂贵的测量补回来。如果这段上下文重要，就在内存里留一个短的环形缓冲，或者把出问题的 batch 用更密的探针重放一遍。
@misc{deng2026pretraining,
title = {Reading a Pretraining Run},
author = {Chunyuan Deng},
year = {2026},
month = sep,
howpublished = {Blog post},
url = {https://charlesdddd.github.io/blog/reading-a-pretraining-run.html}
}