A looped mixture-of-experts runs the same layers three times. Does it do anything different on each pass?
Start with a single MoE layer. A token arrives with hidden state $h_t \in \mathbb{R}^d$, and a small linear router scores every one of the $E$ experts:
In the model we use, $E = 128$. Evaluating all of them would defeat the purpose, so the layer keeps only the best $k = 8$,
renormalises the scores over the survivors, and mixes their outputs:
We will call $\mathcal{S}_t$ the token's path: the eight-expert route it takes through this layer. Stack 48 such layers and a token's full journey through the model is a sequence of 48 such choices.
Now the part that matters for fine-tuning. The other $E-k = 120$ experts are never evaluated, so they appear nowhere in the computation graph, so nothing flows back to them:
Fine-tuning therefore trains the experts that the pretrained router already liked. In principle the router could learn to like different ones — $W_r$ does receive gradient. In practice it barely moves: over a full epoch we measure the routing entropy shifting by 2% and top-1 confidence by 4% (see the probes). Whatever paths pretraining laid down, fine-tuning walks along them.
So here is the question this project starts from. Can we give a token more than one path through the same weights?
The simplest way to get a second path is to run the same layers a second time. Write one transformer layer as
and pick a window of consecutive layers to repeat — for us layers $[a,b) = [12,36)$, so $W = 24$ layers wide, sitting in the middle of the 48. Call the composition of that window $F = T_{b-1} \circ \cdots \circ T_a$.
A token first runs the layers below the window, once:
Then the window runs $K = 3$ times. Each pass $\tau$ produces a candidate state
and the obvious question is how to combine the passes. Simply chaining them ($h_\tau = u_\tau$) makes the model three times deeper on day one, which is a large and untested change to a pretrained network. Instead each pass after the first gets a learned gate $g_\tau \in \mathbb{R}^d$ that decides, per channel, how much of the new candidate to accept:
Finally the layers above the window run once on the last state:
The token has now been through $a + KW + (L-b) = 12 + 72 + 12 = 96$ executed layers using 48 layers' worth of weights. The only new parameters anywhere are the $2d$ gate values.
Set $g_\tau = 0$ and the update above gives $h_\tau = h_{\tau-1}$ for every $\tau \geq 1$, so $h_{K-1} = u_0 = F(h_{-1})$ and the looped model reproduces the base model bit for bit. Fine-tuning then starts from a known-good point and opens the loop only as far as the loss rewards. That is a real advantage: there is no initialisation shock, and no risk that a promising idea is buried under the damage of turning a 48-layer model into a 96-layer one overnight.
It also has a cost, and the cost turns out to dominate everything else in this post. Because $W_r$ is shared across passes, pass $\tau$ can route differently from pass $\tau-1$ only if the router's input moved enough to reorder its top eight. From the update rule, that movement is
which is proportional to the gate. So small gates give small movement, small movement gives the same routing:
The safe initialisation and the goal are in tension: the gates start where the passes are guaranteed identical, and they have to travel a long way before the passes are meaningfully different. We measure how far they actually travel below — the answer is $\|g_\tau\|_{\text{rms}} \approx 0.02$ after a full epoch, and the passes are indeed near-copies.
The probes below say this tension resolves the wrong way: passes 1 and 2 are near-copies of each other. Pathloop attacks that from the router's side rather than the gate's. If the passes will not become different on their own, make them different by construction: during training, let each pass sample part of its own path. Inference is untouched — the served model is plain top-$k$ routing.
Split the budget $k = p + q$. The first $p$ experts are the model's own top choices, kept as-is:
The remaining $q$ slots are for exploration. If every pass sampled from the same pool of non-selected experts they would frequently draw the same ones, which is exactly the collision we are trying to avoid. So we first partition the experts by a random assignment $\beta$, redrawn every step, which hands each pass a disjoint candidate pool:
Within its own pool, pass $\tau$ draws $q$ experts without replacement, with probability proportional to the router's own scores. The standard trick for this is Gumbel top-$q$:
The path this pass actually takes is the union, renormalised over its own members:
Two things are worth noticing. The cost is unchanged — still exactly $k$ experts evaluated per token, so this is not a compute-for-quality trade. And the $q$ sampled experts now sit in the computation graph, so they receive gradient, which the zero-gradient property above otherwise forbids. With $q = 0$ the whole construction collapses back to ordinary top-$k$, which is what makes it safe to schedule.
Sampling only helps if we can tell a good draw from a bad one. Because every pass produces a full-width hidden state, we can read each one through the model's own output head and score it on the true next token:
This is a logit lens: a cheap, honest measure of how much of the answer pass $\tau$ has already assembled. It is not calibrated in absolute terms — mid-depth states read poorly through a head trained on final-depth states — but we only ever use differences.
Which is the second point. The $K$ passes saw the same token in the same context, so comparing a pass against its own siblings cancels token difficulty exactly, with no baseline model and no value network:
Given an advantage per pass, the sampled experts are reinforced by an ordinary policy gradient, where $\pi_\tau$ is the router's probability of having drawn that particular set as an ordered draw without replacement:
A second term pushes the passes to improve monotonically, weighting later passes more:
Why score each pass against the label rather than, say, rewarding the passes for agreeing with each other? Because the cheapest way to make passes agree is to send $g_\tau \to 0$ — the degenerate solution the gates are already sitting next to. Scoring against the label makes the shortcut worthless.
Both terms are multiplied by the exploration schedule and vanish wherever $q = 0$, so $q = 0$ is exactly plain looped SFT, bit for bit. We run this in two stages. Tier 1 samples and logs $R_\tau$ and $A_\tau$ without training on them: it asks whether the comparison carries any signal at all before anything is built on top of it. Tier 2 switches on the two losses. In both, $q$ follows a schedule that returns to $0$ before training ends, so the final model is trained under the same policy it is served with.
Base model is Qwen3-30B-A3B-Base — 30B parameters with 3B active per token, 48 layers, 128 experts per layer, top-8 routing. The loop covers layers 12–35, repeated three times.
We fine-tune on Ling-Coder, a 400k-example coding instruction set, for one epoch (415 steps). A math dataset (OT3) is the alternative we have also used; everything reported here is Ling-Coder.
We evaluate on 14 benchmarks with greedy decoding. Because we fine-tune on coding data, the benchmarks split into two groups, and keeping them apart turns out to matter more than any single number:
Three fine-tuned arms against the untouched base model. "Looped" is the gated loop described above; the two looped arms differ only in how fast the gates are allowed to learn (100× the base rate in the second).
| Category | Benchmark | Base | Plain SFT | Looped SFT | Looped SFT (fast gates) | |
|---|---|---|---|---|---|---|
| coding | HumanEval | in | 72.0 | 90.2 149 | 90.2 299 | 91.5 299 |
| HumanEval+ | in | 66.5 | 84.1 149 | 86.0 299 | 87.2 299 | |
| MBPP | in | 68.5 | 74.5 299 | 73.0 299 | 75.5 299 | |
| MBPP+ | in | 76.0 | 74.4 415 | 73.6 299 | 74.0 299 | |
| LiveCodeBench | in | 14.9 | 31.4 149 | 32.6 299 | 29.7 299 | |
| math | GSM8K | ood | 52.8 | 72.4 149 | 69.2 415 | 65.2 415 |
| MATH500 | ood | 60.8 | 57.6 149 | 59.6 149 | 66.0 149 | |
| MATH500-flex | ood | 60.8 | 58.0 149 | 62.4 149 | 67.2 149 | |
| AMC23 | ood | 27.9 | 32.5 149 | 30.8 149 | 39.6 415 | |
| AIME24 | ood | 4.4 | 10.0 415 | 10.0 299 | 11.7 149 | |
| AIME25 | ood | 2.8 | 5.0 415 | 7.2 415 | 6.7 149 | |
| knowledge | MMLU-Pro | ood | 52.4 | 51.6 149 | 52.8 415 | 51.2 149 |
| GPQA-Diamond | ood | 38.9 | 43.9 299 | 43.4 415 | 43.9 299 | |
| instruction following | IFEval | ood | 35.2 | 53.6 299 | 53.6 415 | 56.0 299 |
| Mean of best, all 14 | 45.27 | 52.81 | 53.18 | 54.67 | ||
| Mean of best, in-domain | in | 59.55 | 70.94 | 71.08 | 71.57 | |
| Mean of best, out-of-domain | ood | 37.34 | 42.74 | 43.23 | 45.27 | |
Each fine-tuned number is that arm's best across the three checkpoints we saved (149, 299, 415); the small grey figure is which one. Taking a per-benchmark maximum is optimistic — it picks the checkpoint after seeing the score — but it is optimistic for every arm equally, and it avoids crediting a method merely for degrading more slowly.
In-domain, looping does nothing. Fine-tuning lifts coding from 59.6 to about 71, and all three fine-tuned arms land within 0.6 of each other. Whatever the loop contributes, it is not coding ability.
Out-of-domain is where the arms separate, but by less than a final-checkpoint comparison suggests. At its best checkpoint plain fine-tuning reaches 42.7 — a real gain of five points over the base model. Standard-gate looping reaches 43.2, barely ahead. Only the fast-gate arm opens a clear margin, at 45.3.
This matters because comparing final checkpoints tells a much flattering story: there, plain fine-tuning scores 37.5 against 40.3 and 40.9 for the looped arms. That gap is almost entirely plain fine-tuning having fallen back to the base model by step 415, not the looped arms having learned more.
For context, here are our five coding benchmarks against open-source coding models, diffusion language models, and Ouro — the one other looped latent-reasoning model with published code numbers. Our rows are the same best-of-checkpoint values as the table above.
| Model / Method | Base model | MBPP | MBPP+ | HumanEval | HumanEval+ | LCB v6 | Avg. |
|---|---|---|---|---|---|---|---|
| Published models (numbers as reported by their authors) | |||||||
| Autoregressive coding models | |||||||
| Qwen2.5-Coder | Qwen2.5-7B | 75.9 | 62.9 | 66.5 | 60.4 | 26.8 | 59.9 |
| OpenCoder | from scratch (8B) | 79.9 | 70.4 | 66.5 | 63.4 | 29.6 | 62.0 |
| OlympicCoder | Qwen2.5-7B | 80.0 | 66.4 | 82.1 | 76.9 | 37.3 | 68.5 |
| Seed-Coder | from scratch | 82.0 | 69.0 | 77.4 | 68.3 | 28.4 | 65.0 |
| Diffusion language models | |||||||
| Dream | Qwen2.5-7B | 68.7 | 57.4 | 56.7 | 50.0 | 18.6 | – |
| LLaDA | from scratch (8B) | 50.1 | 42.1 | 35.4 | 30.5 | 12.4 | – |
| DiffuCoder | Qwen2.5-Coder-7B | 75.1 | 61.9 | 72.0 | 65.2 | 24.5 | 59.7 |
| Dream-Coder | Qwen2.5-Coder-7B | 75.9 | 61.6 | 66.5 | 60.4 | 21.4 | 57.2 |
| d1 | LLaDA + RL | 39.0 | – | 45.5 | – | – | – |
| Looped latent-reasoning models | |||||||
| Ouro | from scratch (2.6B) | 80.4 | 66.6 | 78.2 | 70.7 | 38.7 | 66.9 |
| This work (our eval harness, greedy decoding, best of checkpoints 149 / 299 / 415) | |||||||
| Base model | Qwen3-30B-A3B-Base | 68.5 | 76.0 | 72.0 | 66.5 | 14.9 | 59.6 |
| Plain SFT | Qwen3-30B-A3B-Base | 74.5+6.0 | 74.4−1.6 | 90.2+18.2 | 84.1+17.6 | 31.4+16.5 | 70.9+11.4 |
| Looped SFT | Qwen3-30B-A3B-Base | 73.0+4.5 | 73.6−2.4 | 90.2+18.2 | 86.0+19.5 | 32.6+17.7 | 71.1+11.5 |
| Looped SFT (fast gates) | Qwen3-30B-A3B-Base | 75.5+7.0 | 74.0−2.0 | 91.5+19.5 | 87.2+20.7 | 29.7+14.8 | 71.6+12.0 |
Subscripts are the change from our base model. Underlined is the best published number in that column; bold is the best of our three fine-tuned arms. The two blocks are not a head-to-head: the published numbers come from each paper's own harness and ours from a shared few-shot greedy setup, and the gap is large enough to see — a Qwen3-8B base is reported at 78.2 on HumanEval where our larger Qwen3-30B-A3B base measures 72.0. Read the top block as a scale for what these benchmarks look like, and the bottom block as the comparison that is actually controlled.
Evaluating at three checkpoints shows what is actually happening. Out-of-domain, every arm peaks at step 149 — the earliest checkpoint we saved — and declines from there. Plain fine-tuning declines all the way back to the base model. The looped arms decline more slowly and stop above it.
Read three ways, out-of-domain:
| Out-of-domain mean (9 benchmarks) | Base | Plain SFT | Looped SFT | Looped SFT (fast gates) |
|---|---|---|---|---|
| Best single checkpoint (step 149 for all) | 37.34 | 41.24 | 41.39 | 43.95 |
| Best per benchmark, any checkpoint | 37.34 | 42.74 | 43.23 | 45.27 |
| Final checkpoint (step 415) | 37.34 | 37.49 | 40.29 | 40.95 |
The top two rows say the loop does not lift the peak: with standard gates it lands within 0.2 of plain fine-tuning however you measure. The bottom row says it holds that peak far better — plain fine-tuning gives back all five points it gained, the looped arms give back one to three. Only raising the gate learning rate moves the peak itself, by about 2.7 points.
The means above hide a lot. Below is every benchmark on its own axis, all three arms plotted across the three checkpoints, with the base model as a dashed reference. The panels with a green tint are the five coding benchmarks we actually train on.
We log internal statistics every 25 steps. Two of them reframed the project.
This is the assumption pathloop is built on, and the probes confirm it. Over a full epoch, the routing distribution is almost unchanged:
| Router statistic | Step 0 | Step 400 | Change | What it measures |
|---|---|---|---|---|
| Routing entropy | 0.8905 | 0.9094 | +2.1% | how spread out the expert choice is |
| Top-1 probability | 0.0587 | 0.0563 | −4.1% | confidence in the single best expert |
| Load imbalance (cv) | 1.012 | 1.130 | +11.7% | how unevenly tokens spread over experts |
A full epoch of fine-tuning moves the router by a couple of percent. Routing established in pretraining survives fine-tuning essentially intact — which is exactly why an intervention that forces the router to try alternatives might be worth something.
This is the measurement that matters most, and the one that reframed the project. The probes are keyed by executed position, not by parameter layer, so all 96 executed layers are recorded separately. Parameter layer $L$ appears at executed positions $L$, $L+24$ and $L+48$ — once per loop pass. Comparing those three tells us directly what each pass does with the same weights.
| Measured at each pass | Pass 0 | Pass 1 | Pass 2 | Pass 1 → 2 | Pass 0 → 1 |
|---|---|---|---|---|---|
| Routing entropy | 0.9113 | 0.9136 | 0.9136 | 0.0000 | 0.0023 |
| Top-1 probability | 0.0586 | 0.0529 | 0.0529 | 0.0000 | 0.0057 |
| Representation change (1 − cos) | 0.0144 | 0.0028 | 0.0028 | 0.0000 | 0.0116 |
| Load imbalance (cv) | 1.168 | 1.198 | 1.196 | 0.002 | 0.030 |
Pass 0 is ungated and always applied in full, so it does real work. Passes 1 and 2 are multiplied by a gate that reaches only about 0.02–0.06, so they nudge the state rather than transform it. A nudged state produces nearly the same router scores, so it selects nearly the same experts — and the two nudges land in nearly the same place as each other.
This is a problem for pathloop specifically. Its whole mechanism is to compare passes against each other and reward the better one. If the passes are copies, every comparison is between identical things, the advantage $A_\tau$ is zero, and there is nothing to learn from. We confirmed this directly: in short test runs the spread of reward across passes sat at roughly $10^{-7}$, i.e. exactly zero.
It is also a caveat on the claim of "96 effective layers". By this measurement the model behaves closer to 48 real layers plus two cheap refinement passes.
One honest limit on the table above: these are distributional statistics — entropy, top-1 probability, load spread. Two genuinely different sets of 8 experts could in principle produce the same entropy. The statistics being identical to four decimals is strong evidence the passes route the same way, but it is not proof. We have since added a metric that compares the selected expert sets directly, token by token, and reports what fraction of each token's 8 experts carries over from one pass to the next. It landed shortly after the current run had already started, so that run does not record it — the direct measurement still needs a later one.
Building it surfaced five bugs, every one silent — nothing crashed, and each would have produced a clean-looking run that measured nothing:
The run in progress places exploration in the middle of training — off for the first 100 steps, easing to $q{=}1$, full at $q{=}2$ from 150 to 300, then easing back to zero by 350. The shaded band in Figure 3 shows why: exploring before the gates open would perturb passes that are still doing nothing.
The question this run answers is narrow and prior to everything else: once the gates are open, do the three passes ever become different enough to compare? If the reward spread lifts off the floor, the comparison is real and tier 2 is worth training. If it stays at zero through the exploration window, then the passes are copies no matter what the router does, and the honest next step is not more pathloop but changing how the loop is built — starting the gates away from zero, so the passes are distinct from the beginning.