Asymmetric Causal Encoder-Decoder

Topics We’ll Cover

• Why Prefill Is the Bottleneck in Agent Loops
• Three Shapes: Decoder-Only, Encoder-Decoder, Causal Encoder-Decoder
• Where the Decoder’s KV Comes From
• What Prefill Skips, and the Replay Window
• Activated Parameters and FLOPs per Token
• The KV-Cache Saving, Arithmetic Shown
• Serving It: Asymmetric Prefill and Decode Pools
• Measured: Pool Split and Capacity in a Simulator
• Trade-offs and Open Questions
• Relatives: YOCO, CLA and Cross-Layer Sharing
• What to Take Away

Why Prefill Is the Bottleneck in Agent Loops

A chat turn reads a prompt once and writes a long answer. An agent loop is the other way round: each step appends a tool result or a file to an ever-growing context and then writes a short tool call. The report opens with exactly this: “the widespread adoption of long-horizon agents has made model workloads increasingly input-heavy” (abstract, p. 1).

Three Shapes: Decoder-Only, Encoder-Decoder, Causal Encoder-Decoder

“Causal” is what separates CED from T5: it is a decoder-only model in everything that matters for serving (one stream, prefix caching, autoregressive decode), but with the property that a prompt token’s contribution to every layer’s global KV is fixed once it has passed through the encoder. DeepSeek-V4.1-Flash has 40 layers, split 20 + 20 (Figure 3, p. 7; section 4.2.1, p. 21).

Where the Decoder’s KV Comes From

For global attention, CED treats the bottom L/2 layers as the causal encoder. For a decoder layer l > L/2, the KV entries are “not derived from their respective hidden states H_l” but “projected directly from the hidden state of the (L/2)-th layer, H_L/2, using layer-dependent projection weights” (section 2.2, equation 1, p. 9):

C_l = H_(L/2) · W_l^KV        Z_l = H_(L/2) · W_l^Z        for l > L/2
# C: the decoder layer's global KV entries; Z: their compression weights (the CSA2 compressor)

The sliding-window (SWA) branch is different: every layer, decoder layers included, still computes its local keys and values from its own hidden state H_l (section 2.2, p. 9). That is what forces the replay on the next slide. In the shipped model the decoder’s global KV is also shared across layers by CSA2: only the decoder’s first layer (Full mode) computes it from the encoder state, and later decoder layers reuse it (sections 2.3.1 and 4.2.1, pp. 11 and 22).

What Prefill Skips, and the Replay Window

Prefill computes the encoder for every prompt token, plus the decoder’s global-KV projections from the encoder output. The decoder’s per-layer computation is skipped for all but the last n_win prompt tokens.

decoder-only:  O(N · L)
CED:           O(N · L/2  +  n_win · L/2)  ≈  O(N · L/2)      # for N >> n_win: "effectively halving"

Activated Parameters and FLOPs per Token

Two kinds of sparsity now stack. Mixture-of-experts routing activates a few experts per token (Arch 01: total versus active parameters); CED then decides which layers a token runs at all.

DeepSeek-V4.1-Flash: 40 layers, hidden size 5,120; every layer is DeepSeekMoE with 1 shared and 384 routed experts, 6 routed experts active per token, expert width 2,304 (section 4.2.1, pp. 21–22). Section 4.3.2 and Table 1 report the base model comparable to DeepSeek-V4-Pro-Base on world knowledge with a third of the backbone parameters (p. 23–24).

The same split on a dense model

On a dense Llama-3-70B-shape model split 40 + 40 (the simulator’s illustrative proxy, 4× H100, roofline cost model), a prompt token touches 34.9B matmul parameters against 69.5B for a generated token: a ratio of 0.502, the extra above one half being the decoder’s K/V projections (0.67B). One prefill step for one prompt:

The ratio approaches one half as the prompt grows, because the 128-token replay and the LM head are fixed costs and causal attention grows with the square of the prompt in both halves alike. “Encoder only” is the prefill instance of the asymmetric deployment on slide 07, where the replay moves to the decode instance.

The KV-Cache Saving, Arithmetic Shown

CED saves compute; the KV-cache saving in DeepSeek-V4.1-Flash comes mostly from cross-layer sharing (CSA2), FP4 storage and the bounded replay. The headline numbers, all relative to DeepSeek-V4-Flash at the same sequence length:

So the “about 8×” that circulates is the persistent cache; the cache held in HBM shrank about 4×. The 1/8 is the product of two factors (section 3.2.1, p. 19):

no SWA KV in the persistent cache   ≈ 1/2     # SWA KV was "nearly half" of V4's persistent cache
global KV, CSA2 + FP4               ≈ 1/4
                                    ≈ 1/8

Reconstructing the 890 bytes (our arithmetic, from the report’s configuration)

FP4 entry:  512 channels × (4 bits E2M1) + 512/16 E4M3 scales × 8 bits
         =  256 B + 32 B = 288 B                       # 4.5 bits per value vs FP8's 8: "nearly halves"
KV-producing (Full-mode) layers:
  encoder: 3 groups of 6 CSA2 layers, 1 Full each, compression m = 2  → 3 × 1/2 entry per token
  decoder: 5 groups of 4, only group 1 starts in Full mode,      m = 1  → 1 entry per token
main KV  =  (3 × 1/2 + 1) × 288 B  =  2.5 × 288  =  720 B per token
the rest of the 890 B (≈ 170 B) is indexer K (128-dim, FP4 in V4), 2.5 entries per token

The paper states only the 890-byte total; the split above is a consistency check, not a quoted figure. It also shows how small CED’s own share is: the decoder contributes one KV-producing layer, projected from the encoder state, instead of twenty.

Serving It: Asymmetric Prefill and Decode Pools

DeepSeek deploys with Encoder–Prefill–Decode (EPD) disaggregation: vision encoding, prefill and decoding scale independently (section 3.2, p. 19). CED makes the prefill and decode pools unequal in a new way: they no longer run the same model.

What a prefill instance must hold, for the dense 70B-shape proxy (BF16 weights; KV room = 90% of HBM minus weights, in tokens of KV for all 80 layers):

Measured: Pool Split and Capacity in a Simulator

Six instances of 4× H100 (24 GPUs), split every way between a prefill and a decode pool; for each split, the highest Poisson request rate at which 90% of requests meet both SLOs (TPOT ≤ 25 ms; TTFT ≤ 5× the decoder-only model’s unloaded prefill time for the mean prompt), found by bisection on full simulator runs. Prompt and output lengths are lognormal (cv 0.5); the ratios stand in for agent loops and are illustrative.

Smaller prefill instances

With CED’s prefill instances on 2 H100s each (still 24 GPUs; 8,192 : 128 workload), the best splits sustain:

About the same as with 4-GPU instances: in a roofline, smaller instances add no capacity. Their value is that the encoder fits with room to spare (slide 07). The decoder-only model on 2-GPU instances is left out: its result there is limited by the simulator’s router, not by compute (results.md, section 18).

Source: sections 16–18 of results.md (--ced-encoder-layers, --ced-replay, --ced-replay-on). A dense proxy on a roofline: no MoE, no sparse attention, no prefix-cache hits, no KV compression, so the absolute numbers say nothing about DeepSeek’s model; the direction and rough size of the effect are the point. The simulator’s prefill router ignores work in flight, which slightly favours the model with shorter prefill steps.

Trade-offs and Open Questions

Relatives: YOCO, CLA and Cross-Layer Sharing

CED is “inspired by YOCO” (section 2.2, p. 9) and sits in a family of designs that stop every layer from keeping its own KV cache.

The difference that matters for serving: CLA and CSA2 shrink the cache but leave prefill compute alone; YOCO and CED shrink prefill compute by making the upper layers’ KV a function of a lower layer. CED keeps YOCO’s prefill saving while giving each decoder layer its own projection, and pays for keeping layer-wise sliding windows with the replay. Zamba’s single shared attention layer (Arch 05) is a cousin from the hybrid-SSM side.

What to Take Away

• CED is a decoder-only model with a split personality. One causal token stream; the bottom half is an encoder whose last state feeds every decoder layer’s global KV through that layer’s own projection.
• Prefill runs the encoder, decode runs everything. DeepSeek-V4.1-Flash activates 8B parameters per prompt token and 16B per generated token (pp. 1, 7, 22); prefill work drops from O(N·L) to about O(N·L/2).
• The decoder’s sliding windows need a replay. The last 128 prompt tokens run the decoder approximately; the report finds the error negligible and trains with it.
• The KV saving is mostly not CED. 890 bytes per token in HBM (about 1/4 of V4-Flash) comes from CSA2 sharing and FP4; the about-1/8 persistent cache adds the bounded replay’s removal of SWA KV.
• Asymmetric models want asymmetric pools. In the simulator, halving prefill cost raises capacity most where prompts dwarf outputs, and moving the replay to decode lets a prefill instance hold about half the weights.
• Quality is the open question. No isolated ablation is published; what is lost by computing the prompt’s upper-layer KV from a single mid-depth state remains to be measured.