Let the model decide what it needs to read.
A deep dive into Declarative Attention, a zero-shot protocol that turns a model’s own readable attention plan into a dynamic KV-cache mask.
Gemma-4-31B
Qwen-3.6-27B
sources evaluated
Use ← → or the buttons to move through the deck. Try the interactive demos.
PLAN
Attention is expensive because the model keeps rereading everything.
The problem
During autoregressive decoding, a long-context model repeatedly reads its KV cache. Even if only a small part matters for the next token, vanilla attention scans the full available context.
The idea
Ask the model to declare its attention scope in text: global to navigate, focus to inspect a named region, and local to reason over what it has already extracted.
The benefit
The inference engine parses those declarations and hides irrelevant KV-cache blocks from the attention kernel. The paper reports large token reductions with modest average accuracy drops on its benchmark suite.
What it isn’t
It isn’t retrieval. Retrieval decides what enters the context. Declarative Attention decides what the model reads among the material already present.
“Long-context inference isn’t only a memory problem. It’s a repeated memory-bandwidth problem during decoding.”
Read the prompt once
The input is processed in parallel. Declarative Attention targets what happens after this phase.
Generate one token at a time
Each step consults the accumulated KV cache. With a large context, the read can dominate latency.
Skip blocks that cannot help
If the model has already declared a narrow scope, the serving layer can read fewer KV blocks for that span.
Anywhere a long-lived context grows faster than relevance.
These are implementation hypotheses, not deployments evaluated by the paper.
Tool-heavy agent histories
Keep tool results in the episode, but let the model focus on the few responses relevant to the current decision.
Long research notebooks
Navigate hundreds of experiments globally, extract a few observations, then synthesise locally with explicit provenance.
Repository-scale QA
Use global mode to locate files, focus mode to inspect relevant modules, and local mode to compare extracted constraints.
Multi-document analysis
Address sections or tool-returned passages without repeatedly loading every prior document into every reasoning span.
Traceable workflows
The declared scope creates an auditable record of what the model says it consulted. That helps oversight, but doesn’t prove correctness.
KV-cache tiering
Stable span declarations could give an offloader time to prefetch a named segment before the next reasoning span.
Make the model’s attention plan explicit.
Declarative Attention joins a protocol, a parser, and a block-level serving intervention.
Three modes turn reasoning into an attention schedule.
Find two facts, then do the arithmetic locally.
Navigation is expensive. Extraction is targeted. Synthesis is cheap.
The model’s output is also a control stream. The tags are readable to a person, but they’re meaningful to the inference engine because they change which KV blocks remain visible.
Separate what must stay visible from what can be scoped.
Persistent scaffold
A short system instruction, the user question, the DA instruction, and the generated response stay available in every mode.
Addressable context
The long input is split into numbered “magic chunks”. The paper targets 2,048 tokens with a 2,560-token hard cap, cutting at semantic boundaries where possible.
The parser changes the KV block table at span boundaries.
Parse
Watch the output stream for opening and closing tags.
Round outward
Align kept spans to whole KV-cache blocks, typically 16-32 tokens.
Read less
Let the existing attention kernel consume the smaller block list.
Make the engine understand the declaration, not the kernel.
vLLM hook, block table, existing kernels
The authors extend vLLM through hooks on the attention metadata builder. They rewrite the request’s KV-cache block table at each decode step and leave kernels and scheduler unchanged.
Global attention layers only
Gemma’s sliding-window layers and Qwen’s Gated DeltaNet layers already have context-bounded state or reads, so the mask targets only the global-attention layers where long-context reads grow with N.
while generating:
token = model.next_token()
state = parser.observe(token)
kept = mask_for(state,
scaffold=True,
response_so_far=True)
kv_table = align_to_blocks(kept, block_size=16..32)
attention_metadata.kv_blocks = kv_tableThe paper’s strongest result is a cost-accuracy trade-off.
Average accuracy (%)
Across 15 sources. Higher is better.
The prompt creates overhead. The mask creates the saving.
Full attention
Baseline quality and cost. Every decode step reads the full context.
Same protocol, no intervention
The chunked prompt and extra reasoning steps are nearly lossless for accuracy, but the extra steps make full attention more expensive.
Protocol plus mask
More steps remain, but most steps read a narrow scope. The mask converts protocol overhead into net savings.
Normalised to the paper’s vanilla baseline for Gemma-4-31B. Qwen’s corresponding DA result is 69% of vanilla attended tokens.
Capability helps. Longer context amplifies the upside and the risk.
Backbone scale closes the accuracy gap
Within the evaluated families, relative accuracy improves as model size grows. The paper reports focus-parse success rising from 58% on Gemma-4-E4B to 99% on Gemma-4-31B.
Gemma family
Qwen family
Context length increases absolute savings
On pooled Gemma results, the paper reports about 1M fewer attended tokens per response in the shortest bin and about 21M fewer in the longest bin.
Declarative Attention is promising, but the zero-shot protocol is still brittle.
| Failure mode | What goes wrong | Design response |
|---|---|---|
| Small model adherence | Tags or chunk references fail to parse reliably. | Post-train the policy, validate references, and fall back to vanilla. |
| Bad segmentation | A table, count, or cross-chunk structure is split across focus regions. | Use structure-aware chunks and map-reduce over segments. |
| Long output | Per-segment output makes decode length scale with the document. | Route verbose work through focus and measure total cost, not per-step cost alone. |
| Global-mode burden | Navigation still reads the full context. | Give global mode a compact index or learned sparse scan. |
| Thinking incompatibility | The evaluated models failed to follow the protocol inside thinking traces. | Expose mode changes as native tool-like operations and test separately. |
Build the smallest useful version.
Start with a reversible prototype that can prove quality, parseability, and total serving cost before you touch production kernels.
Five steps from paper idea to an evaluable prototype.
Measure the whole system, not just the mask.
Does the answer stay right?
Compare vanilla, DA no-mask, and DA on the same examples. Include exact format failures and task-specific rubrics.
Does total work go down?
Log decode steps, attended KV positions, KV bytes, mode share, prefill, and wall time. Extra reasoning steps can reverse the apparent gain.
Can it fail safely?
Reject invalid chunk references, detect malformed tags, preserve a vanilla fallback, and make provenance visible in the final answer.
arms = ["vanilla", "da_no_mask", "da"]
metrics = [
"accuracy", "format_validity", "focus_parse_rate",
"decode_steps", "attended_tokens", "kv_bytes", "wall_time"
]
run_same_examples(arms, metrics)
report_by_task_and_context_length()Ship the experiment with guardrails attached.
Protocol layer
Serving layer
Data layer
Product layer
Attention becomes a control surface.
Readable plan
The model states where it intends to attend, making a usually hidden decision legible.
Reversible scope
Focus and local modes hide KV blocks without deleting the underlying context, so later spans can be restored.
New optimisation axis
Post-training could reward answers that are accurate, parseable, and economical with context reads.
The interesting shift isn’t “smarter retrieval”. It’s teaching the serving stack to trust a declared reasoning scope long enough to save a read.
Source, evidence, and a useful caveat.
Primary source
Ho et al., “Language Models Can Control Their Own Attention”, arXiv:2609.02737v1, September 2, 2026.
arXiv abstract and PDF ↗Direct PDF ↗How to interpret the numbers
The headline savings are attended-token reductions measured in the authors’ experiments. The wall-clock figures are roofline projections on a B200 under stated MFU and MBU assumptions, and exclude prefill.
This deck paraphrases the supplied paper and labels implementation suggestions where they go beyond the experiments.
End of deck. Press Overview to jump around, or use the arrow controls to revisit the demos.