Keep state. Write the difference.
- 48 value heads; 128 × 128 recurrent matrices per head.
- Short convolution state is retained alongside the matrix.
previous state → decay → delta write → query readLANGUAGE / QWEN3.8-FLASH-NEXTFast thinking.
Fast thinking.
Your direction.
A hybrid language and vision model that combines Gated DeltaNet, Qwen Sparse Attention and routed experts. Explore how the architecture balances compact recurrent state with selective access to a long context.
QWEN3.8-FLASH-NEXT↗
OPEN THE 3D ARCHITECTURE ATLAS BACKBONE125B parameters
ACTIVE / TOKEN6B parameters
ATTENTIONGDN + QSA
NATIVE CONTEXT262,144 tokens
INSIDE THE MODEL / 3D ARCHITECTURE ATLASA memory.
A memory.
And a way to recall.
Recurrent state carries continuity. Sparse attention brings selected history back into focus.
Drag to rotate · Arrow keys supported
MODULE 01 / RECURRENT MEMORY
36
GATED DELTANET LAYERSKeep state. Write the difference.
Gated DeltaNet updates a bounded recurrent matrix rather than appending every token to a KV list. It decays the previous state, estimates an existing association and writes a gated correction.
previous state → decay → delta write → query readREADING THE DIAGRAM
Each slab is a complete decoder block, including its own MoE FFN. The 3× GDN → 1× QSA pattern repeats 12 times. The side drawing is an FFN detail inset. The animation is explanatory; no model inference is executed.
BACKBONE MAP
48 layers. A deliberate pattern.
36 GDN12 QSA
Attention schedule shown. Every attention mixer is followed by an MoE; four residual branches run through the backbone.
A CLOSER LOOK / QWENThe details behind
The details behind
the diagram.
Configuration, memory and computation—each with a specific role in the model.
Select a block. Read its tokens.
- 12 QSA layers; each head has 256 dimensions.
- Selection reduces the attention read set, not the retained history.
index keys → pool 4 → top 512 blocks → sparse GQAA small selection. A large capacity.
- Channelwise read gates blend the four branches.
- Scalar write gates return the sublayer result; no Sinkhorn residual mixer.
4 branches → gated read → top-10 + shared → gated writeSome capacity starts with a lookup.
- The 125B backbone and 4B MTP module are separate allocations.
- This learned table is not a database of customer conversations.
token IDs → 2/3-gram hashes → table → context gateFOLLOW THE RESEARCHGo straight
Go straight
to the source.
Developer papers, released configurations and reference implementations. Reviewed 10 September 2026.
Q1
PAPER
Q2Qwen3.8-Flash-Next technical report
§§2.1–2.3 · GDN, QSA, residual gates and n-grams
CHECKPOINT
Q3Official model card & configuration
Model overview; linked config.json
SOURCE CODE
Q4Transformers reference implementation
QSAIndexer, Attention, GatedDeltaNet and GatedResidual
IMPLEMENTATION
NVIDIA NeMo architecture guide
N-gram embedding, branch collapse and expert routing
WHAT THIS GUIDE COVERS
125B describes the language backbone; additional n-gram and MTP allocations are listed separately.
262,144 tokens is the native context. Extended-context serving needs a validated configuration.
MADE MEANINGFUL IN YOUR BUSINESSFrom capability
From capability
to possibility.
Customer conversations
Bring product knowledge and conversational assistance closer to your customers.
Agentic workflows
Connect reasoning to approved tools, with application-level control and human oversight.
Knowledge in context
Connect retrieval to managed inference with agreed API access and data processing terms.
Know your inference service scope.
Perchy provides managed AI inference through authenticated APIs. Customers do not receive SSH, root, operating-system or bare-metal access. The service does not provide model-training or fine-tuning functionality. Server administration remains with Perchy.
API availability, model licensing, context length and serving configuration are confirmed for your workload. Parameter counts and context specifications describe the developer’s model, not a promise of Perchy service capacity. Extended context requires a separately validated configuration.
Read the model developer’s documentationKEEP EXPLORINGAnother model.
All models Another model.
Another way to compute.
One conversation.
A whole new trajectory.