Continuity without a growing list.
- 64 heads with 128 × 128 recurrent matrices.
- KDA state is bounded; the entire hybrid model’s memory is not.
Q/K/V → causal conv → gated matrix update → normLANGUAGE / GLM-5.3-FLASHTwo paths.
Two paths.
One intelligence.
A hybrid vision-language model built around linear attention, sparse retrieval and mixture-of-experts computation. Inspect how GLM distributes work across its 45-layer backbone.
GLM-5.3-FLASH↗
OPEN THE 3D ARCHITECTURE ATLAS TOTAL PARAMETERS320B
ACTIVE / TOKEN18B
ATTENTIONKDA + KPool-DSA
CONFIGURED CONTEXT1,048,576 tokens
INSIDE THE MODEL / 3D ARCHITECTURE ATLASLinear continuity.
Linear continuity.
Selective recall.
Two attention mechanisms share a 45-layer backbone, with four residual streams running through the stack.
Drag to rotate · Arrow keys supported
MODULE 01 / RECURRENT MATRIX MEMORY
34
KIMI DELTA ATTENTION LAYERSContinuity without a growing list.
Thirty-four Kimi Delta Attention layers retain recurrent matrix state. Channelwise forget gates and per-head write gates update that state after short causal convolutions.
Q/K/V → causal conv → gated matrix update → normREADING THE DIAGRAM
Each slab includes attention and its own FFN. The 3× KDA → 1× KPool-DSA group repeats 11 times, followed by a final KDA layer. The side drawing is an FFN detail inset. The animation is explanatory; no model inference is executed.
BACKBONE MAP
45 layers. A deliberate pattern.
34 KDA11 KPool-DSA
Attention schedule shown. The first three FFNs are dense; the following 42 are MoE.
A CLOSER LOOK / GLMThe details behind
The details behind
the diagram.
Configuration, memory and computation—each with a specific role in the model.
Pool the index. Retain the history.
- 11 KPool-DSA layers; main sparse attention uses NoPE.
- Exact cached tensors depend on the serving backend.
weighted index pool → top blocks → token IDs → attentionEight paths through the expert bank.
- Selection uses sigmoid scores with an expert correction bias.
- Selected routing weights are normalized, then scaled by 2.5 before weighting expert outputs and adding the shared branch.
router → top 8 → weighted outputs + shared expertFour streams. A learned exchange.
- The four streams belong to one model, not four independent models.
- This residual mixer differs from Qwen’s identity branch bypass.
4 streams → read / sublayer / write + residual mixerFOLLOW THE RESEARCHGo straight
Go straight
to the source.
Developer papers, released configurations and reference implementations. Reviewed 10 September 2026.
G1
CHECKPOINT
G2GLM-5.3-Flash model card & configuration
Introduction; text_config, layer_types and index settings
IMPLEMENTATION
G3NVIDIA NeMo architecture documentation
KDA, KPool-DSA, mHC and dense/MoE schedule
SOURCE CODE
G4Transformers GLM5-Next implementation
Indexer, Attention, HyperConnection and TopkRouter
DEVELOPER ARTICLE
Z.AI GLM-5.3-Flash release
Architecture for extreme efficiency; IndexPool
WHAT THIS GUIDE COVERS
The model card links the earlier GLM-5 report. These exact layer details come from the released 5.3 Flash configuration and implementations.
The configured 1,048,576-token context is a model limit, not a Perchy service-level commitment.
MADE MEANINGFUL IN YOUR BUSINESSFrom capability
From capability
to possibility.
Engineering workflows
Explore coding, reasoning and agentic tasks with application-level tool permissions and oversight.
Long-context recall
Bring large document collections and working context into a carefully measured inference workflow.
Visual understanding
Combine text with supported image and video inputs in your business application.
Know your inference service scope.
Perchy provides managed AI inference through authenticated APIs. Customers do not receive SSH, root, operating-system or bare-metal access. The service does not provide model-training or fine-tuning functionality. Server administration remains with Perchy.
The configured context length is a model specification, not a service-level guarantee. Sparse token selection does not mean the model retains only the selected tokens in memory. Hardware, precision, vision inputs and runtime support determine the production envelope.
Read the model developer’s documentationKEEP EXPLORINGAnother model.
All models Another model.
Another way to compute.
One conversation.
A whole new trajectory.