NoviQNOVIQDocs
Launch app

Architecture

Sharded inference

A model too big for any single GPU is split into contiguous layer blocks and streamed across machines. No node ever holds the whole model.

The core idea

A transformer is a stack of layers. Split the stack into contiguous blocks, put one block on each machine, and stream activations through them in order:

prompt -> embed -> node0 (layers 0..k) -> node1 (k..2k) -> ... -> tail (final layers) -> sample -> next token

Each node holds only its block in VRAM, runs its block's forward pass, and ships the resulting hidden-state tensor to the next node. This is what makes it possible to serve a frontier-scale model on a mesh of consumer GPUs that individually could never load it.

SEALED WIRE - ACTIVATIONS ONLY, NO NODE HOLDS THE WHOLE MODELPrompttokensEmbed / headlayers 0-kWorker Alayers k-2kWorker Blayers 2k-3kTailfinal + logitsTokensampledTrusted node (pinned)Untrusted node (sees only rotated activations)
Contiguous layer blocks across machines. The head embeds token ids; middle blocks only forward hidden states; the tail produces logits.

The WAN performance law

On the open internet the bottleneck is not raw compute - it is the network round-trip between hops. A naive pipeline pays a full round-trip per token per hop and crawls. NoviQ inherits the set of techniques that make it fast anyway:

  • Speculative decoding - a small draft model proposes several tokens that the pipeline verifies in one pass.
  • Asynchronous pipelining - multiple tokens in flight across the ring at once, so hops overlap instead of blocking.
  • Direct-return ring - the tail returns the sampled token to the head directly, saving a full traversal.
  • In-region clustering - nodes that are close in latency are grouped so per-hop cost stays small.

Round-trip is the unit

The scheduler reasons in milliseconds of measured round-trip time, not teraflops. Pairwise RTT is measured at the application level over the same kind of persistent TCP connection the pipeline holds open during decode - not an ICMP ping - so the number reflects real per-hop activation cost.

The ModelRuntime interface

The engine is model-agnostic. Each node implements a small runtime contract for its assigned block:

  • The head stage embeds token ids into hidden states.
  • Middle stages take hidden states in and forward hidden states out - they never touch token ids.
  • The tail stage applies the final norm and language head, returning logits to sample from.
  • An is_trusted flag marks a node's trust tier so the scheduler can pin the leaky boundary blocks (embed and final layers) to trusted nodes and keep the privacy receipt's assumptions honest.

Trust tiers and placement

Not every node is equal. A node's trust tier - operator, staked, or volunteer - decides where it can sit in the pipeline. The embedding and final layers are the leakiest positions, so they are pinned to trusted nodes; untrusted volunteers only ever run middle blocks, and (with obfuscation on) only ever see rotated activations.

Measured throughput

18-25 tok/s

Prior-work throughput on four clustered consumer GPUs over WAN

no full model

Each node holds only its contiguous block in VRAM

256 MiB

Default maximum activation frame on the wire

Where speed sits in the plan

Correctness comes first, then real WAN, then the KV cache, then the privacy proof. Heavier speed optimizations and the permissionless economy come after the network can prove the two things its name promises: the output is right, and the input stayed hidden.
PreviousQuickstartNextThe sealed wire