Mixture of Experts

A model architecture that activates only a subset of specialised sub-models (experts) per input to improve efficiency and capacity.

Also known as: MoE, sparsely-gated MoE

In plain English

Think of it like a team of specialists where only the experts needed for a task are called in. This lets a system be powerful without using all its resources every time.

In practice

Mixture of experts models allow much larger effective model capacity while keeping per-request compute and cost lower, which can reduce inference spend for diverse workloads. They often require specialised routing, load-balancing and serving support, affecting latency, batching and deployment complexity when you adopt them.

Under the hood

Mixture of experts replaces dense layers (commonly feed-forward blocks in Transformers) with many expert subnetworks and a gating network that routes each token or input to a small subset of experts, enabling sparse activation. Training includes auxiliary load-balancing and capacity constraints to avoid expert collapse; inference requires routing, potential cross-device communication and careful batching, which complicates quantisation and latency optimisation.

Example

"Can we evaluate a mixture of experts model for production inference to reduce costs while keeping quality high?"

Related terms

Parameters · Transformer · Foundation Model · Inference

•