A model architecture that activates only a subset of specialised sub-models (experts) per input to improve efficiency and capacity.
Also known as: MoE, sparsely-gated MoE
In plain English
Think of it like a team of specialists where only the experts needed for a task are called in. This lets a system be powerful without using all its resources every time.
In practice
Mixture of experts models allow much larger effective model capacity while keeping per-request compute and cost lower, which can reduce inference spend for diverse workloads. They often require specialised routing, load-balancing and serving support, affecting latency, batching and deployment complexity when you adopt them.
Under the hood
Mixture of experts replaces dense layers (commonly feed-forward blocks in Transformers) with many expert subnetworks and a gating network that routes each token or input to a small subset of experts, enabling sparse activation. Training includes auxiliary load-balancing and capacity constraints to avoid expert collapse; inference requires routing, potential cross-device communication and careful batching, which complicates quantisation and latency optimisation.
Example
"Can we evaluate a mixture of experts model for production inference to reduce costs while keeping quality high?"