Latency

Time delay between sending a request and receiving a response from a system, often measured in milliseconds.

Also known as: response time, round-trip time (RTT)

In plain English

Latency is how long you wait for something to happen after you ask for it, like a webpage loading or a message appearing. If latency is high the app feels slow or laggy; low latency feels instantaneous. Think of it like the wait time for a kettle to boil.

In practice

Latency directly affects user experience and productivity: slower model responses reduce engagement and interrupt workflows. It influences choices about which model to deploy, whether to batch requests, and what hardware to provision because lower latency often costs more. Teams set latency budgets and track tail metrics (p95/p99) to meet SLAs and UI expectations.

Under the hood

Latency comprises network round‑trip time, queuing, preprocessing (tokenisation, retrieval), and model compute/inference time, with the dominant component depending on deployment topology. Engineers reduce latency via model size reduction (distillation, quantisation), accelerator selection (GPU/TPU), batching strategies, caching, and asynchronous pipelines, and by optimising tokenisation and I/O. There is a fundamental trade-off between latency and throughput, so systems are tuned to meet target tail‑latency (p95/p99) SLAs.

Example

"We need to get p95 latency under 300 ms for single-turn chat, otherwise the product team says users will perceive it as too slow."

Related terms

Inference · GPU · Context Window · Token · Retrieval-Augmented Generation

•