Inference serving optimisation

The fraction of draft tokens that the large model agrees with during speculative decoding. A higher rate means more tokens per large-model step and greater speed-up; a low rate wastes the draft's work and may even slow generation.

Example. If the draft proposes 5 tokens per round and on average 3.5 are accepted, the acceptance rate is 70 percent.

Related: Speculative decoding, Draft model, Verification step, Token throughput

Lowering the precision of the intermediate values flowing between layers, not just the stored weights. It allows the matrix multiplications themselves to run in fast integer or FP8 units, but activations contain occasional very large values, so it is harder to do without hurting quality.

Example. With 8-bit weights and 8-bit activations, a matrix multiplication can use the GPU's INT8 tensor cores instead of FP16 ones.

Related: Weight-only quantisation, Outlier features, SmoothQuant, INT8

A communication operation where every GPU contributes a tensor, the tensors are combined (usually summed) and every GPU receives the combined result. Tensor-parallel inference performs it after layers, so its speed limits multi-GPU scaling.

Example. Four GPUs each hold a partial output of a layer and use it to obtain the full sum before the next layer.

Related: Tensor parallelism, NVLink, Interconnect bandwidth, NCCL

The number of arithmetic operations performed per byte moved from memory. Low values mean a task is limited by memory bandwidth and high values mean it is limited by compute; batching raises it because weights are read once for many tokens.

Example. Batch size 1 decoding does about 1 multiply-add per weight read; batch size 64 does about 64 with the same reads.

Related: Roofline model, Memory-bound, Compute-bound, Batch size

A scheme that uses both a scale and a zero-point, so the integer range can be shifted to fit data that is not centred on zero. It represents skewed ranges more precisely, at the cost of extra computation to handle the offset.

Example. ReLU outputs lie in 0 to 6, so an asymmetric 8-bit mapping uses all 256 codes for that range.

Related: Symmetric quantisation, Zero-point, Scale factor, Activation quantisation

Automatically adding or removing serving replicas as traffic changes, based on signals such as request rate, queue length or GPU use. It saves money off-peak but must handle slow start-up, since loading a large model onto a GPU can take minutes.

Example. Replicas scale from 2 to 6 during the evening exam-results rush and back down at night.

Related: Cold start, Queueing delay, Model serving, Cost per token

Activation-aware Weight Quantisation, a post-training method that looks at activations to find the small share of weights that matter most and protects them by scaling before quantising to 4 bits. It is fast, needs little calibration data and is widely supported by serving engines.

Example. An AWQ 4-bit copy of a 7B model is downloaded from a model hub and loaded straight into a serving engine.

Related: GPTQ, Weight-only quantisation, INT4, Post-training quantisation

Running a model over a large stored set of inputs in one scheduled job, with no user waiting for each answer. Because latency does not matter, it can use big batches, cheap spot hardware and off-peak hours to minimise cost.

Example. A bank scores 10 million customer records overnight in one batch job.

Related: Online inference, Spot instance, Throughput, Batch size

A lookup structure mapping a request's logical token positions to the physical memory blocks that hold its KV cache. It lets attention kernels find scattered blocks and lets several requests share identical blocks safely.

Example. Sequence A's table says logical blocks 0, 1, 2 live in physical blocks 7, 3 and 12.

Related: PagedAttention, KV cache, Prefix caching, Copy-on-write

A load-balancing policy that sends a request to the replica most likely to already hold a matching prompt prefix in its KV cache, so prefill work is reused instead of repeated. It must be balanced against keeping load even.

Example. All requests starting with the same 2,000-token system prompt are steered to the same two replicas.

Related: Load balancing, Prefix caching, KV cache, Request scheduler

A small, representative sample of inputs passed through a model to observe the range of its activations, so quantisation scales can be chosen. Poorly matched calibration data leads to bad scales and worse accuracy on real traffic.

Example. An Indian customer-support bot calibrates on 256 real, anonymised Hinglish chats rather than on English Wikipedia text.

Related: Post-training quantisation, Scale factor, Activation quantisation, GPTQ

Splitting a long prompt's processing into smaller pieces that are mixed into batches together with ongoing decoding steps. It prevents one huge prompt from stalling every other user's token stream, smoothing latency at a small cost to the long prompt's own speed.

Example. A 32,000-token prompt is processed 512 tokens at a time, and each chunk shares its step with decoding for other users.

Related: Prefill and decode, Inter-token latency, Continuous batching, Prefill-decode disaggregation

The delay when a new serving instance must be created and its model loaded before it can answer. For large models, downloading weights and moving them into GPU memory can take from seconds to many minutes.

Example. A scaled-to-zero endpoint takes about 90 seconds to answer the first request after idling.

Related: Autoscaling, Model serving, Model loading, Time to first token

A workload whose speed is limited by how many arithmetic operations the chip can perform, not by data movement. Processing a long prompt in one pass and large-batch matrix multiplication usually fall in this category.

Example. Prefilling 4,000 prompt tokens keeps the GPU's arithmetic units near full use.

Related: Memory-bound, Arithmetic intensity, Roofline model, Prefill and decode

A scheduling method for language-model servers where finished requests leave the running batch immediately and waiting requests join at the next generation step, instead of waiting for the whole batch to finish. It keeps the GPU busy and greatly raises throughput when output lengths vary.

Example. If one request in a batch of 32 ends after 20 tokens and another needs 500, a new request takes the freed slot after token 20 rather than after token 500.

Related: Static batching, Iteration-level scheduling, vLLM, Throughput

The price of processing one token, either the fee charged by an API provider or your own hardware cost divided by tokens produced. It is normally quoted per million tokens, with input and output tokens priced differently.

Example. If a GPU costs Rs 200 per hour and generates 2 million tokens in that hour, the cost is about Rs 100 per million tokens.

Related: Tokens per second, GPU hours, Total cost of ownership, Throughput

Keeping part of a model's weights or cache in ordinary computer memory and moving it to the GPU only when needed. It lets a model larger than GPU memory run, but transfers are slow, so speed drops sharply.

Example. A 30B model on a 16 GB GPU keeps some layers in RAM and streams them in per token, running at a few tokens per second.

Related: PCIe, VRAM, Swap space, Out of memory

A CUDA feature that records a whole sequence of GPU operations once and replays it with a single launch call. It removes per-kernel launch overhead, which matters in token-by-token decoding where each step contains many tiny kernels.

Example. Capturing the decode step as a graph cuts CPU launch overhead and lowers per-token time.

Related: GPU kernel, CUDA, Kernel fusion, Time per output token

The small, fast model in speculative decoding that guesses upcoming tokens for the large model to verify. It should resemble the large model's behaviour, ideally sharing its tokeniser, so that many guesses are accepted.

Example. A 68M-parameter model from the same family drafts tokens for a 7B target.

Related: Speculative decoding, Target model, Acceptance rate, Knowledge distillation

Collecting requests that arrive within a short time window and running them as one batch, up to a size limit. It trades a few milliseconds of waiting for much better hardware utilisation, and is a standard feature of model servers such as Triton.

Example. A server waits up to 5 ms to gather up to 16 image-classification requests before launching one GPU call.

Related: Continuous batching, Static batching, Triton Inference Server, Latency

A speculative decoding method that drafts using the large model's own hidden features through a lightweight extra module, rather than a separate small language model. Working at feature level gives higher acceptance rates than many token-level drafters.

Example. A small head takes the target's last hidden state and proposes the next few tokens.

Related: Speculative decoding, Medusa decoding, Draft model, Acceptance rate

Running a trained model on the device that generates the data, such as a phone, camera or embedded board, rather than in a data centre. It reduces latency and bandwidth and protects privacy, but the model must fit tight limits on memory, power and compute.

Example. A retail camera detects empty shelves on its own chip and sends only alerts, not video, to the cloud.

Related: Local inference, Quantisation, Model compression, Latency

An 8-bit floating-point format, usually in two variants: E4M3 with more precision and E5M2 with more range. Recent GPUs multiply FP8 numbers natively, giving about twice the arithmetic speed of 16-bit while keeping the dynamic range that integers lack.

Example. An H100-class GPU can run a transformer's matrix multiplications in FP8 with per-tensor scale factors, saving memory and time.

Related: INT8, BF16, Quantisation, Tensor core

A single-file binary format for storing quantised language models, used by llama.cpp and tools built on it. It packs weights, tokeniser and metadata together and supports many quantisation levels, so models can be run easily on laptops and CPUs.

Example. A file named model-Q4_K_M.gguf holds a 4-bit-ish version of a model that runs on a MacBook without a dedicated GPU.

Related: llama.cpp, Quantisation, Edge inference, CPU inference

The rate of requests completed while still meeting stated latency targets, rather than the raw count of all requests. A server can show high throughput yet low goodput if most responses arrive too late to be useful.

Example. An engine finishes 100 requests per second but only 60 meet the 500 ms first-token target, giving a goodput of 60 per second.

Related: Throughput, Service level objective, Time to first token, Tail latency

A post-training method that quantises a model's weights layer by layer to 3 or 4 bits, using second-order information about the layer's error to adjust the remaining weights after each rounding step. It keeps accuracy well and needs only a small calibration set.

Example. A 13B model is compressed to 4-bit with GPTQ in under an hour on one GPU and then served from a GPU with 12 GB of memory.

Related: Post-training quantisation, AWQ, INT4, Calibration dataset

The unit in which cloud GPUs are billed: one GPU rented for one hour. Estimating serving cost means counting how many GPU hours a workload needs and multiplying by the hourly rate.

Example. Eight GPUs running for 3 hours consume 24 GPU hours.

Related: Cost per token, Spot instance, Reserved capacity, Total cost of ownership

A function that runs on the GPU, executed in parallel by thousands of threads. A model's forward pass is a sequence of kernels; launching many tiny ones wastes time, which motivates combining them.

Example. A softmax and a matrix multiplication are two separate kernels unless fused.

Related: Kernel fusion, CUDA, Warp, CUDA graphs

The share of time a GPU reports being busy. It is a coarse indicator: a GPU can show 100 percent utilisation while its arithmetic units are mostly idle waiting for memory, so it does not by itself show efficient use.

Example. nvidia-smi shows 95 percent utilisation while the model reaches only 30 percent of the chip's peak arithmetic rate.

Related: Model flops utilisation, Memory-bound, nvidia-smi, Throughput

Splitting weights into small blocks, commonly 32 to 128 values, each with its own scale (and sometimes zero-point). The finer granularity keeps 4-bit models accurate, at the cost of storing extra scale values.

Example. With group size 128 and 16-bit scales, 4-bit weights use about 4.125 bits each once scales are counted.

Related: Per-channel quantisation, INT4, GPTQ, AWQ

A type of stacked memory placed right next to a GPU chip, offering much higher data rates than ordinary graphics memory. Its size and speed largely determine which models fit and how fast they can generate tokens.

Example. A data-centre GPU with 80 GB of it can hold the FP16 weights of a 30B model but not a 70B one.

Related: VRAM, Memory bandwidth, Memory-bound, GPU

The software that loads a trained model and executes it efficiently to produce predictions, handling memory, batching and hardware kernels. Examples include vLLM, TensorRT-LLM, ONNX Runtime and llama.cpp, and the choice affects speed and cost.

Example. A team benchmarks two engines on the same model and picks the one giving lower cost per million tokens.

Related: vLLM, TensorRT-LLM, ONNX Runtime, Model serving

A 4-bit whole-number format with only 16 possible values. It cuts weight memory to a quarter of 16-bit storage, but needs finer grouping of scales and careful algorithms to keep model accuracy acceptable.

Example. Llama-style 7B checkpoints in 4-bit form are commonly around 4 GB, small enough to run on a laptop.

Related: Quantisation, INT8, Group-wise quantisation, GPTQ

An 8-bit whole-number format with 256 possible values. Model weights or activations are mapped into it using a scale factor, halving memory compared with 16-bit floats and enabling fast integer arithmetic on many GPUs, CPUs and phone chips.

Example. A weight of 0.37 with scale 0.01 is stored as the integer 37 and recovered as 37 x 0.01 when needed.

Related: Quantisation, INT4, FP8, Scale factor

The gap between consecutive generated tokens as seen by the client. Unlike the average per-token time, it exposes stalls and jitter, for example when a long prefill from another user briefly blocks the batch.

Example. Most gaps are 30 ms, but one arrives after 400 ms because a large prompt was scheduled at that step.

Related: Time per output token, Time to first token, Chunked prefill, Tail latency

The rate at which data can be moved between GPUs or between servers. When a model is spread over several devices, this rate can become the bottleneck that limits how much extra hardware helps.

Example. NVLink offers many times the bandwidth of PCIe, so splitting one model across NVLink-connected GPUs scales better.

Related: NVLink, PCIe, All-reduce, Tensor parallelism

Making scheduling decisions at every single token-generation step rather than once per request. The scheduler can add, remove or pause sequences between steps, which is the idea behind continuous batching in systems such as Orca.

Example. After each decode step the scheduler checks memory and either admits a queued request or pauses a low-priority one.

Related: Continuous batching, Request scheduler, Preemption, Time per output token

Combining several consecutive GPU operations into one kernel, so intermediate results stay in fast on-chip memory instead of being written to and re-read from main GPU memory. It cuts memory traffic and launch overhead.

Example. Fusing a bias add, activation and dropout into one kernel avoids two round trips through GPU memory.

Related: GPU kernel, FlashAttention, torch.compile, Memory-bound

Choosing which stored keys and values to drop when cache memory is full. Policies range from removing whole least-recently-used requests to discarding individual tokens judged unimportant, trading memory saved against risk of losing needed context.

Example. A long chat keeps its first few tokens and latest 2,000 tokens in cache while dropping the middle.

Related: KV cache, Preemption, Sliding window attention, RadixAttention

Storing the cached keys and values in lower precision such as 8-bit or 4-bit. Because the cache grows with context length and batch size, shrinking it lets a GPU serve longer prompts or more users, with a small accuracy risk.

Example. Moving the cache from FP16 to FP8 halves its memory, so a server that held 20 concurrent 8K-token chats can hold about 40.

Related: KV cache, Quantisation, FP8, Context length

An open-source C/C++ inference engine for running language models efficiently on ordinary hardware such as laptops, phones and CPUs, with optional GPU offload. It popularised aggressive quantisation and the GGUF file format for local use.

Example. A developer runs a 4-bit 8B model through llama.cpp on a laptop with no discrete GPU to prototype offline.

Related: GGUF, Quantisation, CPU inference, Edge inference

A method for 8-bit matrix multiplication in large language models that splits out the few outlier dimensions and computes them in 16-bit, while everything else uses INT8. It halves memory with little quality change, but the mixed computation can be slower.

Example. Loading a model with the bitsandbytes library's 8-bit option applies it, letting a 30B model fit on one large GPU.

Related: Outlier features, INT8, Activation quantisation, Quantisation

Spreading incoming requests over several identical servers so no one is overloaded and failed servers are bypassed. For LLM serving, smarter policies consider queue length and cache contents, not just simple rotation.

Example. A router sends a new chat turn to the replica that already holds that conversation's cached prefix.

Related: Autoscaling, Model serving, Cache-aware routing, Queueing delay

Running a model on your own laptop, phone or server instead of calling a hosted API. It keeps data on your device, works offline and has no per-token fee, but you are limited by your hardware and must manage the model yourself.

Example. A hospital in Pune runs a small model on an in-house server so patient notes never leave its network.

Related: Edge inference, Ollama, Quantisation, Model serving

Replacing a large weight matrix with the product of two much thinner matrices that approximate it. This cuts the number of parameters and multiplications when the original matrix carries less independent information than its size suggests.

Example. A 4096 x 4096 matrix (16.8 million values) is approximated by 4096 x 64 and 64 x 4096 matrices, about 0.5 million values.

Related: Model compression, Singular value decomposition, LoRA, Pruning

A speculative approach that adds several extra prediction heads to the main model, each guessing a token further ahead, so no separate draft model is needed. Candidate continuations are checked together using tree attention.

Example. Four extra heads propose the tokens at positions +1 to +4, and the verified ones are kept.

Related: Speculative decoding, Tree attention, EAGLE, Multi-token prediction

Wasted memory that exists in small gaps or reserved-but-unused space, so that a large allocation fails even though total free memory looks sufficient. In LLM serving it appears when each request reserves space for its maximum possible length.

Example. Reserving 4,096 slots for a request that ends at 300 leaves over 90 percent of its reservation unused.

Related: PagedAttention, KV cache, Out of memory, Block table

A workload whose speed is limited by how quickly data can be moved from memory to the compute units, not by arithmetic. Generating one token at a time is typically like this, because every step must read all model weights.

Example. A 70B FP16 model read at 2 TB/s cannot generate a single sequence faster than about 14 tokens per second.

Related: Compute-bound, Memory bandwidth, Arithmetic intensity, Roofline model

Trying a cheap model first and escalating to a stronger one only if the cheap model's answer seems unreliable, judged by a confidence score or a checker. It saves cost when most requests are easy.

Example. A small model answers first; if its confidence is below 0.7, the request is re-run on the large model.

Related: Model routing, Small language model, Cost per token, Confidence score

The family of techniques that make a trained model smaller or cheaper to run, including quantisation, pruning, distillation and low-rank factorisation. The goal is to keep useful accuracy while cutting memory, compute or energy.

Example. A team combines 4-bit quantisation and pruning to shrink an image model from 400 MB to 90 MB for a budget Android phone.

Related: Quantisation, Pruning, Knowledge distillation, Low-rank factorisation

Making a trained model available to applications, typically as a network API that receives inputs, runs the model and returns outputs reliably at the required speed and scale. It includes deployment, batching, scaling and monitoring.

Example. A fraud model is wrapped in an HTTP service that answers within 50 ms for each payment request.

Related: Inference engine, Autoscaling, Latency, Triton Inference Server

NVIDIA's library of fast communication routines, such as all-reduce, for exchanging data between GPUs inside one machine and across machines. Multi-GPU serving and training frameworks rely on it to combine partial results quickly.

Example. A multi-GPU vLLM deployment uses it under the hood for collective operations.

Related: All-reduce, NVLink, Tensor parallelism, CUDA

A command-line tool from NVIDIA that shows the state of GPUs on a machine, including memory used, utilisation, temperature, power and running processes. It is the first thing engineers check when debugging out-of-memory errors or idle GPUs.

Example. Running nvidia-smi shows 38 GB of 40 GB in use, explaining why a second model failed to load.

Related: GPU utilisation, VRAM, CUDA, Out of memory

A high-speed direct link between NVIDIA GPUs, much faster than the standard PCIe slot. It matters when a model is split across several GPUs, because they must exchange partial results at every layer.

Example. Eight GPUs in one server connected by NVLink can run tensor-parallel inference with little communication delay.

Related: Tensor parallelism, PCIe, Interconnect bandwidth, All-reduce

A tool for downloading and running language models locally with a simple command line and local API, built on efficient engines such as llama.cpp. It handles model files, quantised variants and a background server so developers can test models without setting up infrastructure.

Example. Typing ollama run with a model name pulls the weights and opens a chat on your own machine.

Related: llama.cpp, GGUF, Local inference, Model serving

Running a model in response to individual live requests, where each caller waits for the answer. It demands low and predictable latency, so batches are small and hardware is kept ready, which usually raises cost per prediction.

Example. A payment app must approve or block a transaction within 100 ms.

Related: Batch inference, Latency, Model serving, Autoscaling

Open Neural Network Exchange, an open file format for representing a trained model as a graph of standard operations, so a model built in one framework can run in different runtimes and on different hardware.

Example. A PyTorch model is exported to a .onnx file and then served with ONNX Runtime on a CPU-only server.

Related: ONNX Runtime, TensorRT, Model export, Inference engine

A cross-platform engine from Microsoft for running models saved in the ONNX format. It applies graph optimisations and can use different hardware back ends, such as CPUs, GPUs and mobile chips, through pluggable execution providers.

Example. A company runs a scikit-learn and a PyTorch model through one ONNX Runtime service on CPU.

Related: ONNX, TensorRT, Inference engine, Execution provider

An HTTP interface that copies the request and response format of OpenAI's chat and completion endpoints. Many open-source servers offer it, so applications written for one provider can switch by changing only the base URL and model name.

Example. Pointing an existing client's base URL at a self-hosted vLLM server lets the same code call an open model.

Related: vLLM, Model serving, Streaming response, Inference engine

A few hidden dimensions in large language models whose activations are far bigger than the rest. They make naive low-precision quantisation of activations lose accuracy, and much research on quantisation exists to handle them.

Example. One channel with values near 100 while others sit near 1 forces a per-tensor INT8 scale so coarse that ordinary channels lose most of their precision.

Related: Activation quantisation, SmoothQuant, LLM.int8(), Per-channel quantisation

A technique that stores a request's KV cache in fixed-size blocks that need not sit side by side in GPU memory, tracked with a table like an operating system's page table. It removes fragmentation and waste, letting a server hold many more concurrent requests.

Example. With 16-token blocks, a 300-token conversation uses 19 blocks that can be scattered anywhere in memory.

Related: vLLM, KV cache, Memory fragmentation, Block table

Giving each output channel (row) of a weight matrix its own scale factor instead of one for the whole tensor. Because different channels have different magnitudes, this reduces rounding error noticeably and is a common default for weights.

Example. A layer with 4096 output channels stores 4096 scales alongside its INT8 weights.

Related: Per-tensor quantisation, Group-wise quantisation, Scale factor, INT8

Using a single scale factor for an entire tensor. It is simple and fast, but one large value can force a coarse scale that crushes the many small values, so accuracy is usually lower than with finer groupings.

Example. One outlier of 60 in a tensor of mostly values below 1 makes an INT8 scale of about 0.47, rounding most small weights to zero.

Related: Per-channel quantisation, Group-wise quantisation, Outlier features, Scale factor

A latency figure that says what share of requests finished within a given time, such as p50 for the median or p95 and p99 for slower cases. Percentiles describe real user experience better than an average.

Example. If p95 is 1.2 s, then 95 percent of requests completed in 1.2 seconds or less.

Related: Tail latency, Latency, Service level objective, Throughput

Converting an already trained model to lower precision without retraining it. A small calibration dataset is often run through the model to pick good scaling values. It is quick and cheap, but at very low bit widths accuracy can drop more than with methods that train with quantisation in mind.

Example. A team takes a finished chatbot model, runs 512 sample prompts to calibrate, and exports an 8-bit version in an afternoon.

Related: Quantisation, Calibration dataset, Quantisation-aware training, GPTQ

Pausing a running request so its resources can be given to others, usually when GPU memory for the KV cache runs out. The paused request's cache is either swapped to CPU memory or discarded and recomputed later, costing extra time.

Example. When memory fills up, the engine evicts the newest request's cache and re-runs its prompt once space frees.

Related: KV cache eviction, Request scheduler, PagedAttention, Swap space

Running prompt processing and token generation on separate groups of GPUs. Prefill is compute-heavy and decode is memory-heavy, so separating them lets each be tuned and scaled independently and avoids interference, at the cost of transferring the KV cache between machines.

Example. Four GPUs handle prompts and eight handle generation, with the cache sent over a fast interconnect after each prompt.

Related: Prefill and decode, Chunked prefill, KV cache, Time to first token

Reusing the stored KV cache for a prompt beginning that has been seen before, so the model does not recompute it. It cuts time to first token and cost when many requests share a long system prompt, document or conversation history.

Example. A support bot with a 3,000-token policy prompt computes it once; every later request reuses its cache and only processes the customer's new question.

Related: KV cache, Prompt caching, RadixAttention, Time to first token

A simple speculative technique that needs no draft model: it finds the latest words of the output in the prompt and proposes the words that followed them there as guesses. It works well when answers copy from the input, such as summaries or code edits.

Example. When rewriting a document with small edits, most output tokens are copied runs from the prompt and are accepted in bulk.

Related: Speculative decoding, N-gram speculation, Acceptance rate, Draft model

Removing weights, neurons, attention heads or whole layers that contribute little to a model's output, then usually fine-tuning to recover accuracy. It shrinks the model and, if the removed parts are structured, can speed up inference.

Example. Dropping 30 percent of the smallest-magnitude weights in a vision model and fine-tuning for one epoch recovers most of the original accuracy.

Related: Structured pruning, Unstructured pruning, Sparsity, Model compression

Storing a model's numbers in fewer bits, for example 8-bit or 4-bit integers instead of 16-bit floats. The weights take less memory and move faster through the chip, so the model runs on cheaper hardware, usually with a small loss in output quality that must be measured.

Example. A 7-billion-parameter model needs about 14 GB in 16-bit form but roughly 3.5 GB with 4-bit weights, so it fits on a 8 GB consumer graphics card.

Related: Post-training quantisation, INT8, INT4, Dequantisation

Training or fine-tuning a model while simulating low-precision arithmetic in the forward pass, so the weights learn to tolerate rounding error. It typically preserves more accuracy at very low bit widths than converting afterwards, but it costs extra training compute.

Example. A speech model is fine-tuned for a few epochs with fake 4-bit rounding inserted, then deployed on a phone chip that only supports integer maths.

Related: Post-training quantisation, Fake quantisation, Straight-through estimator, Quantisation

The time a request waits before processing starts because the server is busy. It grows sharply as load approaches capacity and is often the largest part of user-visible latency at peak times.

Example. At 90 percent utilisation, requests wait behind others for several seconds even though each needs only 200 ms of compute.

Related: Tail latency, Request scheduler, Autoscaling, Throughput

A technique in the SGLang serving system that keeps KV caches in a radix tree keyed by token sequence, so requests with common prefixes automatically share cached computation and least-recently-used branches are evicted when memory is short.

Example. Two agent runs that share the same tool instructions and first three turns reuse the same tree branch.

Related: Prefix caching, SGLang, KV cache eviction, PagedAttention

The component of a serving engine that decides which waiting requests run next, how many tokens are processed per step and when to pause or drop work. Its policy affects fairness, latency and total throughput.

Example. A scheduler admits requests first-come-first-served but lets short prompts jump ahead if the GPU has spare room.

Related: Continuous batching, Preemption, Queueing delay, Prefill-decode disaggregation

Cloud computing capacity, such as GPUs, bought or committed for a fixed period at a lower hourly price than on-demand. It lowers cost for steady traffic but leaves you paying for idle hardware if demand falls.

Example. A company commits to 4 GPUs for a year for baseline traffic and rents extra on demand for peaks.

Related: Spot instance, GPU hours, Autoscaling, Total cost of ownership

A chart-based way to reason about performance: achievable speed is capped either by peak compute or by memory bandwidth multiplied by arithmetic intensity, whichever is lower. It shows whether a kernel should be optimised for data movement or for arithmetic.

Example. A kernel with intensity 10 FLOPs per byte on a 2 TB/s chip is capped at 20 TFLOPs, however high the chip's peak compute is.

Related: Arithmetic intensity, Memory-bound, Compute-bound, Memory bandwidth

The number that maps between real values and their low-precision integers in quantisation. A real value is divided by the scale and rounded to store it, and multiplied by the scale to recover it. Its choice sets the balance between rounding error and clipping.

Example. If the largest weight is 2.54 and INT8 tops out at 127, the scale is 2.54 / 127 = 0.02.

Related: Quantisation, Zero-point, Calibration dataset, Per-channel quantisation

Storing past answers and returning one when a new query means the same as an earlier query, judged by embedding similarity rather than exact text. It saves whole model calls, but risks returning a wrong answer if similarity is judged too loosely.

Example. 'How do I reset my password?' and 'password reset steps' hit the same stored answer when similarity exceeds a chosen threshold.

Related: Prompt caching, Embedding similarity, Cache hit, Cost per token

A pattern where in every small group of weights a fixed number must be zero, such as 2 out of every 4 values. Because the pattern is regular, recent NVIDIA GPUs can skip the zeros in hardware and roughly double matrix-multiplication throughput.

Example. A 2:4 sparse layer stores two non-zero weights per block of four plus small index bits, halving weight storage.

Related: Sparsity, Structured pruning, Tensor core, Pruning

A measurable performance target that a service commits to, such as 99 percent of requests receiving a first token within 800 ms. It guides capacity planning and is used to judge whether an inference setup is good enough.

Example. The team sets an objective of p95 time to first token below 1 second and a monthly availability of 99.9 percent.

Related: Tail latency, Goodput, Autoscaling, Percentile latency

An open-source framework and serving runtime for language models that pairs a programming interface for multi-step prompting with an efficient backend featuring prefix-sharing caches and structured output support. It targets complex workflows such as agents and multi-call pipelines.

Example. An application that calls a model five times with a shared context runs faster because the runtime reuses the shared cache.

Related: RadixAttention, vLLM, Prefix caching, Structured decoding

A small, fast, programmable memory inside each GPU processing block that threads in the same block can use to exchange data. Efficient kernels load tiles of data into it once and reuse them many times instead of re-reading slow main memory.

Example. A matrix-multiplication kernel loads a 64 x 64 tile into it and uses each value 64 times.

Related: Streaming multiprocessor, High bandwidth memory, GPU kernel, Kernel fusion

A language model with relatively few parameters, from tens of millions to a few billion, designed to run cheaply on modest hardware or devices. It is often good enough for narrow tasks and is much cheaper and faster to serve.

Example. A 3B-parameter model fine-tuned on Hindi customer emails classifies them on a single small GPU.

Related: Model routing, Knowledge distillation, Edge inference, Quantisation

A method that makes activation quantisation easier by shifting difficulty from activations to weights: it divides outlier-heavy activation channels by a factor and multiplies the matching weights by the same factor, leaving the output unchanged while flattening the ranges.

Example. Applying it lets a large language model run all its matrix multiplications in INT8 with almost no accuracy loss.

Related: Activation quantisation, Outlier features, INT8, Post-training quantisation

The fraction of values in a model or tensor that are exactly zero. High sparsity can save memory and compute if hardware or software skips the zeros, which is why it matters for pruning and for mixture-of-experts style designs.

Example. A weight matrix with 800 zeros among 1,000 entries has 80 percent sparsity.

Related: Pruning, Semi-structured sparsity, Unstructured pruning, Model compression

A way to speed up text generation without changing the output distribution. A small, fast draft model proposes several next tokens and the large model checks them all in one pass, keeping the correct ones. When the draft is often right, several tokens are produced for the price of one large-model step.

Example. A 1B draft model proposes 4 tokens; the 70B model accepts 3 of them in one verification pass, so 4 tokens emerge from a single big-model step.

Related: Draft model, Acceptance rate, Verification step, Medusa decoding

A cloud machine rented at a steep discount on the condition that the provider can take it back at short notice. It suits fault-tolerant or batch inference jobs but is risky for latency-sensitive endpoints.

Example. A nightly job that embeds 5 million documents runs on spot GPUs and resumes from a checkpoint if one is reclaimed.

Related: Reserved capacity, GPU hours, Batch inference, Autoscaling

Grouping several requests and running them together until every one has finished. It is easy to implement, but short requests must wait for the longest one and the GPU sits partly idle, so it wastes capacity for text generation.

Example. Eight prompts are batched; seven finish in 30 tokens but the eighth needs 400, so seven slots idle for most of the run.

Related: Continuous batching, Batch size, Padding, Throughput

One of the many parallel processing blocks inside a GPU, holding arithmetic cores, registers and fast local memory. Work is split into thread blocks that are assigned to these blocks, so a chip's count of them gives a rough sense of its parallel capacity.

Example. An A100 GPU has 108 of them, each running many thread blocks at once.

Related: CUDA, Warp, Shared memory, Tensor core

Removing whole groups of parameters, such as entire channels, attention heads or layers, so the resulting model is genuinely smaller and runs faster on ordinary hardware. It is less flexible than removing single weights but gives real speedups.

Example. Removing 8 of 32 attention heads from every layer produces a smaller dense model that needs no special sparse kernels.

Related: Pruning, Unstructured pruning, Sparsity, Model compression

A scheme that maps real values onto integers using only a scale, with zero mapped to integer zero. The range is centred, which simplifies arithmetic and speeds up kernels, but wastes some codes if the data is lopsided.

Example. For weights between -3 and 3 in INT8, the scale is 3 / 127 and zero stays exactly 0.

Related: Asymmetric quantisation, Scale factor, Zero-point, INT8

The response time experienced by the slowest requests, usually reported as the 95th or 99th percentile. Averages hide it, but it drives user frustration and service-level agreements, and heavy load or long prompts inflate it.

Example. A service with a median of 0.8 s and a p99 of 6 s means 1 request in 100 waits at least six seconds.

Related: Latency, Percentile latency, Service level objective, Queueing delay

The large, high-quality model whose outputs are the ones actually wanted, in setups where a smaller helper model proposes text first. Its verification decides which proposed tokens are accepted, so final quality equals what it alone would produce.

Example. A 70B model acts as the target while a 1B model drafts.

Related: Draft model, Speculative decoding, Verification step, Acceptance rate

A specialised unit inside NVIDIA GPUs that multiplies small blocks of numbers in low precision far faster than general-purpose cores. Deep learning speed depends heavily on using these units, which support formats like FP16, BF16, INT8 and FP8.

Example. Matrix multiplications in FP16 run on tensor cores at many times the speed of standard FP32 cores.

Related: CUDA, Mixed precision, FP8, GPU

NVIDIA's toolkit for optimising trained networks for its GPUs. It fuses layers, chooses fast kernels, and supports lower precisions such as INT8 and FP8, producing an engine file tuned for one GPU type.

Example. Compiling an image classifier with TensorRT and INT8 cuts its latency on a T4 GPU compared with running it directly in PyTorch.

Related: ONNX, TensorRT-LLM, Kernel fusion, Inference engine

NVIDIA's library for running large language models efficiently on its GPUs. It supplies optimised attention kernels, quantisation, in-flight batching and multi-GPU parallelism, and is often deployed behind Triton Inference Server.

Example. An enterprise compiles a Llama-family model into a TensorRT-LLM engine for its A100 servers.

Related: TensorRT, Triton Inference Server, Continuous batching, FP8

The average time taken to produce each generated token after the first. Its inverse is the streaming speed a user sees, and it is governed largely by how fast weights and the KV cache can be read from memory.

Example. At 25 ms per token, a reply streams at 40 tokens per second, faster than most people read.

Related: Time to first token, Inter-token latency, Memory-bound, Latency

The delay between sending a request and receiving the first generated token. It covers queueing and processing the whole prompt, and is the main figure for how quickly a chat interface feels responsive.

Example. A user sends a 2,000-token prompt and the first word appears after 0.6 seconds.

Related: Time per output token, Latency, Prefill and decode, Prefix caching

A measure of generation speed: how many tokens a model produces each second, for one user (streaming speed) or across all users on a server (total throughput). The two numbers differ and answer different questions.

Example. A server producing 2,400 tokens per second across 60 users gives each about 40 tokens per second.

Related: Throughput, Time per output token, Cost per token, Continuous batching

A PyTorch feature that traces a model's operations and compiles them into optimised code, fusing operations and generating efficient kernels. It can speed up inference with a single line, though the first call is slow while compiling.

Example. Wrapping a model with torch.compile(model) gives faster repeated calls after a warm-up run.

Related: Kernel fusion, GPU kernel, ONNX Runtime, TensorRT

The full cost of running a system over its life, not just the sticker price: hardware or cloud fees, power, staff time, monitoring, downtime and upgrades. It is the fair basis for comparing self-hosting with a paid API.

Example. A cheap-looking self-hosted model becomes dearer than an API once two engineers' time and idle GPU hours are counted.

Related: Cost per token, GPU hours, Reserved capacity, Model serving

A masking scheme that lets a model score several branching candidate continuations in one forward pass, where each candidate token only attends to its own ancestors. It is used to verify many speculative alternatives at once.

Example. Two candidates for the next word and two follow-ons each are checked in one pass rather than four.

Related: Medusa decoding, Speculative decoding, Attention mask, Verification step

An open-source server from NVIDIA that hosts models from several frameworks behind HTTP and gRPC endpoints. It offers dynamic batching, running multiple model copies and chaining models in pipelines, so one deployment can serve a whole application.

Example. One Triton instance serves an ONNX classifier and a TensorRT-LLM chat model on the same GPU node.

Related: Dynamic batching, TensorRT-LLM, Model serving, ONNX Runtime

Setting individual weights to zero wherever they are least important, without regard to position. It can reach high sparsity with little accuracy loss, but ordinary GPUs cannot exploit scattered zeros, so speedups need special hardware or kernels.

Example. Zeroing 50 percent of weights by magnitude halves the non-zero count, yet dense matrix multiplication takes the same time.

Related: Pruning, Structured pruning, Sparsity, Model compression

In speculative decoding, the single forward pass of the large model over all proposed draft tokens at once. It computes the large model's own probabilities for each position, letting the system accept the matching prefix and resample the first mismatch.

Example. After 4 draft tokens, one forward pass yields 5 sets of probabilities: one per draft position plus a bonus next token.

Related: Speculative decoding, Target model, Acceptance rate, Draft model

An open-source engine for serving large language models with high throughput. It combines PagedAttention for memory management, continuous batching and many optimised kernels, and exposes an OpenAI-compatible HTTP API, making it a common choice for self-hosted LLM serving.

Example. A startup serves an open 8B model behind vLLM and points its existing OpenAI client at the new base URL.

Related: PagedAttention, Continuous batching, Prefix caching, OpenAI-compatible API

A group of 32 GPU threads that execute the same instruction together in lockstep. If threads in a warp take different branches, the paths run one after another, so kernels are written to keep warps uniform.

Example. A kernel launched with 256 threads per block contains 8 such groups.

Related: Streaming multiprocessor, CUDA, GPU kernel, Shared memory

A scheme that compresses only the stored weights, for example to 4 bits, while activations stay in 16-bit. Because text generation is limited by how fast weights can be read from memory, this cuts memory traffic and speeds decoding without needing integer matrix hardware.

Example. An AWQ 4-bit checkpoint keeps activations in FP16; weights are unpacked to FP16 inside the kernel just before each multiplication.

Related: Quantisation, Activation quantisation, AWQ, Memory-bound

An integer offset in asymmetric quantisation that says which stored integer represents the real value zero. It lets an uneven range of values, such as post-activation outputs that are all positive, use the whole integer range without wasting codes.

Example. For values between 0 and 6 stored as unsigned 8-bit, the zero-point is 0; for values between -1 and 5 it is about 43.

Related: Scale factor, Asymmetric quantisation, Symmetric quantisation, INT8