Evergreen Intelligence · Technology Fundamentals · October 15, 2026
 |
| Image credit: Pexels |
When you type a question into a chatbot, ask an image model to create a picture, or run a recommendation engine, the system is performing inference: a forward-only pass through fixed model weights that turns your input into a usable result. Unlike training—which discovers and updates the model’s parameters—inference freezes those parameters and applies them.
This article explains the mechanics of inference for modern generative models, the two computational phases that dominate performance, the latency/throughput/cost trade-offs that shape production systems, and how those workloads connect to high-bandwidth memory (HBM), accelerators, and infrastructure economics.
From Prompt to Tokens: The Basic Pipeline
Raw text never enters the model directly. The serving stack first applies a model-specific prompt template (system instructions, conversation history, special tokens) and then tokenizes the result. A tokenizer splits text into subword units drawn from a fixed vocabulary—typically tens of thousands of entries—and maps each unit to an integer ID.
Those IDs are looked up in an embedding table and become dense vectors. The vectors, together with positional encodings, travel through the transformer layers. At the end of the network, the final hidden state is projected into logits—scores over the entire vocabulary. A sampling step (greedy, temperature, top-k, top-p, etc.) selects the next token. The chosen token is detokenized into text and, if generation continues, fed back into the model as part of the growing sequence.
The loop repeats until a stop condition is met: an end-of-sequence token, a maximum length, or an external policy. The entire process is a pure forward pass; no gradients are computed, and no weights are updated.
Prefill and Decode: Two Phases, Two Bottlenecks
Modern decoder-only transformers split inference into two distinct phases with opposite hardware characteristics.
Prefill (Prompt Processing / Context Encoding)
The entire prompt is processed in parallel in a single forward pass. Query, key, and value projections are computed for every token simultaneously; attention runs as large matrix multiplications. Intermediate key and value tensors are written into the KV cache so they never need to be recomputed. Prefill is typically compute-bound: the GPU’s arithmetic throughput (FLOPs) limits speed. It determines time-to-first-token (TTFT)—the delay before the user sees the first output token.
Decode (Token Generation)
After the first token is produced, generation becomes sequential and autoregressive. For each new token, the model computes a query for that token only, attends over the growing KV cache, runs the feed-forward layers, and samples the next token. The new key/value pair is appended to the cache. Because each step must re-read the full set of model weights (and an expanding cache) from memory to produce only one token, decode is memory-bandwidth-bound. Throughput is measured in tokens per second and is usually far lower than prefill rates—often tens to a few hundred tokens per second versus thousands during prefill.
The KV cache is the short-term memory that makes long-context generation practical. Without it, every new token would require recomputing attention over the entire history, an O(n²) cost that quickly becomes prohibitive. Cache size grows linearly with sequence length and number of layers/heads, so long contexts or high concurrency quickly pressure HBM capacity.
Latency, Throughput, and Serving Infrastructure
Production inference systems must simultaneously satisfy three competing goals:
- Latency — How quickly a single user receives a response (TTFT and inter-token latency). Interactive chat demands low single-digit-second first-token times and smooth streaming.
- Throughput — How many tokens or requests the system can process per unit time across all users. Higher batch sizes improve hardware utilization and lower cost per token, but can increase per-request latency.
- Cost — Dollars (and watts) per million tokens or per successful request.
These goals conflict. Aggressive batching raises throughput and reduces cost but can hurt tail latency. Continuous batching and paged attention allow the scheduler to pack requests of different lengths dynamically, improving utilization without forcing every request to wait for the longest sequence. Serving stacks typically include a tokenizer/detokenizer layer, a scheduler that manages the KV cache and request queues, an optimized runtime, and a model-parallel or tensor-parallel distribution layer when the model no longer fits on a single accelerator.
Inference Costs and the Shift from Training Economics
Training is a large, finite capital expenditure: weeks or months of tightly coupled accelerator clusters that produce a set of weights. Inference is a continuous operating expense that runs for the entire life of the deployed model. At scale, lifetime inference cost routinely exceeds the original training cost—often by a large multiple—because the model answers millions or billions of queries.
Industry analyses indicate that inference already accounts for the majority of AI infrastructure demand and is projected to reach roughly 60% of AI compute demand by 2030. Per-token prices have fallen sharply, yet total spend continues to rise because usage volume grows faster than unit costs decline. Reasoning models that generate long internal chains of thought further amplify inference compute, sometimes by one or two orders of magnitude relative to simple chat.
Why HBM, Accelerators, and Packaging Matter for Inference
Decode’s memory-bandwidth wall makes high-bandwidth memory the critical resource. Generating each token requires streaming the model weights (or the active subset) plus the KV cache from HBM into the compute units. Higher HBM bandwidth directly raises tokens-per-second; higher HBM capacity supports longer contexts and more concurrent sessions before the cache spills.
AI accelerators—GPUs, custom ASICs, and specialized inference chips—are therefore designed around large HBM stacks and the advanced packaging techniques that attach them. The same packaging and memory constraints that limit how many accelerators can be shipped also limit how much inference capacity the industry can deploy.
For deeper treatment of these hardware foundations, see The CODEW’s existing Technology Fundamentals pieces:
- What is HBM and why it matters for AI
- What is an AI accelerator?
- AI Inference vs AI Training: What’s the Difference
The CODEW Lens
AI inference is no longer a secondary afterthought to training. It is the continuous, production workload that turns trained intelligence into useful answers, and its performance and cost characteristics are governed by memory bandwidth, KV-cache management, and the physical limits of advanced packaging and HBM supply. Understanding the prefill/decode split, the latency–throughput–cost triangle, and the hardware that feeds those phases is now foundational knowledge for anyone building, buying, or analyzing AI systems.
What Is AI Inference? is part of Evergreen Intelligence → Technology Fundamentals within The CODEW Intelligence — structured research covering the models, platforms, chips, infrastructure, and economics that shape the AI economy.
THE CODEW · TECH & MARKET INTELLIGENCE · EVERGREEN INTELLIGENCE · TECHNOLOGY FUNDAMENTALS
Reviewed by Erwin Castro
on
Thursday, October 15, 2026
Rating:
No comments: