A fraud alert fires before your card transaction clears. Your phone unlocks the instant your face hits the frame. A chatbot streams back a reply token by token. All three are the same thing under the hood: AI inference, the operational heartbeat of every deployed model, and quietly one of the most expensive and contested problems in enterprise computing right now.
Mechanically, it is dull. Inference is a single forward pass through a neural network. The parameters are frozen, no gradients get computed, no weights move. Input goes in one end, activates layers of learned transformations, and comes out as a probability distribution, a classification label, or a token. Append that token to the context and run it again until the output is done.
The read-only part matters more than people realize. The model you query today is stuck at the state of its training data at a fixed cutoff. It cannot learn from your question, cannot pull in yesterday’s news, cannot correct a mistake it made in the previous session. Every call is stateless with respect to the model’s parameters, even when the application layer is dutifully holding your conversation history. People conflate session memory with the model actually learning something, and that confusion sits at the root of a lot of misplaced expectations about what these systems do.
Set training beside inference and the asymmetry jumps out.
| Dimension | Training | Inference |
|---|---|---|
| Compute demand | 10–100× higher; multi-GPU clusters, weeks | Lower per request; optimized for throughput |
| Duration | Hours to weeks | Milliseconds to seconds |
| Data | Large labeled historical datasets | Live, unlabeled, real-world data |
| Parameter updates | Yes (forward + backward pass) | No (forward pass only) |
| Primary business concern | Model accuracy | Latency, cost, scalability |
Training is capex you pay once. Inference is opex that scales linearly with every query, and often worse than linearly. At millions of concurrent requests the math turns brutal, which is exactly why squeezing inference has grown into its own engineering discipline.
That discipline starts with the shape of the deployment, because inference in production wears at least four different faces. Real-time online inference handles one request at a time in milliseconds for chatbots, payment fraud detection, recommendation engines. SUSE puts the practical threshold at sub-100ms for anything a human is waiting on, and above that the interaction just feels broken. Hitting that at scale takes purpose-built serving infrastructure, not whatever general-purpose compute you had lying around.
Batch inference throws latency away to buy efficiency. Thousands or millions of records get chewed through asynchronously on a schedule: overnight credit scoring, bulk image tagging, document classification pipelines. Nobody is waiting, so the system optimizes for throughput and drives the per-inference cost way down. Streaming sits in the middle, what Mirantis frames as continuous processing of data coming off event streams or sensor feeds, live video analysis, IoT telemetry, log anomaly detection. The model has to keep pace with the data rate but escapes the hard per-request ceiling that interactive apps impose.
Edge inference takes the model off the cloud entirely and drops it onto the device, a phone, an industrial sensor, the computer inside a self-driving car. You do it to kill the network round-trip, to keep sensitive data local, and to keep working when the connection dies. The catch is hardware: an edge device has a fraction of the memory bandwidth and compute of a data center GPU, so fitting a capable model into that envelope demands aggressive optimization. Google Cloud names a fifth idea worth separating out, AI serving, the act of packaging a trained model and exposing it as an API endpoint, with load balancing, versioning, autoscaling, and health checks. It is not inference itself, but you cannot ship inference to production without it.
Cost compounds with scale, so a whole layered toolkit exists to shrink the compute and memory each inference needs without wrecking accuracy. Quantization is the one everybody reaches for first. A model trained in 32-bit floating point can usually drop to 8-bit integers with barely any accuracy loss, cutting the memory footprint by 4× and speeding up the matrix math on hardware that eats integers faster than floats. Push to 4-bit or lower, and it gets interesting, with tradeoffs that swing hard depending on the architecture and the task. Pruning goes at it differently, stripping out parameters that barely move the output, either individual weights (unstructured) or whole attention heads and layers (structured). Structured pruning runs faster on stock hardware without needing exotic sparse-matrix kernels.
The KV cache is a bottleneck of its own, and vLLM is the piece Red Hat bakes into its stack to fix it. Autoregressive LLM inference has to hold the key-value attention cache for every live request in GPU memory, and naive implementations fragment that memory so badly they choke the number of requests a single GPU can serve. vLLM’s PagedAttention manages the cache the way an operating system manages virtual memory pages, and GPU utilization under concurrent load climbs sharply as a result.
Mixture of Experts changes the model itself rather than patching it after the fact. Only a slice of the parameters, specific expert subnetworks, fire for any given input token. A model carrying 100 billion parameters total might light up only 10 to 20 billion per forward pass, cutting compute while keeping the representational muscle of the full count. Mixtral and Google’s Gemini 1.5 both run this way. Underneath it all, the workload has pulled real money into custom silicon. GPUs still dominate because massively parallel hardware maps cleanly onto the matrix multiplies that eat most of the runtime. Google’s TPUs are ASICs built for exactly this, with memory and interconnect tuned for transformer inference at scale. FPGAs give programmable acceleration for particular architectures, and a whole category of inference-specific ASICs, designed from scratch for forward-pass math instead of general parallel work, is drawing investment from incumbents and startups alike.
Past a certain model size, Micron’s framing goes, memory bandwidth is the wall, not compute. A model with hundreds of billions of parameters has to haul every one of them out of memory into the compute units on every forward pass. How fast data moves from memory to processor, measured in GB/s, sets the ceiling no matter how many FLOPS the GPU can theoretically hit. A GPU pinned at 100% utilization but starved for bandwidth is not running faster inference; it is sitting there waiting.
High Bandwidth Memory answers this in data center GPUs like NVIDIA’s H100, stacking the memory die right next to the compute die and wiring them together with thousands of parallel paths. The H100 SXM variant pushes roughly 3.35 TB/s. GDDR6X in consumer and workstation cards manages around 1 TB/s. LPDDR5X in phones and edge devices crawls along at 60 to 80 GB/s, which is the real reason edge inference of a large model needs aggressive quantization and architectural surgery rather than a few software tweaks. The wall gets worse as models grow. A 70-billion-parameter model in 16-bit precision needs about 140 GB just to hold the weights, which blows past a single H100’s 80 GB of HBM3 and forces you into model parallelism across multiple devices, dragging its own latency and interconnect tax behind it.
The commercial side has split into layers, each with its own competitive logic. Google Cloud serves inference through Vertex AI on TPU pods and GKE-managed infrastructure, and running Gemini on the same plumbing it rents out gives Google a tight loop between building models and optimizing how they serve. Red Hat comes at it Kubernetes-native and enterprise-first: OpenShift AI and RHEL AI give you a managed environment for inference on-prem or in hybrid cloud, with vLLM baked in for LLM serving and MoE support, aimed squarely at organizations that will not put sensitive workloads on public cloud. SUSE’s AI Factory, built with NVIDIA, chases both edge and cloud from a Kubernetes foundation, and that partnership is the point, since it hands you clean access to TensorRT and Triton, GPU optimization libraries that are otherwise a pain to wire into enterprise Linux. Mirantis goes multi-tenant with k0rdent, an “Inference Mesh” abstraction and a Model Registry for versioning across distributed deployments, which lands on a genuine operational headache: teams running dozens of models need governance and resource isolation that vanilla Kubernetes will not hand you.
Cost is the tension that bites today. Training is a huge one-time bill that keeps falling as hardware improves and distillation spits out smaller capable models. Inference is forever, and it grows with every user you add. A feature that pencils out at 10,000 daily actives can be flatly unviable at 10 million. That reality is reshaping how people deploy: smaller specialized models instead of one giant generalist, aggressive caching of common outputs, tiered serving that shunts easy queries to cheaper models.
Cloud versus edge has no tidy resolution either. Cloud gives you elastic scale and the biggest models, but you eat network latency (20 to 100ms round-trip to a well-placed data center, more once your users spread out), egress costs, and privacy exposure. Edge erases all three and then hits a hardware ceiling climbing slower than data center silicon. Autonomous vehicles, medical imaging at the bedside, industrial process control all need latency and reliability that a cloud architecture cannot promise. Everyone is building toward a hybrid split, big foundation models in the cloud and small specialized ones at the edge, but the tooling to manage that split cleanly is still half-baked.
The frozen model runs deeper than either. Every deployment out there is running a model that stopped learning at a fixed moment. RAG patches over it by shoving retrieved documents into the context window at inference time; web search does a version of the same trick; continuous fine-tuning pipelines periodically refresh the weights on new data. None of these fix the architecture. They are workarounds that pile on latency, complexity, and cost. A model that could rewrite its own parameters mid-inference would be a fundamentally different beast, and it would drag stability, safety, and reproducibility questions in with it. Until then, the gap between what users expect a model to know and the frozen reality of its weights gets papered over with application-layer engineering.
Inference is where AI quits being a research artifact and becomes a product. The training runs grab the headlines, but the decisions here- what to optimize, where to deploy, how to survive the cost at scale- settle the whole economics of AI. The thing that quietly breaks the most ambitious AI products over the next few years will not be model quality. It is the inference bill nobody modeled before they shipped.