As large language models move into local environments, performance differences between hardware configurations become more apparent. The same model can feel sluggish on a CPU but responsive on a GPU. The main reason is how memory is structured and accessed, which matters more here than raw compute power. Understanding the relationship between memory bandwidth, latency, and parallelism helps explain why GPUs typically outperform CPUs for LLM inference.
The bottleneck of system RAM
When an LLM is executed on a CPU, it relies on system RAM for storing model weights and intermediate computations. While RAM is relatively fast compared to storage, it supplies far less bandwidth than the compute units can consume, and streaming a whole model through it for every token exposes that gap.
Several limitations emerge:
- Lower bandwidth. Typical system RAM provides tens of gigabytes per second
- Latency is harder to hide, since with few cores in flight each memory wait stalls the work instead of being overlapped
- Repeated weight streaming means that weights are re-read from memory for every token, and on a narrow bus each pass costs far more time than it does on a GPU
Matrix multiplications and tensor operations dominate an LLM’s work. Both mean shifting large blocks of data, quickly and over and over. Starve them of bandwidth and the processor waits more than it computes.
In addition, CPUs are designed for general-purpose tasks. They offer flexibility but lack the large-scale parallelism that works through a whole prompt at once, which is why a CPU takes longer to reach the first token.
The result is:
- Slower inference
- Higher energy consumption
- Poor scalability as model size increases
Why GPUs perform better
GPUs are designed with a very different workload in mind. Their architecture is optimized for parallel computation and high-throughput data access, which matches what LLMs need.
These advantages come from four properties that are easy to confuse. Figure 1 compares them side by side for a CPU paired with system RAM and a GPU paired with vRAM.
Figure 1
Two places to keep a model
A CPU with system RAM and a GPU with vRAM differ across four properties that are easy to confuse.
How fast data moves between memory and the compute units, measured in gigabytes per second.
GPU vRAM moves data roughly 10 to 36 times faster than system RAM, depending on the card and the system RAM it is compared against. This is the main reason token generation speeds up on a GPU.
Higher memory bandwidth
GPU memory, often referred to as vRAM, provides higher bandwidth. Modern GPUs reach a thousand gigabytes per second and beyond, 1,008 GB/s on an RTX 4090 and 1,792 GB/s on an RTX 5090. Model parameters are therefore accessed far more quickly.
That bandwidth allows:
- Faster loading of weights
- Reduced waiting time between operations
- More efficient execution of large tensor computations
Bandwidth also sets a ceiling on generation speed. During generation the model reads its full set of weights from memory for every token it produces, so with one token per forward pass the token rate cannot exceed the memory bandwidth divided by the model’s size in memory. That ceiling bounds a single stream, since batched decode reads each weight once for many sequences. That rule describes a dense model. A mixture-of-experts model reads only its active experts for each token, so its ceiling follows the active parameters rather than the whole weight file. Figure 2 estimates that ceiling for several model sizes and precisions.
Figure 2
Each token re-reads the whole model
During generation the model reads all of its weights from memory for every new token, so bandwidth sets a ceiling on speed.
Theoretical decode ceiling (tokens per second)
Parallelism
GPUs contain thousands of cores capable of executing operations simultaneously. This is particularly effective for:
- Matrix multiplications
- Attention mechanisms
- Batch processing of tokens
Instead of processing operations sequentially, GPUs distribute the workload across many cores, greatly increasing throughput.
A wider path to the compute units
When a model is fully loaded into vRAM, both the parameters and intermediate results remain close to the compute units. The weights still stream to the compute units for every token, but they travel over a much wider path.
Keeping every layer on that wider path leads to:
- Faster inference speeds
- More stable latency
- Better utilization of hardware resources
The reverse case explains what happens when a model’s layers are split between the GPU and system RAM, whether because the model is too large to fit or because only part of it was offloaded. Every token must still read the layers left in system RAM at that slower tier’s bandwidth, so the effective speed follows the slowest part of the path. Figure 3 shows how much of the model must sit on the GPU before the faster memory takes over, which sets up the comparison in the example that follows.
Figure 3
The partial-offload cliff
When a model does not fit in vRAM, the layers left in system RAM dominate the memory time of every token.
Example: Qwen 3 32B performance
CPU vs GPUInference
Why a local LLM crawls on a CPU but flies on a GPU. It comes down to memory bandwidth, latency, and parallelism, plus keeping the whole model in fast GPU memory instead of shuttling it through system RAM.
CPU vs GPU Inference. Why a local LLM crawls on a CPU but flies on a GPU. It comes down to memory bandwidth, latency, and parallelism, plus keeping the whole model in fast GPU memory instead of shuttling it through system RAM. Key topics covered: CPU and system RAM, Memory bandwidth, GPU and vRAM, Parallelism, Data movement, Model placement, Benchmark setup, Throughput results.
The effect of memory placement is shown by the comparison below. A 32B parameter model was run using 6-bit quantization in two configurations.
Measured result · Qwen 3 32B, 6-bit
6.80tok/s
Model split between vRAM and system RAM.
18.03tok/s
Entire model in the RTX 5090’s 32 GB of vRAM.
Same model and prompt in LM Studio. Moving the whole model into vRAM also cut time to first token from 0.38 s to 0.12 s and dropped system RAM use from 7.36 GB to 3.07 GB. The screenshots below show both runs.
CPU / partial GPU setup
- Model partially relying on system RAM
- Performance: 6.80 tokens per second
In this setup, throughput was constrained by the layers left in system RAM, which every token had to re-read at that slower tier’s bandwidth. Figure 4 shows the run in LM Studio.
Full GPU offload
- Model fully loaded into GPU memory
- Hardware: NVIDIA GeForce RTX 5090 with 32 GB vRAM
- Performance: 18.03 tokens per second
The move from 6.80 to 18.03 tokens per second is a 165% increase in throughput. Figure 5 shows the setting that places every layer on the GPU, and Figure 6 shows the resulting run. The improvement comes from bandwidth. With every layer in vRAM, each token reads the weights over a 1,792 GB/s path instead of pulling part of them from system RAM at roughly 90 GB/s.
Practical implications
Model placement matters
Getting as much of the model as possible into GPU memory is the single most effective step that can be taken. Partial offloading carries a cost. Every token re-reads the layers left in system RAM at that tier’s slower bandwidth, and the effective speed follows the slowest part of the path.
Bandwidth often dominates compute
For single-stream decode, memory bandwidth is the primary constraint at any precision. Quantization does not create that limit. It raises the ceiling by shrinking the bytes each token must read. Systems with higher memory throughput tend to perform better even if raw compute is similar.
Hardware selection should reflect workload
For interactive applications such as chat interfaces or real-time analysis, GPU-based inference sustains the higher token rate. CPU-based setups may still be suitable for offline or low-frequency tasks where latency is not critical.
Best practices
A few practical steps consistently improve performance:
- Prefer full GPU offloading when memory allows
- Select quantization levels that fit entirely within vRAM
- Monitor memory usage to avoid spillover into system RAM
- Benchmark token throughput under realistic workloads
These considerations often yield larger gains than minor algorithmic optimizations.
Key Takeaways
Decoding reads every weight from memory to produce each token, so a single stream cannot exceed memory bandwidth divided by the model’s size in memory. Lifting that ceiling takes faster memory, and more cores will not do it.
Time per token adds up across tiers, so a split model is held back by the part left behind. Figure 3 opens with 90% of the layers in vRAM and still shows only 620 GB/s of effective bandwidth, about 35% of the full-vRAM speed.
vRAM size decides whether the weights get the fast path at all. The speed of that path comes from bandwidth. Qwen 3 32B at 6-bit needs roughly 24 GB of weight memory by the nominal 0.75 bytes per parameter, and closer to 27 GB as an actual 6-bit K-quant file. Either way, all 64 of its layers fit inside the RTX 5090’s 32 GB.
The same model and prompt went from 6.80 to 18.03 tokens per second, a 165% increase, once all 64 layers sat in vRAM. Time to first token fell from 0.38 s to 0.12 s and system RAM use dropped from 7.36 GB to 3.07 GB.
Storing weights at 6-bit costs about 0.75 bytes per parameter, so fewer bytes stream for each token. The gain comes from moving fewer bytes rather than from doing less arithmetic, since 6-bit weights are unpacked to a wider format before the math runs.
Figure 2 puts the 32B 6-bit ceiling at 75 tokens per second on an RTX 5090, while the measured run reached 18.03. The estimate is computed from the weight memory alone, while activations, the key-value cache and quantization scales add memory traffic of their own and kernel launches add time on top.
Data & License
No third-party dataset. The benchmarks and measurements shown are the author’s own, recorded on the hardware and software described in this article.
© 2025 Philip Sarajlic. All rights reserved for the article’s original text and figures.
Code examples in this article are licensed under the Common Public Attribution License Version 1.0 (CPAL-1.0), an OSI-approved copyleft license based on the Mozilla Public License 1.1. Initial Developer: Philip Sarajlic.
Attribution required by CPAL Exhibit B: © 2025 Philip Sarajlic · “Based on code by Philip Sarajlic” · philipsarajlic.com · no graphic image. This attribution must be displayed in Larger Works.
Modifications must be released in Source Code form under CPAL-1.0. Making the code usable by anyone other than you over a network is External Deployment under the license and is treated as distribution, so the Source Code must be made available to those users.
Full text: opensource.org/license/cpal-1-0 (SPDX identifier CPAL-1.0)
