Choosing the right hardware for local LLMs starts with GPU VRAM, not CPU performance. Your graphics card’s memory determines which large language models (LLMs) you can run, which quantization formats you can use, and how much context your model can handle. A GPU with 16GB of VRAM can run many 7B–14B models at 4-bit quantization, while 24GB provides more flexibility for larger models, including some 32B models with optimized settings. Running a 70B model at 4-bit typically requires around 42–47GB of memory for model weights and loading overhead, making a high-memory GPU, multiple GPUs, or a suitable unified-memory Mac necessary.
Running LLMs locally gives you greater control over your data, reduces dependence on cloud AI services, and eliminates per-request API fees when you use local inference software. However, performance depends on more than GPU memory alone. System RAM, CPU capabilities, memory bandwidth, model quantization, context length, and inference software all affect the overall experience. Understanding these hardware requirements helps you choose a PC or workstation that matches your AI workloads without overspending.
What do these terms mean?
An LLM, or large language model, generates text one piece at a time. Those pieces are called tokens, roughly parts of words. A name like “8B” means about 8 billion parameters, the internal numbers it learned in training.
VRAM is the memory on your graphics card — separate from system RAM, and much faster for this work. The context window is how much text the model can keep in view at once. The KV cache is the extra memory used to remember that context while it writes, and it grows as the conversation gets longer.
Understanding Local LLM Hardware:
A large language model (LLM) generates text by predicting one token at a time. Tokens are small units of text that may represent complete words, parts of words, or punctuation. Model sizes such as 7B, 14B, and 70B refer to approximately 7 billion, 14 billion, and 70 billion parameters, respectively. Parameters are the numerical values the model learns during training.
GPU VRAM is the dedicated memory on a graphics card used to store model weights and support inference. System RAM serves a different purpose, although some configurations can offload model data to it. The context window determines how much input and conversation history a model can process at once, while the KV cache (key-value cache) stores intermediate attention data used during text generation. As the context grows, KV cache memory usage generally increases.
These components work together to determine whether a model fits your hardware and how efficiently it generates responses. Before buying a GPU for local AI, compare its VRAM capacity with your target model’s quantization level, context requirements, and expected workload.
Why is VRAM the budget that decides what you can run?
If a model does not fit in VRAM, part of it spills into system RAM, and answers get much slower because system RAM is far slower than VRAM.
That is why capacity beats raw speed here. The RTX 5080 and the RTX 5060 Ti 16GB both have 16GB of VRAM — NVIDIA lists the RTX 5080 at 16GB GDDR7 and the RTX 5060 Ti at 16GB or 8GB GDDR7 — so they fit the same model sizes. The RTX 5080 answers faster, but it does not unlock a bigger model.
Plain-English version: Quantization means storing each of the model’s numbers in fewer bits, like rounding a long decimal to a shorter one. At full precision, called FP16, each parameter takes 2 bytes. At Q8, about 1 byte. At Q4 — the common 4-bit formats such as Q4_K_M in llama.cpp — roughly 0.57 bytes once format overhead is counted. The quality loss is small at Q4 and barely noticeable at Q8. The math used by technical guides is: parameters × bytes per parameter, plus KV cache, plus about 15 to 20 percent overhead.
The table below uses that math. Figures are estimates for total memory in use, not just the download size, and they include overhead and a moderate context. Longer context raises every row.
| Model Size | 4-bit Quantization (Q4) | 8-bit Quantization (Q8) | FP16 (16-bit Precision) |
|---|---|---|---|
| 7B–9B Parameters | Approximately 6–7 GB VRAM — suitable for budget GPUs and local AI experimentation. | Approximately 9–11 GB VRAM — offers a balance between memory usage and model precision. | Approximately 16–18 GB VRAM — requires more memory to preserve higher numerical precision. |
| 14B Parameters | Approximately 10–12 GB VRAM — suitable for capable consumer GPUs with moderate memory. | Approximately 16–18 GB VRAM — benefits from GPUs with higher VRAM capacity. | Approximately 30 GB VRAM — typically requires a high-memory GPU or multi-GPU setup. |
| 32B Parameters | Approximately 20–22 GB VRAM — suitable for high-end consumer GPUs or optimized configurations. | Approximately 36–38 GB VRAM — generally requires professional GPUs or multiple GPUs. | Approximately 68 GB VRAM — typically requires high-memory professional GPUs or a multi-GPU configuration. |
| 70B Parameters | Approximately 42–47 GB VRAM — requires substantial GPU memory or multi-GPU inference. | Approximately 75–80 GB VRAM — generally requires enterprise-class GPUs or multiple GPUs. | Approximately 145 GB VRAM — typically requires multiple high-memory GPUs or specialized server hardware. |
Download sizes line up with the Q4 column: the Ollama library lists a Llama 3.1 8B Q4_K_M file at about 4.9GB and a Qwen 2.5 14B file at about 9.0GB, before cache and overhead. A 70B Q4 download is about 43GB, so it cannot fit on a 24GB or 32GB card once the cache is added.
Which graphics card should you buy?
Best for a budget start: a 16GB card
Best for getting started is the GeForce RTX 5060 Ti 16GB — buy the 16GB version, not the 8GB version. It runs 7B to 9B models with room for longer context, and 14B models at Q4 comfortably, which covers most personal chat, writing, and coding help. The RTX 5080, also 16GB, is faster if you also game, but buys speed, not capacity. Prices move fast: the card launched at $429, and an October 2026 tracker put the lowest new asking price at $617.99. Checked: October 23, 2026, BuySellRAM October 2026 market update. Check current listings before you budget.
Best for 24GB value: used RTX 3090 or RX 7900 XTX
Best for 24GB on a budget is a used GeForce RTX 3090, listed by NVIDIA at 24GB GDDR6X. At 24GB you can run 32B models at Q4 with a modest context — a clear step up in answer quality. The cautions are the usual used-market ones: no warranty, unknown prior use, and a 350W-class card that needs good airflow. Test it under load right after buying, from a seller with returns. One tracker put the used RTX 3090 average at $1,343 across 301 listings in September 2026. Checked: October 23, 2026, PCServerAndParts used GPU guide citing ResalePrices. Check current listings.
The new alternative is the AMD Radeon RX 7900 XTX, listed by AMD at 24GB GDDR6 and 355W typical board power. The trade-off is software, not hardware: most tools target NVIDIA CUDA first. AMD’s ROCm docs list the RX 7900 XTX as supported on Windows for runtime and HIP SDK, and Ollama and llama.cpp have AMD paths, but expect more setup friction and check your tool’s support list first. On Linux the AMD path is better established.
Best for a single card with no compromise: RTX 5090 32GB
Best for the most you can fit on one card is the GeForce RTX 5090. NVIDIA lists it at 32GB GDDR7, 575W total graphics power, and a 1,000W required system power. It runs 32B models at Q4 with generous context, or at Q8 with a shorter context — but still not a 70B model at Q4 once cache is added. It launched at $1,999, yet one October 2026 tracker put the lowest new asking price at $4,920.99. Checked: October 23, 2026, BuySellRAM October 2026 market update. Check current listings.
Best for very large models: more than one card, or a Mac
Best for 70B and larger is either two graphics cards in one PC, so their VRAM adds up for a split model, or an Apple Mac with unified memory — one pool shared by CPU and GPU, so the full amount can hold model weights. Apple’s Mac Studio specs list M5 Max up to 128GB of unified memory, and M5 Ultra at 256GB or 512GB, enough for a 70B model at Q4 or Q8 in one box. The trade-offs are cost, memory fixed at purchase, and Apple’s Metal and MLX software path. Multi-GPU PCs keep the CUDA tools, but add power, heat, and setup complexity.
Who should not buy yet?
Do not buy a small card expecting ChatGPT-level answers from a huge model: a 70B model forced onto a 16GB card will mostly run from system RAM and reply slowly. Do not buy for a model size you have not tried, and do not buy a second card before checking your power supply and case space.
What do the CPU and system RAM actually do?
Once a model fits fully in VRAM, the graphics card does almost all the work. The CPU handles loading and any layers that spill over, so a modern mid-range CPU is enough for single-user use.
System RAM is your overflow space. Tools can offload part of a model to RAM when VRAM runs out, which lets an oversized model run at all — at a clear speed cost, because data crosses the slower RAM-to-GPU connection on every token. Match RAM to VRAM at a minimum: 32GB is a sensible start for a 16GB or 24GB card, and 64GB if you plan to offload larger models.
How much storage do model files need?
Model files are big, and most people keep several: one 8B Q4 file is about 5GB, a 14B file about 9GB, a 32B file roughly 19 to 20GB, and a 70B Q4 file about 43GB. Four or five models can easily pass 100GB. Put them on an NVMe SSD so models load quickly when you switch.
What about power draw?
Power follows the card ratings. A single RTX 5090 is rated at 575W, and NVIDIA requires a 1,000W system supply for it. Two cards can pass 700W for the graphics cards alone, before the CPU and rest of the system, so a high-wattage, good-quality supply is not optional there. Size it for the full system, and check connector types before you buy.
Try before you buy hardware
You may not need new hardware to learn what you need. Ollama runs models from a simple command line and local API, and downloads them for you. LM Studio is a desktop app for searching, downloading, and chatting with models, with a local API as well. Under both sits llama.cpp, the engine that works with quantized GGUF files. Install one, run a 7B or 8B model at Q4 for a week of real tasks, and you will know whether your limit is memory or speed — and what, if anything, is worth buying.
Conclusion
Choosing the right hardware for local LLM inference starts with GPU VRAM. A 16GB GPU suits many 7B–14B models at 4-bit quantization, while 24GB offers more flexibility for 32B models. Running 70B models typically requires 48GB or more of available memory, potentially through multiple GPUs or a high-memory Mac. Pair your GPU with at least 32GB of system RAM, sufficient NVMe SSD storage, and a power supply that supports your complete system. Before upgrading, test your existing PC using Ollama or LM Studio with an 8B model. Evaluate response speed, memory usage, and output quality to determine which hardware upgrade best fits your local AI workloads.
Frequently Asked Questions
How much VRAM do I need to run a local LLM?
For most people, 16GB is the practical starting point. It runs 7B to 9B models with room for longer chats, and 14B models at 4-bit quantization. Step up to 24GB if you want 32B models, which give noticeably stronger answers. Below 8GB, stick to small 3B to 7B models at Q4 and keep context short.
Can I run a 70B model on a 24GB card?
Not fully, and not well. A 70B model at Q4 needs about 42 to 47GB in total once the KV cache and overhead are included, and its download alone is about 43GB. On a 24GB card, most of it would sit in system RAM and reply slowly. Use two cards, a high-memory Mac, or choose a 32B model instead.
Is an AMD card okay for local LLMs?
Yes, with a caveat about setup effort. The RX 7900 XTX gives you 24GB for less than most new 24GB NVIDIA options, and AMD’s ROCm documentation lists it as supported on Windows and Linux. In practice, most tools are built for NVIDIA CUDA first, so expect extra setup steps and check your tool’s AMD support page before buying.
Do I need a fast CPU for a local LLM?
No, not for normal single-user chat. If the model fits in VRAM, the graphics card does the heavy lifting and a modern mid-range CPU is fine. The CPU matters more when you offload layers to system RAM, or run a model with no graphics card at all. In those cases, more cores and faster RAM both help, but it will still be slower than GPU use.
How much system RAM should I pair with my graphics card?
Start with at least as much system RAM as VRAM, and prefer more. For a 16GB or 24GB card, 32GB of system RAM is a sensible baseline for loading models and normal desktop use. Choose 64GB if you plan to offload part of a larger model to RAM, run long contexts, or keep several programs open while a model is loaded.
Can I just use ChatGPT instead of buying hardware?
Sometimes, yes — and it is the cheaper option if you only need occasional help. Buying hardware makes sense when you want your prompts to stay private on your own machine, need to work offline, use it heavily every day, or want to run a specific open model. Try a small free model locally first, then decide with real experience instead of guessing.
