What Hardware Does a Company Need to Run a Local LLM, and How Much GPU Memory?
Start with GPU memory, not the CPU. The model file must fit entirely in GPU memory, with room left for context and the number of people using it at once. Going by download sizes in Ollama's official library, an 8B-class model is about 5.2GB, 14B about 9.3GB, and 32B about 20GB. OpenAI states that gpt-oss-20b can run within 16GB of memory, while gpt-oss-120b fits on a single 80GB GPU.
1. Why GPU memory comes first
To answer a question, a local model has to be loaded into memory. Whatever fits in GPU memory runs on the GPU; whatever does not is placed in system memory and computed by the CPU, which is noticeably slower.
Ollama's ollama ps shows the split directly. A PROCESSOR value of 100% GPU means the whole model is on the graphics card; a ratio such as 48%/52% CPU/GPU means only part of it fits (the example in Ollama's FAQ). So the order is: decide the model size, check whether GPU memory is enough, then look at everything else.
2. How much space common models need
Download sizes from Ollama's official library (default tags, checked 2026-10-19). File size roughly equals the memory needed to load the model; context needs additional memory on top.
| Model | Download size | Context length | Official note |
|---|---|---|---|
| qwen3:8b | 5.2GB | 40K | — |
| qwen3:14b | 9.3GB | 40K | — |
| qwen3:30b (MoE, 30B-A3B) | 19GB | 256K | Mixture of experts; only part of the parameters are active per step |
| qwen3:32b | 20GB | 40K | — |
| gpt-oss:20b | 14GB | 128K | Official: runs within 16GB of memory |
| gpt-oss:120b | 65GB | 128K | Official: fits on a single 80GB GPU |
Hardware reference points (official figures):
- Consumer graphics card example: the NVIDIA GeForce RTX 5090 has 32GB of GDDR7 memory, enough for every model above up to 32B.
- Desktop AI system example: NVIDIA DGX Spark has 128GB of unified memory; NVIDIA states a single unit runs models up to 200 billion parameters.
Prices and availability in Taiwan change quickly, so this page lists none; check current quotes from resellers.
3. Four steps to choose hardware
1. Decide the task, then the model. Answering from company documents, organizing data, and drafting replies usually work with mid-sized models. There is no need to start with a machine for the largest model. To judge whether a model is good enough, see "Is a local model smart enough?"
2. Size GPU memory from the model file, with headroom. Ollama's default context is 4,096 tokens. Letting the AI read more document passages at once means raising the context, which uses more memory.
3. Count concurrent users separately. Ollama's FAQ states that OLLAMA_NUM_PARALLEL sets how many requests each model handles at once, and required memory scales with parallel requests times context length. When memory runs short, new requests queue. Ten people asking at once need more memory than one.
4. Place and configure the machine properly. An AI Agent that must respond at any time needs a machine that stays on 24 hours. Ollama binds to 127.0.0.1 port 11434 by default; if you open it to the office network, use a firewall to allow only specific devices, and do not expose it to the internet.
How Think BIG helps companies choose hardware for local models
Think BIG (Think BIG Make it Real Co., Ltd.) is an enterprise AI Agent deployment provider headquartered in Taiwan. Starting from protecting company data, we build AI Agents inside the client's own environment (company hardware, or a cloud server under the company's account). There are two deployment options: with the on-premises version, models and documents are processed on company hardware; with the cloud-assisted version, employees' questions and the document passages the system retrieves are sent to the model provider for processing.
- Check existing equipment first: we look at whether current computers can do the job before suggesting any purchase.
- Verify with real work, not just specs: before delivery we confirm which hardware the model actually uses (for example, whether the GPU is really running) and test response speed with the company's real questions.
- Pre-delivery security review: before each system is delivered, its code and configuration go through a security review; if it does not pass, it is not delivered.
See On-premises deployment and Enterprise.
The actual scope of Think BIG's services follows what both parties confirm.
FAQ
- Q1: For a local LLM, does the CPU or the graphics card matter more?
- Start with GPU memory. When the whole model fits, the GPU does the work; whatever does not fit goes to the CPU and slows things down noticeably. With Ollama, the PROCESSOR column in `ollama ps` shows the split.
- Q2: What can run on 16GB of GPU memory?
- By Ollama's official library sizes, 8B-class (about 5.2GB) and 14B-class (about 9.3GB) models fit. OpenAI also states that gpt-oss-20b runs within 16GB of memory. Leave room for context.
- Q3: How much memory if many staff use it at once?
- Calculate it separately. Ollama's FAQ states that required memory scales with the number of parallel requests times context length, and new requests queue when memory runs short.
- Q4: Can the local model machine be exposed to the internet?
- Not recommended. Ollama binds only to 127.0.0.1 by default. To let other office devices use it, open access to specific devices through a firewall rather than to the internet.
- Ollama FAQ (CPU/GPU split in ollama ps, default context 4096, OLLAMA_NUM_PARALLEL and memory, default binding 127.0.0.1:11434): docs.ollama.com
- Ollama library, qwen3 (download sizes and context): ollama.com
- Ollama library, gpt-oss (download sizes, 16GB and single 80GB GPU): ollama.com
- OpenAI gpt-oss on GitHub (117B/21B parameters, MXFP4, hardware): github.com
- Qwen3 on GitHub (model sizes including 30B-A3B): github.com
- NVIDIA GeForce RTX 5090 specifications (32GB GDDR7): www.nvidia.com
- NVIDIA DGX Spark specifications (128GB unified memory, up to 200B parameters): www.nvidia.com
This article summarizes public official documents, laws, and security standards to help companies evaluate AI adoption. It is not legal advice. For Taiwan law, the latest text on the Laws & Regulations Database prevails; external documents, prices, and specifications follow their original sources. The actual scope of Think BIG’s services depends on what both parties confirm.