THINK BIG
Local AI · Context Length · Memory Management
Give Your Local AI Too Much “Memory” and It Can Drag Down the Whole Machine — How to Size the Context Length
If a local AI keeps getting slower and the rest of the computer bogs down with it, an unused GPU is one suspect. An oversized context length is another. Run ollama ps and check two columns, PROCESSOR and CONTEXT, to see how much context is actually allocated and whether the model has spilled onto the CPU.
(We use the open-source tool Ollama as the example because its documentation spells out the defaults and the checks clearly. Other local runtimes work along similar lines under different setting names.)
This piece pairs with our Issue 2, “Running a Model Locally: Is Your GPU Actually Doing Anything?” Issue 2 covered models that never touch the GPU. This one covers the other case: the GPU is in use, but the context is set so large that memory runs out. Both start with the same command.
One. What’s Happening: Context Takes Memory Too
Context length is how much material the model can carry while it answers, measured in tokens. Ollama defines it as the maximum number of tokens the model has access to in memory.[2]
It isn’t free. Ollama’s documentation states that a larger context length increases the memory needed to run a model, and that you should make sure you have enough VRAM before raising it.[2] The FAQ adds that when a model serves several requests at once, required memory scales with parallel requests multiplied by context length; the default for parallel requests is 1.[1]
Watch the default. Ollama’s FAQ says the default context window is 4,096 tokens.[1] The separate “Context length” page describes VRAM-based defaults instead: 4k below 24 GiB, 32k from 24 to 48 GiB, and 256k at 48 GiB or more.[2] Because the two pages differ, we don’t rely on either from memory. We read the CONTEXT column in ollama ps and go by what the machine has actually allocated.[2]
Changing it is documented too. To change the server-wide default, set the OLLAMA_CONTEXT_LENGTH environment variable. To change it for one model or one session, use /set parameter num_ctx in the CLI, or pass num_ctx in the API options.[1]
Two. Perspectives: Go Big, or Stay Stable?
The case for larger contexts: Ollama’s “Context length” page recommends at least 64,000 tokens for context-heavy tasks such as web search, agents, and coding tools. The same page says that for best performance you can use a model’s maximum context length while avoiding offloading the model to the CPU.[2]
The case for counting the cost: The FAQ explains that Flash Attention can significantly reduce memory use as context grows, and that the K/V cache (where context is held) can be quantized. q8_0 uses roughly half the memory of f16, and q4_0 roughly a quarter, but precision loss may be more noticeable at larger context sizes, and the impact depends on the model and task.[1]
These views don’t really conflict. The documentation’s assumption throughout is that the model fits: open the context up, but check the memory budget first and confirm nothing has been pushed to the CPU.[2]
Three. What We’ve Learned in Practice
These are observations from our own local AI agent deployments. They are not industry statistics and don’t describe any specific site.
1. “It’s getting slower — is the GPU too old?” is a common question. The common answer is configuration. A pattern we see often: someone raises the context sharply so the AI “remembers more,” memory can’t hold it, part of the model ends up on the CPU, and response times stretch out.
2. Be extra careful on unified-memory machines. When the graphics chip and the system share one pool of memory, context isn’t only eating “the GPU’s share.” The symptoms we commonly see: responses get progressively slower, the whole computer turns sluggish, and the disk starts writing heavily to swap. At that point it isn’t just the AI that’s slow; every other program is being held back too.
3. Measure the need before picking a number. We don’t start with a fixed figure. The steadier sequence is to run a few real tasks, see how many tokens each actually sends, and then decide how much headroom to leave. Ollama’s API responses include prompt_eval_count, the number of input tokens in that request, which is a good enough basis.[3]
4. Check CONTEXT and PROCESSOR before handover. In ollama ps, PROCESSOR shows 100% GPU, 100% CPU, or a split between them;[1] CONTEXT shows the allocated context length.[2] We screenshot both columns at handover.
Four. Our Recommendations: Three Steps
1. Measure, then set. Take the three to five tasks you run most often (answering support emails, summarizing meeting notes, searching internal documents), run each a few times, and record prompt_eval_count.[3] Take the highest value, add room for the answer, and start there.
2. Run ollama ps right after any change. If PROCESSOR shows any CPU share, the model plus its context no longer fits in VRAM.[1] Bring the context back down, or look at Flash Attention or K/V cache quantization. If you quantize, re-check quality on your real tasks.[1]
3. Size parallelism and context together. If several people will use it at once, memory needs multiply.[1] Decide how many concurrent requests you need first, then set the context.
In short: a big context is fine if the machine can hold it. If it can’t, the whole computer slows down with it.
Measure first, then set. Every setting should answer “why this number?”
(The following is Think BIG’s own practice, kept separate from the reporting above.)
When we deploy local AI agents, context length isn’t a guess: set from real workloads — we measure input size on the client’s own tasks before fixing context length and parallelism; leave evidence at handover — screenshots of the PROCESSOR and CONTEXT columns from ollama ps, handed to the owner with the settings; re-verify after changes — after adjusting context, quantization, or parallelism, we rerun the same tasks to confirm speed and answer quality are still acceptable.
Our position: every setting should come with an answer to “why this number?” Issue 13 extends this to model selection, using the same logic as a foundation.
Frequently Asked Questions
- Q1: What is context length?
- The amount of material the AI can carry while it answers, counted in tokens. Ollama defines it as the maximum number of tokens the model has access to in memory.[2]
- Q2: What is Ollama’s default context length?
- The FAQ says 4,096 tokens; the “Context length” page describes VRAM-based defaults of 4k, 32k, or 256k.[1][2] Because they differ, check the CONTEXT column in
ollama psfor what your machine actually uses.[2] - Q3: How do I change it?
- Use the
OLLAMA_CONTEXT_LENGTHenvironment variable for the server-wide default, andnum_ctxfor a single model or request (/set parameter num_ctxin the CLI, or the API options).[1] - Q4: Is a bigger context always better?
- Ollama recommends at least 64,000 tokens for agents and coding tools, and also notes that larger contexts need more memory, so confirm you have enough VRAM.[2] Our advice is to measure your actual need first.
- Q5: What if memory is tight but I need a longer context?
- Ollama’s documentation points to Flash Attention and K/V cache quantization (
q8_0at roughly half,q4_0at roughly a quarter off16memory). How much quantization affects quality depends on the model and task, so test it yourself.[1] - Q6: Is this the same problem as “the GPU isn’t being used”?
- Not quite, but you check it the same way. Our Issue 2 covered a model not using the GPU; this article covers a context too large for memory. In both cases, start with the PROCESSOR column in
ollama ps.[1]
[1] Ollama, “FAQ” (context window settings, the PROCESSOR column in
ollama ps, OLLAMA_NUM_PARALLEL, Flash Attention, K/V cache quantization); accessed October 24, 2026: docs.ollama.com/faq ·
[2] Ollama, “Context length” (definition, VRAM-based defaults, recommendation for agent tasks, the CONTEXT column in ollama ps); accessed October 24, 2026: docs.ollama.com/context-length ·
[3] Ollama, “Usage” (API response fields including prompt_eval_count); accessed October 24, 2026: docs.ollama.com/api/usage
The first-hand lessons, recommendations, and editor’s note present Think BIG’s practical views and do not constitute industry statistics or evidence of customer outcomes.