Is a Local LLM Smart Enough, and How Do You Tell Whether It Works for Your Company?

1. Why leaderboards are not enough

Leaderboards measure general ability. Your company needs answers from your documents, in your language, to your questions. OpenAI's evaluation best practices say the same: design task-specific evals that reflect real-world usage. Anti-patterns it lists include relying only on generic academic metrics, test data that does not match production traffic, and treating "it seems to work" as a pass.

So the question is not "which model is stronger" but "on our 50 questions, how many does it get right, where does it fail, and what does each failure cost?"

2. Three configuration reasons a local model seems "dumb"

Before concluding anything, rule out problems unrelated to model ability:

1. Context too short for the documents. Ollama's default context is 4,096 tokens. When AI searches company documents, retrieved passages go into the context; what does not fit is cut off, and the model answers from incomplete material. Raise it with OLLAMA_CONTEXT_LENGTH or num_ctx, at the cost of more memory (see "What hardware does a local LLM need?").
2. Model not fully on the GPU. If the PROCESSOR column in ollama ps shows a CPU/GPU split, part of the model runs on the CPU and responses slow down noticeably. That is a speed problem, not an intelligence problem, but it is often mistaken for "local models don't work".
3. Reasoning settings left at default. Some models let you set reasoning effort; OpenAI's gpt-oss accepts low, medium, or high in the system prompt. Testing multi-step questions at low effort is not a fair test.

3. A five-step comparison test

1. Collect questions. Pick 30 to 50 from what employees actually ask, customer service logs, and internal email. Include three kinds: answerable from the documents; not in the documents, where the AI should say it doesn't know or hand off; and questions the user has no permission to see.
2. Write correct answers and sources. For each question, record the answer and which passage of which document it comes from.
3. Run both under the same conditions. Same documents, same retrieval settings, same questions for the local and cloud models.
4. Score question by question. Use four categories (correct, partly correct, wrong, answered when it should have declined) and record response time. Where it fails matters more than the total.
5. Set the threshold in advance. For example, a required share of correct answers on answerable questions, and zero made-up answers on questions that should be declined. Adopt only if met. When you change models or settings later, rerun the same set.

The usual outcome is not "all local" or "all cloud" but a split: which work goes to the local model, and which work, where the data allows, uses the cloud-assisted version.