When local makes sense – and when it does not
A local language model (LLM) runs on a computer inside your own network. Prompts, documents and answers stay there. That is the main reason small businesses are interested. What an LLM actually is, and why it does not "think", is a topic in its own right: in short, it is a statistical model that predicts plausible text.
Local usually makes sense when
- you work with text that should not end up with a cloud provider: HR matters, client or patient data, quotes, internal minutes,
- the internet connection is unreliable, or the system is meant to run offline on purpose,
- many people in the office do simple tasks now and then, and a licence per person does not pay off,
- you want to automate recurring jobs, for example summarising new documents every night.
Local is usually the wrong choice when you need the best possible quality. The large cloud models are clearly stronger at difficult writing, long documents, programming and reasoning than anything that fits on an office computer.
Model size and memory: the key calculation
Models are named after their number of parameters: 8B means eight billion. Locally you almost always run a compressed ("quantised") version with roughly 4 bits per parameter. Rules of thumb for that version:
| Model size | Memory for the model | Sensible hardware |
|---|---|---|
| 3–4B | approx. 2–3 GB | almost any current PC with 16 GB RAM |
| 7–8B | approx. 4–5 GB | 16 GB RAM or a graphics card with 8 GB |
| 12–14B | approx. 8–10 GB | graphics card with 12–16 GB |
| 27–32B | approx. 17–20 GB | graphics card with 24 GB or 32 GB+ unified memory |
| 70B | approx. 40–50 GB | 64 GB+ memory, very expensive |
On top of that comes memory for the context – the text the model "sees" at once. If you want to process long documents in one go, you need several gigabytes more. If the model does not fit entirely into graphics memory, the processor handles the rest. That works, but it is noticeably slower.
Without a graphics card: possible, but slow
A model can run on the processor (CPU) alone. Speed depends mainly on how fast the memory is, not on the number of cores. On an ordinary office PC with 32 GB RAM, a 7–8B model typically produces a few words per second. That is enough for:
- jobs that run in the background or overnight,
- short summaries and email drafts, if you are a little patient,
- a first test of whether local models can handle your texts at all – without buying any hardware.
If several people are to work smoothly in a chat window during the day, a graphics card or a machine with fast unified memory is the better choice.
Three typical setups and what they cost
| Setup | Suitable for | Hardware (approx., October 2026) |
|---|---|---|
| Existing PC, 32 GB RAM, CPU only | testing, background jobs, models up to 8B | SEK 0, possibly more RAM |
| PC with a 16 GB graphics card (e.g. RTX 5060 Ti 16 GB) | models up to approx. 14B, fast, several users taking turns | card approx. SEK 6,500–7,500, plus a suitable PC |
| Mini PC or workstation with lots of unified memory (e.g. with AMD Ryzen AI Max and 64–128 GB) | models up to approx. 32B and beyond, quiet, frugal | depends heavily on memory – compare current prices |
Hardware prices fluctuate a lot, especially for graphics cards and memory – check the current price before you buy. Then there is electricity: a PC with a graphics card draws a few hundred watts under load and much less when idle. Running around the clock, that is roughly a few hundred kronor a year.
By comparison, cloud licences are charged per person per month. With five people you quickly pass SEK 10,000 a year, every year. Whether a cloud assistant pays off is covered in Is Copilot for Microsoft 365 worth it?
Software: free, but not maintenance-free
A typical setup consists of a program that runs the models (for example Ollama or LM Studio) and a ChatGPT-style web interface (for example Open WebUI) that everyone in the office reaches through the browser. The models themselves can be downloaded freely, from families such as Llama, Mistral, Qwen and Gemma. Check the licence: many allow commercial use without further ado, some come with their own conditions.
What a local model is good for in the office
- Summarising minutes, long email threads and internal reports,
- drafting standard letters, quote texts and replies to common enquiries,
- rewording and translating between English, Swedish and German – usually good enough for internal use,
- questions about your own documents: manuals, procedures, templates. The model finds the relevant sections and answers from them. How well that works depends heavily on how tidy and up to date the documents are,
- structuring: notes into a table, bullet points into prose.
Clear, specific instructions matter even more with small local models than with the big cloud ones: give the context, the format you want and an example.
What it cannot do
- No current knowledge: the model only knows what was in its training data and does not search the web.
- It makes things up: small models invent facts, legal references and figures more often than large ones. Anything that goes to customers needs a human check.
- Arithmetic and logic: for calculations, tax questions or contracts it is not a reliable tool.
- Very long documents: a 200-page contract will not fit in one piece with little memory.
- It does not run itself: without someone looking after it, it quickly becomes outdated.
Operation, updates and data protection
A local model is a server like any other and needs the same care:
- Reachable on the local network only, no port forwarding to the internet. Access from outside via VPN only.
- A login for every user, with a separate administrator account.
- Updates for the operating system, Ollama and the web interface on a regular schedule. Test new model versions first, then switch.
- Chat histories are data like any other: who can see them, how long are they kept, are they in the backup?
"Local" does not automatically mean GDPR-compliant. Personal data is still personal data on your own server – the only thing that disappears is the cloud provider as a recipient. The EU AI Act (Article 4) also addresses staff understanding of the AI tools they use. In a small office, a short introduction and a written rule on what the model may be used for go a long way. This is not legal advice.
A note on licences: I generally recommend one-off purchased software over subscriptions, above all Office as a purchased version and Windows Server: no running licence costs, programs and data stay usable, no price rises, predictable and local. To be honest: purchased versions get security updates only for a fixed period, and email or video conferencing are services that have to run. Anyone who definitely wants Microsoft 365 will of course get it set up by me too. The trade-off is in Microsoft 365 or an alternative for small businesses.
Checklist before you buy
- Write down three to five specific tasks the model should handle.
- Test with real (anonymised) sample texts on an existing PC before buying hardware.
- Decide how many people will use it at the same time.
- Choose the model size first – then the hardware.
- Agree who installs updates and who is the contact person.
Frequently asked questions
Do I need a graphics card?
No. Small models up to about 8B run on the processor if there is enough memory (32 GB is comfortable). For several people to work smoothly, a 16 GB graphics card or a machine with lots of unified memory is clearly better.
Is a local model as good as ChatGPT?
No. For summaries, drafts and internal texts it is often enough, but on difficult tasks the large cloud models are clearly ahead. So test with your own texts first.
Is it automatically GDPR-compliant if everything is local?
No. Only the external provider disappears. You still need to manage access rights, how long chat histories are kept, and backups.
Does it work without internet?
Yes. Internet is only needed to download software and models and for updates. After that the system runs fully offline.
No affiliate links. Recommendations are based purely on hands-on experience.
Want to know whether a local language model can handle your texts? I test it with your sample documents, recommend hardware and set everything up – the "Local language model – trial setup" package for SEK 5,000 excl. VAT (hardware not included). In English, Swedish or German; reply within 24 hours.
IT consultant in Sundsvall Book a free call