Local AI · On-premise LLM · Private AI
Your own AI on your own server,
from hardware to application.
I run open language models on a server in your office or data centre. Because I also look after infrastructure, I plan hardware, virtualisation, network and backup together with the AI application that runs on top.
Discuss your projectFree initial conversation · Directly with Daniel Gläser
- 01Internal request
- 02Model on-site
- 03Review result
- 04Data stays put
Local or cloud
Local AI is not always worth it. Sometimes it is the only fitting option.
The decision depends on the kind of data, how much AI is used, the quality required and who runs the system. The guide on local versus cloud AI works through the costs of both.
Local makes sense when …
Prompts and documents stay on your own network. No data goes to an AI provider for using the model.
- sensitive data is involved, such as HR, health or client records.
- professionals bound by confidentiality, such as law firms, tax advisers or medical practices, want to use AI.
- many staff use AI every day and predictable costs matter more than per-seat billing.
- systems have to work offline or on a network separated from the internet.
An EU cloud is better when …
A cloud service with a suitable contract and EU hosting can be the simpler choice. Data flows and agreements are reviewed for the specific use.
- the task needs the strongest available models, for example for complex reasoning.
- AI is used only occasionally and a server would sit idle most of the time.
- nobody should have to look after hardware, updates and monitoring.
- a quick start without buying hardware is the priority.
Building blocks
What a local AI setup consists of.
Local AI is built in layers. Every tool named here is an example; the choice follows your tasks and your hardware.
Open language models
Common open model families include Qwen from Alibaba, Gemma from Google, Mistral, gpt-oss from OpenAI and Llama from Meta, each in several sizes. Licences differ: Gemma 4, gpt-oss and many Qwen and Mistral models are released under Apache 2.0. For the multimodal Llama 4 models, Meta's acceptable use policy grants no licence rights to companies with their principal place of business in the EU. I check the licence of every model before it is used.
Runtime
The runtime loads the model and serves it through an API. Ollama is easy to set up, llama.cpp also runs on hardware without a large graphics card, and vLLM is designed for many concurrent requests on GPU servers.
Interface with user management
Your team works in a browser chat, e.g. Open WebUI. Users, groups and permissions control who may use which models and knowledge bases. Depending on your environment, sign-in can be linked to existing accounts. Open WebUI has its own licence too, and I check its terms beforehand.
Connecting your documents
With RAG, the model searches approved manuals, contracts or policies and answers with a source reference. It works locally just as it does in the cloud, and the documents stay on your server.
AI chatbotConnecting workflows
The local model can also work inside automated workflows, for example to sort emails or summarise documents. The runtime's API is the connection point.
Workflow automationHardware & infrastructure
Hardware decides which models can run.
This is where my focus lies: I plan the AI application and the infrastructure beneath it, rather than you buying both separately from different providers.
Memory sets the model size
A model plus its context has to fit into fast graphics memory or unified memory. More memory allows larger models and longer documents. Quantised models need less memory at the cost of some accuracy.
Plan for concurrent use
Whether five or fifty people send requests at the same time changes what is needed in compute, memory and runtime. We size for realistic use and test it in the pilot.
Virtualisation & network
The AI can run on dedicated hardware or as a virtual machine on Proxmox with the graphics card passed through. On the network it gets its own segment and is reachable only from authorised devices.
IT infrastructureAn honest cost comparison
Purchase, electricity, maintenance and whether the hardware is actually used all belong in the comparison with cloud subscriptions. The guide contains a worked example with its assumptions.
Read the cost comparisonOperations
Your own AI server needs care like any other server.
Updates
Operating system, drivers, runtime and interface receive regular updates. New model versions are tested on your tasks before they replace the old one.
Monitoring
Load, memory, temperature and response times are monitored. If the service stops or storage fills up, someone is notified.
Backup
Configuration, user accounts, chat histories where wanted and the knowledge base are backed up. The models themselves can be downloaded again. Restores are tested.
Internal access only
The AI is reachable only on the internal network, and via VPN when travelling. It is not exposed to the open internet.
Permissions & retention
We define who may see which knowledge bases, how long chat histories are kept and who can view logs.
Quality & data protection
What local models do well, and where they reach their limits.
For many office tasks, open models are now genuinely useful: summarising text, drafting emails, tidying up minutes, translating or answering questions about your own documents. For complex multi-step reasoning, the strongest cloud models are usually ahead. Whether a local model is good enough for your tasks is shown not by a leaderboard but by a pilot with your own examples.
Running locally simplifies data protection because no data goes to an AI provider for model use. It does not make it unnecessary. Legal basis, access rights and deletion periods are reviewed for the specific use. If a service provider has access to data during maintenance, a data processing agreement may be required. Many businesses combine both: sensitive tasks locally, selected demanding tasks in an EU cloud.
Working together
A clear path to everyday use.
Clarify tasks and data
Record use cases, types of data, number of users and access requirements.
Test models in a pilot
Compare suitable open models on your own tasks, on test or loan hardware.
Build the server
Set up hardware, virtualisation, network segment, interface, permissions and backup.
Run and improve
Plan updates, monitoring and model changes; review feedback from the team.
Before we begin
Good questions. Clear answers.
What is local AI?
Local AI means a language model runs on a server in your own business or a rented data centre rather than with a cloud provider. Prompts and documents stay on your own network. It uses open models whose weights may be downloaded and run independently.
Is there an internal alternative to ChatGPT?
Yes. An interface such as Open WebUI combined with an open language model gives you a browser chat that works in a similar way and runs on your server. It does not match the strongest cloud models in every task, but for summaries, drafts and questions about your own documents it is often enough.
What hardware do I need to run my own language model?
Fast memory is what counts: the graphics memory of a GPU or unified memory determines how large a model can run. The number of concurrent users and the length of documents matter too. I size the hardware for your tasks and test it in a pilot first.
May I use open models commercially?
That depends on the model. Many current models are released under Apache 2.0, which allows commercial use. Others have their own licences with restrictions, for example the multimodal Llama 4 models for companies based in the EU. The licence is checked for every model before use.
Does local AI automatically meet GDPR requirements?
No. It avoids passing data to an AI provider, but the other obligations remain, such as legal basis, access control and deletion periods. These are reviewed for the specific use.
What does an in-house AI server cost?
Cost factors are hardware, electricity, setup and ongoing maintenance. How the total compares with cloud subscriptions depends mainly on the number of users. We start with a pilot whose effort is agreed before implementation and decide on the purchase afterwards.