Services
AI & Software
Solutions
Training
Tools
Insights
References
Contact
Free initial consultation Ask a first question by email
About Deutsch
← AI & Software

Run local AI

An open-source model on your own hardware: selection, setup, connection and operation. Data stays in the building, without a public model API.

The server sits in the back room, the GPU fans are running, and in the project office someone asks for the third time whether the invoices really do not leave the building. A local model is not a diet version of the cloud. For many tasks in a business it is the better choice, because the data stays in the building and the cost is predictable. It is not self-running, though: hardware, updates and quality checks are part of it.

How I work: with source · with approval · in your system.

Starting point

The reasons for local operation are usually the same: customer data, personnel files or design documents must not leave the building. Or cloud cost per task is not viable at high volume. Sometimes the internet connection is the bottleneck. In all three cases the question is not “local model yes or no”, but which task has to run locally and which does not.

What I do

First I clarify which tasks the model should solve and what quality that needs. Then the hardware: a single GPU carries an assistant with sources and classification. Larger models with long contexts cost more, and at some point the bill is more honest than the cloud.

Then I set up the service: model server, connection, permissions, monitoring, update path. Where it makes sense, everything runs in containers so an update does not take down the whole system. The service gets a private connection, not a public address.

The connection includes the fallback. If the model fails, the application must keep working: for text tasks through rule-based processing, for questions through a marked text search. A silent failure that invents plausible answers would be worse than a visible gap.

Handover includes an operating concept: who restarts the service, how a model switch is tested, what happens on failure. A local system without these answers is an experiment.

Result

You get a running local AI service, connected to the intended systems, plus an update and operating concept and the agreed fallback behaviour. On request with a test catalogue that secures later model changes.

Example

Three systems in operation, none with a public model API:

My own agent runs locally and does without a cloud; the reasoning is in the article on OpenClaw. The knowledge assistant runs as a standalone package on its own GPU, privately connected, answering questions from a versioned knowledge vault. In my own ERP a local model handles invoice and text tasks; if the service is missing, rule-based processing takes over so invoice capture does not stall.

All three are my own projects, not client work. They show that the operation works, not what it would look like in your company.

Limit

Local models are often weaker than the largest cloud models on complex texts and rare domain questions. Hardware costs money once and needs power, cooling and someone to maintain it. I do not give an availability SLA for this operation; that is a decision for the client.

The second limit is speed. A model on a single GPU answers slower than a cloud service with many machines behind it. For an assistant in day-to-day project work that is enough; for a chat with many parallel users it becomes a patience problem, and therefore a capacity plan.

How it runs

  1. Clarify tasks and quality requirements
  2. Select hardware and model, compare cost
  3. Set up the service, connect privately, set permissions
  4. Test questions from operations, compare with cloud candidates
  5. Define the fallback, hand over the operating concept

Common questions

Which hardware do I need?

An assistant with sources and medium-sized models usually needs one GPU with enough memory. The exact size depends on the model and context and is calculated before purchase.

Is the quality good enough?

For classification, summaries and questions about your own documents, often yes. For difficult domain texts you have to measure. The test catalogue shows the difference before a decision.

What happens on failure?

The service fails, the application keeps running and reports “unknown” or uses a rule-based fallback. A silent failure would be worse than a visible gap.

Can we move to the cloud later?

Yes, if the connection is encapsulated. The switch is then a configuration question, not a new build. The reverse holds too: if you start in the cloud, individual tasks can move local later.

The next step costs nothing.

30 to 60 minutes, no strings attached: where is the biggest friction, what is the most effective lever, and is a collaboration a fit?