Cost & TCO

On-premise AI cost: what a private AI server really costs.

On-premise AI costs €3,500 to €4,500 in hardware for a small-team box, and €13,000 to €16,000 for a production server; builds start from €4,000 on top. A small deployment lands near €8,000 all in, a departmental one between €20,000 and €30,000: one time, not per month.

Search this question and you get two answers, both useless to you. One camp prices H100 clusters and lands at €250,000 to €600,000, which is what an enterprise AI platform costs. The other camp does token-per-day break-even math with American electricity rates, which is what an engineer optimising an API bill needs. Almost nobody prices the thing a 20 to 200 person European company actually buys.

So here are our numbers. We scope and build these systems, in euros, in the EU, and the ranges below are what we quote. More useful than the prices: the reason most of these projects come in over budget has nothing to do with the price of a GPU.

The short answer, in euros.

Three build tiers cover nearly every business we talk to. The small box gives you 48GB of VRAM across the pair, which comfortably runs a 24B to 32B class open model at good quality. Ten people using it at once is not fine, for reasons I’ll get to. The middle tier pays most of the premium of the professional tier without buying the headroom that makes the premium worth paying. The production box is the tier where a promise about fifteen concurrent people is one we’re willing to put in writing.

On top of hardware sits the build: indexing your documents, wiring the model into the systems your team already uses, access control, audit logging, and the fine-tuning if your use case needs it. Our builds start from €4,000 and scope up from there depending on how deep the integration goes.

So a real entry point for a small team is roughly €8,000 all in. A production deployment for a department lands between €20,000 and €30,000. Those are the numbers, and they are one time, not per month. Now the part that decides whether you spend €8,000 or €25,000, and it isn’t the model you want to run.

Parameter count is the wrong spec. Concurrency is the right one.

Every hardware guide sizes the box by model size. “96GB runs a 70B model.” True, and misleading, because it describes one person asking one question with an empty context.

Here’s what actually happens on a GPU serving a team. The model weights load once and sit there: a 32B model at 4-bit quantisation takes roughly 20GB and never moves. Everything left over goes to the KV cache, which is the working memory of each live conversation. Every concurrent user, every long document pasted into context, every agent running in the background allocates its own slice of that leftover space, and the slice grows with context length.

Do the arithmetic on the small box. 48GB total, 20GB of weights, 28GB free. A user working with an 8,000 token context might take 1.5GB to 3GB depending on the model architecture. Fifteen of those and you’re out of memory, except you don’t get a clean error. You get requests queuing, then timing out, then a team that quietly goes back to pasting client data into ChatGPT because the internal one “is slow.” That’s the failure mode. Not a crash. Erosion.

There’s a second version of the same mistake on the unified-memory desk boxes that get recommended constantly. Yes, 96GB or 128GB of unified memory will load a 70B model. It will also generate at a handful of tokens per second at that size, which is tolerable for one patient person and unusable for a support team. Memory capacity tells you what fits. Memory bandwidth tells you how fast it moves. The marketing quotes the first number.

The practical rule we use when scoping: count your peak concurrent users, not your headcount. A 40 person company where six people use AI heavily and everyone else asks it something twice a week has a peak concurrency of about eight. That’s a very different box than 40 seats implies, and it’s usually cheaper than the client expected.

The four lines on the bill, and the one that isn’t there.

• Hardware, once. The numbers above. This is equipment. It depreciates on your books, the VAT is reclaimable, and it doesn’t reprice itself because a vendor changed a pricing page. Plan a three to five year life, same as any server.
• The build, once. Getting documents indexed into a retrieval pipeline that gives correct answers with citations is real engineering. So is network isolation, so are roles and permissions, so is fine-tuning a model on your contracts or tickets. This line varies more than the hardware line, which is why we scope it against your actual documents before quoting a number.
• Power, monthly, and smaller than you fear. Take the small box: call it 900W under sustained load, but it isn’t under sustained load all day. An office system doing real work might average 200 to 300 kWh a month including idle. At the €0.15 to €0.25 per kWh range most EU commercial customers pay, that’s €30 to €75 a month. The production box roughly doubles it. This is a real line, not a scary one, and it’s a rounding error against the API bill it replaces. Rack-scale clusters are a different conversation; you are not buying a rack-scale cluster.
• Ops, monthly. Patches, model upgrades, monitoring, backups. Somebody owns this or the box becomes a liability. Either your IT people take it or we run it. What it is not is a per-seat licence that grows when you hire.

The line that isn’t there is the interesting one: there’s no meter. Once the system is live, the marginal cost of the next question is zero. Not “low”. Zero. Your team can dump a 300 page tender into it forty times in an afternoon and the bill doesn’t move. That changes behaviour more than any feature does, because people stop rationing.

The serving stack is a cost decision disguised as a technical one.

This one costs European SMEs real money and I’ve never seen it in a cost guide. Almost everyone starts with Ollama, because it’s two commands and it works. For one person on a laptop, it’s the right tool. Under concurrent load it handles requests poorly, and the visible symptom is that your box feels underpowered. The invisible consequence is that somebody concludes they need to buy a bigger GPU. The other four gaps that tool leaves open are in Ollama for business.

Swap in a proper inference server, vLLM or SGLang, and the same silicon serves several times as many people, because continuous batching lets the GPU work on many sequences at once instead of queuing them. That’s not a tuning tweak. That’s the difference between the €4,000 box being enough and the €14,000 box being necessary.

We’ve watched a client’s spec drop a full tier from this change alone. Before you price hardware, price your serving stack, because the wrong default is the most expensive free software you’ll ever install.

The euro lines that American cost guides skip.

Electricity is two to three times US rates. A guide written around $0.13 per kWh understates your power line by a factor of two. It’s still small. It’s just not the number on the page you’re reading.

VAT and capex treatment. A server is a capital asset. Your accountant handles it the way they handle the rest of the equipment, and the VAT comes back. Subscriptions are operating expense that scales with headcount forever. That distinction lands differently with a Greek or German CFO than a cumulative spend chart does.

The legal work that stops happening. This is the biggest uncounted saving on the EU side. Sending company data to a US-hosted model means a data processing agreement, a review of the transfer mechanism, a vendor security assessment, and a policy telling employees what they may and may not paste. Every one of those is billable hours, and they repeat whenever the vendor changes a sub-processor. When inference happens on a machine in your own rack, the question “where did our client’s data go” has a one word answer, and most of that paperwork disappears. We wrote up the detail in our guide to GDPR and LLM hosting.

Compliance timing. EU AI Act transparency obligations have applied since August 2026, with high-risk obligations following in December 2027. If you’re going to face those questions anyway, the architecture you pick now decides how hard they are to answer. Our page on EU AI Act obligations has the timeline.

Break-even, calculated honestly.

Most break-even charts in this category are sales collateral. Here’s the version that will occasionally talk you out of buying. Take your actual current AI spend. Not a hypothetical seat count: the real number on the card statement, API metering plus per-seat tools plus whatever three departments expensed separately. Call it X per month.

Which is fine, because for most of our clients cost isn’t the reason. The reason is that a category of work can’t go to a cloud model at any price: the privileged file, the client contract under NDA, the tender documents, the patient records, the source code. A subscription buys you nothing for that work. If the highest value use case in your business is the one you’re not allowed to run, the comparison isn’t €8,000 against €250 a month. It’s €8,000 against not doing it.

And the honest inverse: if you have no compliance driver, a small API bill, and no data that can’t leave the building, stay on the cloud. We say this in assessments and we’ve written down the criteria in when not to self-host an LLM. An unused server is a worse outcome than a subscription.

Rent the exact card before you buy it.

The most expensive mistake in this space isn’t overpaying for a GPU. It’s buying the wrong one, and the fix costs about €300.

Professional GPUs rent by the hour, roughly €1 to €2 for the class of card in the production tier. Rent the one you think you need. Load the model you think you’ll run. Point it at your real documents, not a demo dataset, and have five colleagues hammer it at the same time on a Tuesday afternoon. Measure tokens per second at your actual concurrency, with your actual context lengths.

A hundred and fifty hours of that costs a few hundred euros and answers the only question that matters: which tier do you actually need. It also occasionally reveals that a fine-tuned 24B model does your job as well as the 70B you were about to buy hardware for, which is the single largest cost saving available in this whole exercise. We build this into engagements as a matter of course. Test on rented iron, buy once, buy right.

The GPU market makes this more valuable than it was two years ago. Memory supply has been tight, prices move month to month, and configurations have been quietly discontinued mid-year. A quote in this market has a shelf life. Measuring twice is worth more than it used to be.

Where the budget usually breaks.

Three overruns account for most of them, and none is a hardware price.

• Buying before measuring. A box arrives, then the use case gets defined around it. Sometimes it fits. Often you own a €14,000 machine running a workload a €4,000 machine would have handled, or the reverse, which is worse.
• Sizing for the biggest model instead of the real task. Extraction from your documents, drafting in your formats, classifying your tickets: these are scoped tasks, and a well fine-tuned mid-size model handles them at a quality your team will accept. Frontier reasoning is a different job, and for that, cloud is still ahead. We'll tell you when your task is in that category.
• Nobody owning ops. The system runs beautifully for four months, a model release makes something better available, nobody upgrades, and by month twelve your team thinks local AI is mediocre. It isn't. Yours got old. Budget the ops line or hand it to someone who will.

What to do with these numbers.

Take your last three months of AI spend, count your realistic peak concurrent users rather than your headcount, and write down which of your documents legally cannot go to a US cloud. Those three figures determine your tier, and you can work them out this week without talking to anyone.

If the answer looks like a small box, you can run a local LLM for your business for less than a mid-range company car and stop metering your team’s curiosity.

If it looks bigger, or if you want the arithmetic done against your real numbers rather than our ranges, book an assessment. Thirty minutes, and you get a written plan with real costs before anything is built. Sometimes that plan says the cloud is fine. We put that in writing too.