Case study · AI & LLM integration

Private AI translation with self-hosted LLMs

How I built AI-powered translation for a large multilingual organisation that couldn't send its documents to a public AI service: open models served with Ollama and llama.cpp, a Node.js backend, a Vue 3 / Nuxt interface and AWS, with every document staying in-house.

Client
Large multilingual organisation (confidential)
Role
Senior full-stack engineer, contract
Stack
Ollama · llama.cpp · Node.js · Vue 3 · Nuxt · MongoDB · AWS Lambda & EC2
Focus
Data privacy, internal tools, AI translation

The challenge: AI translation without leaking data

Teams worked with documents in many languages every day. Commercial AI translators were fast, but every request meant sending internal, sometimes sensitive text to a third party, which the organisation's data-protection rules didn't allow.

The brief was clear: give people modern, AI-quality translation inside the tools they already use, while making sure no document ever leaves infrastructure the organisation controls.

The approach: a self-hosted LLM behind a lean Node.js service

Instead of calling a public API, the models run on servers inside the organisation's own cloud account. A small Node.js service sits in front of them and is the only thing the rest of the system talks to.

Architecture of the private AI translation system A Vue 3 and Nuxt interface sends requests to a Node.js API. The API queues jobs on AWS Lambda, which calls Ollama or llama.cpp running open models on a private EC2 server, with results cached in MongoDB. Everything runs inside the organisation's own cloud account. Your cloud account · data never leaves Vue 3 / Nuxt internal tools Node.js API auth · routing AWS Lambda job queue Ollama / llama.cpp open LLMs · EC2 GPU MongoDB cache · history
Every component runs inside the organisation's own cloud account.

Key decisions

  • Ollama for model serving. It runs open models behind one stable API, so models can be compared and swapped without touching application code.
  • llama.cpp where efficiency matters. For lean, high-throughput workloads, llama.cpp runs quantised models (GGUF) with lower memory and CPU/GPU overhead, so more work fits on the same server.
  • A queue in front of the GPU. Translation requests go through AWS Lambda jobs, so busy moments queue up gracefully instead of overloading the model server.
  • Caching repeated work. Many documents share phrases and boilerplate. Results are stored in MongoDB, so the same text is never translated twice.
  • A lean Node.js layer. Authentication, logging and routing live in one small service with few dependencies. That means less to audit, which matters as much as the model itself for security.
  • Built into existing tools. The Vue 3 / Nuxt interface plugs into the internal apps people already used, so adoption didn't depend on learning a new product.

What it delivered

  • Documents stay in-house. No text is sent to an external AI provider, which keeps the data-protection review simple.
  • AI-quality translation inside existing tools. People translate where they already work, not in a separate app.
  • Model-independent. Better open models can be adopted as they're released, with no changes to the apps.
  • Predictable costs. Costs are fixed infrastructure instead of per-request API fees that grow with every document.

Self-hosted LLM vs Claude API: when to use which

Self-hosting isn't always the right answer. For most companies the best setup mixes both:

Self-hosted (Ollama / llama.cpp)

  • Sensitive or regulated data (GDPR, contracts, health, finance)
  • High-volume, repetitive tasks such as translation, classification and summaries
  • Predictable, fixed infrastructure cost

Claude API

  • AI agents that use tools and work across your systems (MCP)
  • Complex reasoning, long documents and code
  • Fast to start, no GPU servers to run

I build both, and often route each request to the right model automatically: private data goes to the self-hosted model, complex agent work goes to Claude.

FAQ

Is a self-hosted LLM GDPR compliant?

Self-hosting doesn't make a system compliant on its own, but it removes the biggest risk: personal or confidential data being sent to a third-party AI provider. When the model runs on infrastructure you control, in a region you choose, data residency, retention and access are under your rules, which makes a GDPR assessment much simpler.

Is Ollama ready for production use?

Yes, for internal tools with predictable load. Ollama makes it simple to run and swap open models behind a stable API. For production you add what it doesn't do for you: a queue in front of it, authentication, monitoring, caching of repeated requests and sizing the GPU server for your peak load.

Ollama or llama.cpp: which should I use?

Ollama is the fastest way to get open models running behind a simple API, with easy model management, so it's ideal for getting started and for internal tools. llama.cpp is the engine underneath many local-LLM tools: using it directly gives finer control over quantisation, memory and performance, which pays off for high-volume or resource-constrained deployments. A common path is to prototype with Ollama and move hot paths to llama.cpp.

Self-hosted LLM or Claude API: which should I choose?

Use a self-hosted model when data must stay in-house or volume is high and repetitive, like translating internal documents. Use Claude when you need stronger reasoning, long context or tool use, for example AI agents that work across your systems. Many products combine both, routing each request to the right model.

Can you add private AI features to an existing Node.js or Vue app?

Yes. The AI part usually lives behind a small Node.js service with its own API, so it can be added to an existing Vue, Nuxt or React product without a rewrite, and the model behind it can be changed later without touching the frontend.