Setting up Ollama in Docker properly: security, GPUs, performance and operations
Part 1 got a model running on your own machine with one docker run command. That’s perfect for experimenting, but not for a server your team or your apps depend on.
The biggest reason is security. Ollama has no authentication. Anyone who can reach port 11434 can use your hardware, download or delete your models, and read whatever comes back. Security researchers have repeatedly found exposed Ollama servers on the internet: Cisco Talos found over 1,100 in 2025, and later scans counted far more.
This part builds a setup you can rely on:
- Ollama is never directly reachable; it sits on a private Docker network.
- A reverse proxy adds TLS and a token, and blocks the model management endpoints.
- Models are pre-downloaded, the container is health-checked, and versions are pinned.
- Memory, GPU and concurrency settings are chosen on purpose, not left to chance.
The target setup
1. First decide: who actually needs to reach it?
The safest Ollama is one nobody else can reach. Choose the smallest option that works:
- Only apps on the same server: put them on the same Docker network and publish no ports at all. No proxy is needed.
- People or apps inside your organisation: keep it on the internal network or behind a VPN (or a private network such as Tailscale), and add the token proxy below.
- Apps on the internet: only with TLS, a token, rate limiting and blocked admin endpoints, and ideally an IP allow list. Think hard before choosing this option.
A Linux trap to know about: ports published by Docker bypass UFW firewall rules, because Docker writes its own iptables rules. “The firewall blocks 11434” doesn’t help if Compose publishes it. The reliable fix is not publishing the port.
2. The Docker Compose file
Create a folder with these files: compose.yaml, Caddyfile and .env.
# compose.yaml
name: llm
services:
ollama:
image: ollama/ollama:${OLLAMA_VERSION} # pinned, never "latest"
container_name: ollama
restart: unless-stopped
volumes:
- ollama-models:/root/.ollama
environment:
OLLAMA_KEEP_ALIVE: 24h # keep the model loaded, so no slow first answer
OLLAMA_CONTEXT_LENGTH: 8192 # default is 4096; more context = more memory
OLLAMA_NUM_PARALLEL: 2 # requests served at once per model
OLLAMA_MAX_LOADED_MODELS: 1 # don't let a second model push the first out of VRAM
OLLAMA_MAX_QUEUE: 64 # beyond this, callers get 503 instead of waiting forever
OLLAMA_FLASH_ATTENTION: 1
OLLAMA_KV_CACHE_TYPE: q8_0 # roughly halves context memory, small quality cost
OLLAMA_NO_CLOUD: 1 # local models only, nothing sent to Ollama's cloud
# No "ports:" section. Ollama is only reachable on the internal network.
networks: [llm]
healthcheck:
test: ["CMD", "ollama", "list"]
interval: 30s
timeout: 10s
retries: 3
start_period: 30s
deploy:
resources:
limits:
memory: 32g # stop a runaway model taking the whole host
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
model-init:
image: ollama/ollama:${OLLAMA_VERSION}
depends_on:
ollama:
condition: service_healthy
environment:
OLLAMA_HOST: http://ollama:11434
entrypoint: ["/bin/sh", "-c", "ollama pull llama3.2 && ollama pull nomic-embed-text"]
networks: [llm]
restart: "no"
caddy:
image: caddy:2
restart: unless-stopped
ports:
- "80:80"
- "443:443"
environment:
OLLAMA_API_TOKEN: ${OLLAMA_API_TOKEN}
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile:ro
- caddy-data:/data
depends_on: [ollama]
networks: [llm]
open-webui: # optional chat UI for people
image: ghcr.io/open-webui/open-webui:${OPEN_WEBUI_VERSION}
restart: unless-stopped
environment:
OLLAMA_BASE_URL: http://ollama:11434
WEBUI_AUTH: "True"
volumes:
- open-webui:/app/backend/data
depends_on: [ollama]
networks: [llm]
networks:
llm:
volumes:
ollama-models:
caddy-data:
open-webui:
# .env: keep this file out of source control
OLLAMA_VERSION=<pinned version from github.com/ollama/ollama/releases>
OPEN_WEBUI_VERSION=<pinned version>
OLLAMA_API_TOKEN=<long random value, e.g. from: openssl rand -hex 32>
What each decision buys you:
- No
portsonollama: the API can’t be reached from outside Docker at all. - A pinned version: an update can’t change behaviour, or break things, without you choosing it.
model-init: the models your apps need are pulled automatically on a fresh server, so nobody needs todocker execin.- A health check:
docker compose psshows whether Ollama is actually answering, and dependent services wait for it. - A memory limit: a model too big for the box fails on its own, instead of taking the server down with it.
No NVIDIA GPU? Remove the reservations block. For AMD GPUs, use the ollama/ollama:<version>-rocm image and pass /dev/kfd and /dev/dri as devices instead.
3. Caddy: TLS, a token, and blocked admin endpoints
Caddy gets and renews TLS certificates automatically. This config checks for the token and refuses the endpoints that change models:
# Caddyfile
llm.example.com {
route {
# Model management stays on the server itself, never through the proxy
@admin path /api/pull /api/push /api/delete /api/create /api/copy /api/blobs/*
respond @admin "Forbidden" 403
# Every other request needs the bearer token
@unauthorised not header Authorization "Bearer {$OLLAMA_API_TOKEN}"
respond @unauthorised "Unauthorised" 401
reverse_proxy ollama:11434
}
}
chat.example.com {
reverse_proxy open-webui:8080
}
For an internal-only server, replace the public host name with an internal one and Caddy’s tls internal option, or put your organisation’s own certificate in place.
Start the stack and test it:
docker compose up -d
docker compose ps # ollama should show "healthy"
curl https://llm.example.com/api/version # 401
curl -H "Authorization: Bearer $OLLAMA_API_TOKEN" https://llm.example.com/api/version # works
curl -X DELETE -H "Authorization: Bearer $OLLAMA_API_TOKEN" https://llm.example.com/api/delete # 403
A single shared token is a good start. When several apps or teams use the server, give each its own token, so you can revoke one without affecting the rest: one @unauthorised matcher per token, or an API gateway in front.
4. GPU setup (NVIDIA)
Install the NVIDIA driver on the host and the NVIDIA Container Toolkit, then let Docker use it:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
docker run --rm --gpus all ubuntu nvidia-smi # should list your GPU
After starting the stack, check the model is really on the GPU:
docker exec ollama ollama ps
# NAME SIZE PROCESSOR UNTIL
# llama3.2:latest 3.6 GB 100% GPU 24 hours from now
If PROCESSOR shows a split such as 40%/60% CPU/GPU, the model doesn’t fit in video memory. Part of it is running on the CPU, which is much slower. Use a smaller model, a smaller quantisation, or less context.
5. Size it properly
Memory is the main constraint. A rough way to think about it:
memory needed ≈ model size + (context length × parallel requests × per-token cost) + overhead
- Model size: roughly the download size. Default tags are usually 4-bit quantised, which is a good balance. 8-bit versions give slightly better quality for about double the memory.
- Context length: how much text a single request can hold. Doubling
OLLAMA_CONTEXT_LENGTHroughly doubles that part of the memory. OLLAMA_NUM_PARALLELmultiplies the context memory, because each parallel slot gets its own.OLLAMA_KV_CACHE_TYPE=q8_0roughly halves the context memory. It needs flash attention, which the Compose file turns on.
Aim to fit the whole model, plus its context, in GPU memory, with some headroom. Then load-test with your real prompts and watch ollama ps and nvidia-smi.
6. Pin your prompts and settings with a Modelfile
If several apps share one assistant, define it once as a custom model, with its system prompt and settings under version control:
# Modelfile
FROM llama3.2
PARAMETER temperature 0.2
PARAMETER num_ctx 8192
SYSTEM "You are a concise assistant for an internal software development team. If you don't know, say so."
docker cp Modelfile ollama:/tmp/Modelfile
docker exec ollama ollama create team-assistant -f /tmp/Modelfile
Apps then ask for team-assistant, and you change its behaviour in one place. Add the create step to model-init, so a rebuilt server gets it automatically.
7. Use it from C#
The same IChatClient code from part 1, now with the token, a longer timeout (long answers can take more than HttpClient’s default 100 seconds), plus logging and OpenTelemetry tracing from Microsoft.Extensions.AI:
using Microsoft.Extensions.AI;
using OllamaSharp;
var http = new HttpClient
{
BaseAddress = new Uri(builder.Configuration["Ollama:BaseUrl"]!), // https://llm.example.com/
Timeout = TimeSpan.FromMinutes(5)
};
http.DefaultRequestHeaders.Authorization =
new("Bearer", builder.Configuration["Ollama:ApiToken"]); // from user-secrets or Key Vault
builder.Services.AddChatClient(new OllamaApiClient(http, "team-assistant"))
.UseLogging()
.UseOpenTelemetry();
Keep the token in user-secrets locally and in Key Vault (or your pipeline’s secret store) everywhere else, never in appsettings.json.
8. Keep it running
Updates. Read the release notes, change OLLAMA_VERSION in .env, then:
docker compose pull
docker compose up -d
docker compose ps # healthy again?
Test on a non-production server first if apps depend on it. To roll back, put the old tag back and run the same commands. Update regularly: Ollama has had security fixes, and an old version is a risk if anything can reach it.
Backups. Models can be downloaded again, so you don’t need to back them up. Back up what you can’t recreate:
compose.yaml,Caddyfileand Modelfiles: keep them in git.envsecrets: keep them in your secret store- the
open-webuivolume, if people use it: it holds their accounts and chat history
Monitoring.
docker compose psand the health check: alert ifollamais unhealthy or restartingdocker logs ollama: load errors and out-of-memory messagesGET /api/ps(from inside the network): which models are loaded and wherenvidia-smi, or NVIDIA’s DCGM exporter for Prometheus: GPU memory, use and temperature- your app’s own traces (OpenTelemetry, as above): response times and errors as users see them
Rate limiting. One heavy user can saturate a GPU. OLLAMA_MAX_QUEUE stops an unlimited backlog, but per-client rate limits in the proxy, or ASP.NET Core rate limiting in an app layer in front, keep things fair.
9. Privacy and data
- Prompts and answers stay on your server, which is often the reason for self-hosting. Make sure your logs don’t quietly copy them somewhere else.
OLLAMA_NO_CLOUD=1stops cloud models and features being used by accident, so nothing is sent off the server.- The server only needs internet access to download models. In a locked-down network, allow only Ollama’s model registry, or pull models on another machine and copy the volume across.
- Check the licence of each model before using it in a product. They differ.
Checklist
- Ollama has no published port; only the proxy does, and only if it’s needed.
- Access goes through TLS and a token (or a VPN), with admin endpoints blocked.
- Image versions are pinned, and updates are deliberate and tested.
- Models are pre-pulled by
model-init, and custom models come from Modelfiles in git. - A health check, restart policy and memory limit are set.
- The GPU is confirmed with
ollama psshowing 100% GPU. - Context length, parallel requests and loaded models are sized for your memory, then load-tested.
- Secrets are in a secret store; config is in git; Open WebUI data is backed up.
- Health, logs, GPU and response times are monitored.
- Model licences are checked for your use.
Sources
- Ollama Docker documentation — images, GPU flags, volumes
- Ollama FAQ — environment variables and their defaults, proxies, context length
- Detecting exposed LLM servers: a Shodan case study on Ollama — Cisco
- 175,000 Ollama servers are exposed to the internet — Twingate
- Understanding and securing exposed Ollama instances — UpGuard
- Packet filtering and firewalls — Docker docs (published ports and UFW)
- Installing the NVIDIA Container Toolkit — NVIDIA
- Caddyfile request matchers — Caddy docs
- Open WebUI documentation
- Microsoft.Extensions.AI — Microsoft Learn