Ollama + Open-WebUI + NVIDIA GPU

A Docker Compose stack for running local LLMs on your own hardware with NVIDIA GPU acceleration, a web UI, and one-command model setup via profiles.

What's Included

Service	Purpose	Port
Ollama	Local LLM inference engine with NVIDIA GPU support	11434
Open-WebUI	ChatGPT-style web interface	8080
Watchtower	Automatic container updates	-
Model Profiles	One-command download & registration of curated models	-

Quick Start

# Clone and run
git clone <this-repo>
cd ollama
docker compose up -d

# Pull a model (manual)
docker exec -it ollama ollama pull deepseek-r1:14b

# Or use profiles (automated - downloads GGUF, creates Modelfile, registers)
docker compose --profile qwen3-coder-unsloth up

# Open the UI
open http://localhost:8080

---

Model Profiles

This stack includes pre-configured model profiles that automate the entire setup process:

1. Download GGUF from HuggingFace
2. Create optimized Modelfile with proper templates
3. Register with Ollama

Available Models

Profile	Model	Size	VRAM	Category	Notes
deepseek-r1-14b	DeepSeek R1 14B	14B	~9GB	Baseline	Excellent reasoning
qwen25-14b	Qwen 2.5 14B	14B	~9GB	Baseline	Strong coding, JSON
gpt-oss-heretic	OpenAI GPT-oss 20B	20B MOE	~12GB	Bold	Abliterated OpenAI model
glm-47-heretic	GLM 4.7 Flash	30B MOE	~16GB	Bold	Uncensored, fast
qwen3-coder	Qwen3 Coder 30B-A3B	30B MOE	~8GB	Coder	From Ollama library
qwen3-coder-abliterated	Qwen3 Coder Abliterated	30B MOE	~17GB	Coder	Uncensored
qwen25-coder-32b	Qwen2.5 Coder 32B	32B	~15GB	Coder	Max capability
qwen3-coder-unsloth	Qwen3 Coder (Unsloth)	30B MOE	~15GB	Tool	SOTA with tool calling
watt-tool-8b	Watt-Tool 8B	8B	~6GB	Tool	Purpose-built for tools
groq-tool-8b	Llama-3-Groq Tool-Use	8B	~6GB	Tool	Groq fine-tune
darkest-universe	MN-Darkest-Universe 29B	29B	~16GB	Grey Zone	Security research

Profile Commands

# Individual models
docker compose --profile qwen3-coder-unsloth up   # SOTA tool-calling coder
docker compose --profile watt-tool-8b up          # Fast tool-calling
docker compose --profile deepseek-r1-14b up       # Baseline reasoning

# Group profiles
docker compose --profile tool up                  # All tool-calling models
docker compose --profile coder up                 # All coder models
docker compose --profile baseline up              # All baseline models
docker compose --profile bold up                  # All modified/abliterated
docker compose --profile grey-zone up             # Security research models
docker compose --profile all up                   # Everything (~100GB+)

# List all available profiles
docker compose config --profiles

---

Agentic Coding Integration (OpenCode, Goose, etc.)

The tool-calling models support function/tool calling for use with agentic coding assistants:

• OpenCode - Terminal-based AI coding assistant
• Goose - AI developer agent
• Claude Code (via intercept proxy)

OpenCode Configuration

Create ~/.config/opencode/opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "theme": "catppuccin",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama (local)",
      "options": {
        "baseURL": "http://localhost:11434/v1"
      },
      "models": {
        "qwen3-coder-unsloth": {
          "name": "Qwen3 Coder 30B-A3B Unsloth (SOTA Tool Calling)",
          "limit": {
            "context": 32768,
            "output": 65536
          }
        },
        "watt-tool-8b": {
          "name": "Watt Tool 8B - Tool Calling (local)",
          "limit": {
            "context": 8192,
            "output": 4096
          }
        },
        "groq-tool-8b": {
          "name": "Llama 3 Groq 8B Tool Use (local)",
          "limit": {
            "context": 8192,
            "output": 4096
          }
        }
      }
    }
  }
}

Usage with OpenCode

# First, start the model
docker compose --profile qwen3-coder-unsloth up

# Then run OpenCode with local model
opencode --model ollama/qwen3-coder-unsloth

Tool-Calling Model Comparison

Model	Speed	Quality	Best For
qwen3-coder-unsloth	Medium	SOTA	Complex agentic tasks, large codebases
watt-tool-8b	Fast	Good	Quick iterations, simple tool calls
groq-tool-8b	Fast	Good	Groq-optimized workflows

---

Prerequisites

NVIDIA Container Toolkit (Ubuntu/Pop!_OS 24.04)

GPU passthrough requires the NVIDIA Container Toolkit. If docker compose up fails with "could not select device driver nvidia", install it:

# Add NVIDIA's apt repository
sudo wget -qO /etc/apt/keyrings/nvidia-container-toolkit.asc \
  https://nvidia.github.io/libnvidia-container/gpgkey

echo "deb [signed-by=/etc/apt/keyrings/nvidia-container-toolkit.asc] \
  https://nvidia.github.io/libnvidia-container/stable/deb/amd64 /" \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Install and configure
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPU access:

docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi

---

Understanding Model Names

Ollama uses a structured naming convention:

deepseek-r1:14b-instruct-q5_K_M
     │       │      │       │
     │       │      │       └─ Quantization (memory/quality tradeoff)
     │       │      └───────── Variant (instruct, chat, code, vision)
     │       └─────────────── Parameters (billions)
     └─────────────────────── Model Family

Quantization Reference

Level	Quality	Use Case
q8_0	~99%	Maximum quality, if VRAM allows
q6_K	~97%	Excellent quality
q5_K_M	~95%	Great balance
q4_K_M	~92%	Most popular choice
q3_K_M	~85%	Noticeable degradation
IQ4_XS	~90%	Imatrix-optimized, smaller

VRAM Requirements

Model Size	q4_K_M	q5_K_M	q6_K	q8_0
7-8B	~5GB	~6GB	~7GB	~9GB
13-14B	~8GB	~10GB	~12GB	~15GB
30B MOE	~15GB	~18GB	~22GB	~30GB
30-34B Dense	~18GB	~22GB	~26GB	~34GB

---

Architecture

┌─────────────────────────────────────────────────────────┐
│                    Docker Network                       │
│                    (ollama-net)                         │
│                                                         │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐  │
│  │  Open-WebUI │───▶│   Ollama    │    │ Watchtower  │  │
│  │   :8080     │    │   :11434    │    │  (updates)  │  │
│  └─────────────┘    └──────┬──────┘    └─────────────┘  │
│                            │                            │
│         ┌──────────────────┼──────────────────┐         │
│         │    Model Setup Services (profiles)  │         │
│         │  setup-* → download GGUF            │         │
│         │  register-* → create & register     │         │
│         └──────────────────┴──────────────────┘         │
│                            │                            │
└────────────────────────────┼────────────────────────────┘
                             │
                    ┌────────▼────────┐
                    │   NVIDIA GPU    │
                    │ (CUDA Runtime)  │
                    └─────────────────┘

Volume Layout

Volume	Container Path	Purpose
ollama	/root/.ollama	Model weights, config
open-webui	/app/backend/data	User data, chat history
models	/models	Shared GGUF storage (persists downloads)
data	/data	Application data

---

Performance Tuning

The compose file includes optimizations for local use:

Variable	Value	Effect
OLLAMA_FLASH_ATTENTION	1	Faster inference on modern GPUs
OLLAMA_KEEP_ALIVE	24h	Keep models loaded (no reload delay)
OLLAMA_MAX_LOADED_MODELS	2	Hot-swap between 2 models
OLLAMA_NUM_PARALLEL	4	Concurrent request handling
OLLAMA_GPU_OVERHEAD	0	Use all available VRAM

---

Common Commands

# Lifecycle
docker compose up -d              # Start (detached)
docker compose down               # Stop
docker compose logs -f ollama     # Follow logs

# Model management
docker exec -it ollama ollama list          # List downloaded models
docker exec -it ollama ollama pull <model>  # Download a model
docker exec -it ollama ollama rm <model>    # Delete a model
docker exec -it ollama ollama ps            # Show running models

# Profile management
docker compose --profile <name> up          # Setup specific model
docker compose config --profiles            # List all profiles

# Monitoring
watch -n1 nvidia-smi              # GPU utilization
docker stats                      # Container resources

# Volume management
docker exec -it ollama ls -la /models       # Check downloaded GGUFs
docker exec -it ollama du -sh /models/*     # Check GGUF sizes

---

Tested Configurations

Platform	GPU	Status
Pop!_OS 24.04	Quadro RTX 5000 (16GB)	✅ Working
Arch Linux	Various NVIDIA	✅ Working
Windows 11 + WSL2	RTX Series	✅ Working
Ubuntu 24.04	RTX 3080/4080	✅ Working

Linux (Arch/Elementary OS)

Windows 11 (WSL2)

---

Troubleshooting

"could not select device driver nvidia"

NVIDIA Container Toolkit not installed. See Prerequisites.

Model won't load / OOM errors

Model too large for available VRAM. Try a smaller model or lower quantization:

# Use a smaller 8B tool model instead of 30B
docker compose --profile watt-tool-8b up

Profile download stuck or failed

# Check download progress
docker compose logs -f setup-<model-name>

# Force re-download (removes cached GGUF)
docker exec -it ollama rm -f /models/<gguf-file>.gguf
docker compose --profile <model> up --force-recreate

Tool calling not working

Not all models support tool/function calling. Use models from the tool profile:

docker compose --profile tool up

Slow inference

1. Verify GPU is being used: docker exec -it ollama ollama ps
2. Check VRAM: nvidia-smi
3. Reduce OLLAMA_NUM_PARALLEL if context is large

Open-WebUI not connecting to Ollama

Ensure both containers are on the same network:

docker network inspect ollama_ollama-net

---

Resources

• Ollama Model Library
• Open-WebUI Documentation
• OpenCode AI
• Unsloth GGUF Collection
• NVIDIA Container Toolkit

---

License

MIT - Do whatever you want with it.

---

Maintained by @usrbinkat