Ollama#

Ollama runs on three hosts. work-laptop-m4 (M4 Pro, Metal) is the primary inference engine for the document pipeline; mokou (GTX 1080, Pascal) is the always-on fallback; ereshkigal runs a loopback ollama loading only nomic-embed-text (RAG embeddings) — see the AI overview for the full topology.

Ollama on work-laptop-m4 (darwin, primary)#

The M4 runs via tsunaminoai.ollama (modules/darwin/ollama — a launchd system daemon; neither nix-darwin nor a user agent fit, see the module header). Metal needs no acceleration overrides. It binds its Tailscale IP only (100.74.80.8:11434) — never 0.0.0.0 on a work machine that roams.

Model budget on 24 GB unified memory (GPU-usable ≈ 16 GB by default):

A 32b-q4 (~20 GB) does not fit alongside anything else on 24 GB — skip it on this machine. Laptop asleep/away = requests fail over to mokou at 7b quality; that is by design.

Ollama on mokou (fallback)#

Hardware#

The CUDA SM 6.1 problem#

nixpkgs builds Ollama with CUDA support targeting SM 7.5+ (Turing and newer) by default. Pascal (GTX 10-series, SM 6.1) is excluded, so a stock services.ollama.acceleration = "cuda" silently falls back to CPU inference — you get correct output but at 1/10th the speed.

The fix is to override cudaArches at the package level:

services.ollama = {
  enable = true;
  acceleration = "cuda";
  package = pkgs.ollama.override {
    acceleration = "cuda";
    cudaArches = ["61"];   # SM 6.1 = GTX 1080 / Pascal
  };
  ...
};

This rebuilds Ollama with a PTX/SASS target for SM 6.1. Build time is significant (~15–30 min on first switch); after that inference is GPU-accelerated and sub-second for 3b models.

NixOS configuration#

# hosts/x86_64-nixos/mokou/default.nix
services.ollama = {
  enable = true;
  acceleration = "cuda";
  package = pkgs.ollama.override {
    acceleration = "cuda";
    cudaArches = ["61"];
  };
  host = "0.0.0.0";   # listen on all interfaces, not just localhost
  port = 11434;
  loadModels = [
    "minicpm-v"          # Pipeline VLM: vision OCR (paperless-gpt + paperless-reocr)
    "qwen2.5:3b"         # Text model for tagging / titling / agents
    "qwen2.5-coder:7b"   # Code model for offline flake maintenance (llm CLI)
    "qwen2.5-coder:3b"   # Smaller coder fallback under VRAM pressure
    "nomic-embed-text"   # Embeddings for RAG / semantic search
  ];
  home = "/data/ollama";
  environmentVariables = {
    OLLAMA_MAX_QUEUE = "4";
  };
};

Models live on an external LUKS-encrypted ext4 disk (cryptdata) mounted at /data, so mokou can’t use the module’s default DynamicUser. Instead it defines a static users.users.ollama system user (with home = "/data/ollama") and forces the service off DynamicUser:

# hosts/x86_64-nixos/mokou/default.nix
systemd.services.ollama.serviceConfig = {
  DynamicUser = lib.mkForce false;
  PrivateUsers = lib.mkForce false;
  User = lib.mkForce "ollama";
  Group = lib.mkForce "ollama";
};

(ereshkigal’s local ollama keeps the default home = "/var/lib/ollama".)

host = "0.0.0.0" is required so other hosts on Tailscale can reach the endpoint. The firewall opens port 11434 (networking.firewall.allowedTCPPorts = [11434]), which exposes it on the LAN and Tailscale — keep it off the internet.

Models#

loadModels pre-pulls these on activation so the first request doesn’t stall waiting for a download — and so the coder models are present offline, which is the whole point of the llm CLI outage workflow. The two qwen2.5-coder models fit the 8 GB card (Ollama unloads idle models, so the vision and coder models swap rather than co-resident).

Compute capability reference#

If you have a Turing or newer GPU, the nixpkgs default works without the override.