Ollama on mokou#

Ollama runs on mokou (GTX 1080 Ti, Pascal architecture) and serves the entire network as a shared inference endpoint over Tailscale.

Hardware#

The CUDA SM 6.1 problem#

nixpkgs builds Ollama with CUDA support targeting SM 7.5+ (Turing and newer) by default. Pascal (GTX 10-series, SM 6.1) is excluded, so a stock services.ollama.acceleration = "cuda" silently falls back to CPU inference — you get correct output but at 1/10th the speed.

The fix is to override cudaArches at the package level:

services.ollama = {
  enable = true;
  acceleration = "cuda";
  package = pkgs.ollama.override {
    acceleration = "cuda";
    cudaArches = ["61"];   # SM 6.1 = GTX 1080 / Pascal
  };
  ...
};

This rebuilds Ollama with a PTX/SASS target for SM 6.1. Build time is significant (30–60 min); after that inference is GPU-accelerated and sub-second for 3b models.

NixOS configuration#

# hosts/x86_64-nixos/mokou/default.nix
services.ollama = {
  enable = true;
  acceleration = "cuda";
  package = pkgs.ollama.override {
    acceleration = "cuda";
    cudaArches = ["61"];
  };
  host = "0.0.0.0";   # listen on all interfaces, not just localhost
  port = 11434;
  loadModels = [
    "qwen2.5-vl:7b"      # VLM for vision OCR (paperless-gpt)
    "qwen2.5:3b"         # Text model for tagging / titling / agents
    "nomic-embed-text"   # Embeddings for RAG / semantic search
  ];
  home = "/var/lib/ollama";
  environmentVariables = {
    OLLAMA_MAX_QUEUE = "4";
  };
};

host = "0.0.0.0" is required so other hosts on Tailscale can reach the endpoint. The firewall allows port 11434 inbound on the Tailscale interface only.

Models#

loadModels pre-pulls these on activation so the first request doesn’t stall waiting for a download.

Compute capability reference#

If you have a Turing or newer GPU, the nixpkgs default works without the override.