AI / Local Inference#

This flake runs a fully local AI stack — no cloud APIs, no data leaving the network. Inference is served by work-laptop-m4 (M4 Pro, Metal) as the primary engine with mokou (GTX 1080) as the always-on fallback; consumers all point at the failover proxy on ereshkigal :11434 and never need to know which one answered.

Stack overview#

flowchart TD subgraph m4["work-laptop-m4 — M4 Pro 24GB (Metal, PRIMARY)"] ollamaM4["Ollama 100.74.80.8:11434"] ollamaM4 --- m4txt["qwen2.5:7b-instruct\n(pipeline text)"] ollamaM4 --- m4vlm["minicpm-v\n(pipeline vision)"] ollamaM4 --- m4big["qwen2.5:14b / olmocr2:7b-q8\n(batch quality — runbook only)"] end subgraph mokou["mokou — GTX 1080 (SM 6.1, FALLBACK)"] ollamaMk["Ollama :11434"] ollamaMk --- mkvlm["minicpm-v"] ollamaMk --- mktxt["qwen2.5:7b-instruct / :3b"] ollamaMk --- mkcoder["qwen2.5-coder:7b/:3b\n(offline flake maintenance)"] end subgraph ereshkigal["ereshkigal"] proxy["nginx ollama failover proxy :11434\n(3s connect timeout → backup)"] emb["local Ollama 127.0.0.1:11435\nnomic-embed-text (always-on embeddings)"] pgpt["paperless-gpt :8013\n(auto-tag sidecar)"] owui["Open-WebUI :3000\n(LLM chat + RAG)"] allm["AnythingLLM :13001\n(RAG chat sidecar)"] end subgraph dev["dev shell / nix run"] agents["agents package\nsummarize / triage / classify …"] llmcli["llm CLI\n-t nix-maint"] kiwixask["kiwix-ask\n(grounded retrieval)"] end subgraph voile["voile — NAS"] kiwix["Kiwix :8092\n(offline Wikipedia / DevDocs / SE)"] end proxy -->|"primary"| ollamaM4 proxy -->|"backup"| ollamaMk pgpt --> proxy owui --> proxy allm --> proxy agents -->|"OLLAMA_HOST"| proxy llmcli -->|"OLLAMA_HOST"| proxy kiwixask -->|"retrieve articles"| kiwix kiwixask -->|"ground answer"| llmcli

Components#

Component Host Purpose
Ollama work-laptop-m4 Primary inference (Metal; tsunaminoai.ollama launchd daemon)
Ollama mokou Fallback inference, GPU-accelerated (model parity with the M4’s pipeline set)
ollama failover proxy ereshkigal nginx upstream pool on the canonical :11434 (tsunaminoai.ollamaProxy)
embeddings Ollama ereshkigal Loopback :11435, nomic-embed-text only — always-on for RAG
paperless-gpt ereshkigal LLM auto-tagging sidecar for Paperless-NGX
paperless-curate ereshkigal (CLI) Bulk archive curation: summaries, controlled tags, correspondents, hand-written re-OCR
Open-WebUI ereshkigal LLM chat + document RAG (HTTP 3000 / HTTPS 3001)
AnythingLLM ereshkigal RAG chat sidecar over the document library (HTTP 13001 / HTTPS 13002)
agents CLI Personal data archeology tools
llm CLI dev shell Offline flake-maintenance assistant (llm -t nix-maint)
llm-maint dev shell llm -t nix-maint with live repo facts (nixpkgs lock, modules, hosts) injected
kiwix-ask dev shell Retrieval-grounded answers from voile’s Kiwix

Key design decisions#

Everything runs locally. Documents contain personal financial, medical, and legal data — sending them to an external API is not acceptable.

One endpoint, transparent failover. Every consumer points at the ereshkigal proxy. The M4 is a laptop — it sleeps and travels — so absence costs one 3-second connect timeout, after which requests go straight to mokou for 30-second windows. The family-facing path never depends on the laptop.

Model parity is the failover contract. The pipeline only references models loaded on BOTH the M4 and mokou (qwen2.5:7b-instruct-q4_K_M, minicpm-v, nomic-embed-text). The M4’s bigger models (qwen2.5:14b, richardyoung/olmocr2:7b-q8) are used only by the re-processing runbook, which targets the M4 directly and bypasses the proxy on purpose.

Embeddings stay on ereshkigal. nomic-embed-text on the loopback :11435 is tiny, CPU-fast, and always-on — RAG indexing and query embedding never fail over.

Small models with constrained output. For the pipeline models, reliability comes from forcing tight output formats (single word, JSON schema, CSV) rather than open-ended prose. See the individual pages for prompt patterns.

Offline maintenance, not just offline inference. The proxy also backs the llm CLI and kiwix-ask, so the flake itself stays maintainable during a cloud-AI outage: qwen2.5-coder:7b (on mokou) stands in for ChatGPT/Claude, and voile’s offline Kiwix archive stands in for the open web.