Library Re-processing Runbook#

Operator-run campaign to re-process the (~3,000-doc) Paperless library end to end: clean up the tags/correspondents, re-OCR the hand-written docs with a real vision model (replacing tesseract’s garbage text layer), regenerate their titles, backfill summaries/metadata, then a full RAG re-index.

Run the phases in order — later phases depend on earlier ones (the collapse enriches the alias map that backfill matches against; titles compose from the re-OCR’d content and the clean correspondents; the final --full index re-embeds everything at once).

Vision model — richardyoung/olmocr2:7b-q8. An OCR-tuned Qwen2.5-VL-7B that transcribes cursive far better than the on-ingest minicpm-v. At ~9.5 GB (8.1 GB weights + a 1.4 GB vision projector) it lives on the M4 only — it would CPU-spill on mokou’s 8 GB GTX 1080 (the same failure that made qwen2.5vl:7b return empty output). So campaign traffic must point directly at the M4, never the failover proxy. paperless-reocr pre-flights the model against OLLAMA_HOST and aborts in one line if it’s missing, and now checkpoints a per-doc error (not empty content) on any mid-run Ollama failure — a misrouted run fails loudly instead of half-applying.

Pull the Ollama-registry build (ollama pull richardyoung/olmocr2:7b-q8), not the hf.co/richardyoung/olmOCR-2-7B-1025-GGUF one — the latter ships a separate mmproj file and 400s at registration (ollama#15447). The re-processing target has curate.visionModel = "richardyoung/olmocr2:7b-q8" and curate.visionNumCtx = 16384 baked in, so on ereshkigal sudo paperless-reocr already uses it — you only need to override OLLAMA_HOST to the M4-direct URL.

Preconditions#

  • caffeinate -dimsu running in a spare terminal on the M4 for any long run (Jamf blocks pmset; caffeinate needs no privileges). Sleep/roam just pauses at a checkpoint; --retry-errors mops up.
  • Point OLLAMA_HOST directly at the M4, not the proxy, so a failover can’t swap the model under you. Verify the vision model is actually pulled (curl $OLLAMA_HOST/api/tags; the self-healing ollama-models daemon pulls it, but confirm before a long run).
export OLLAMA_HOST=http://work-laptop-m4.armadillo-banfish.ts.net:11434
export STATE_DIR=/var/lib/paperless-curate/reocr-v2-2026-07   # campaign-scoped
sudo mkdir -p "$STATE_DIR"
T=$(sudo cat /run/secrets/paperless/api-token); P=http://127.0.0.1:8011
curl -s "$OLLAMA_HOST/api/tags" | jq -r '.models[].name'      # must list your vision model

Phase 0 — Safety#

sudo systemctl start paperless-backup-stage.service borgmatic.service  # fresh backup
sudo systemctl stop paperless-curate-catchup.timer                     # no interleaved writes
curl -s -H "Authorization: Token $T" "$P/api/documents/" | jq .count   # baseline doc count

Phase 1 — Tag hygiene (exclude born-digital chat logs)#

The hand-written tag is polluted with born-digital chat logs (MSN/AIM/Trillian exports, document_type id 15) — clean text that must not be re-OCR’d (a VLM pass would overwrite good content). Fix the data once so every consumer benefits:

# 1. Confirm the chat-log document type id (observed: 15).
curl -s -H "Authorization: Token $T" "$P/api/document_types/" | jq -r '.results[]|"\(.id)\t\(.name)"'
# 2. Audit the candidates BEFORE untagging (save the list).
curl -s -H "Authorization: Token $T" \
  "$P/api/documents/?tags__id__all=68&document_type__id=15&fields=id,title&page_size=250" \
  | jq -r '.results[]|"\(.id)\t\(.title)"' | sudo tee "$STATE_DIR/untag-chatlogs-audit.tsv"
# 3. Bulk-remove the hand-written tag (id 68) from those docs.
ids=$(cut -f1 "$STATE_DIR/untag-chatlogs-audit.tsv" | jq -Rn '[inputs|tonumber]')
curl -s -X POST -H "Authorization: Token $T" -H "Content-Type: application/json" \
  "$P/api/documents/bulk_edit/" \
  -d "{\"documents\":$ids,\"method\":\"modify_tags\",\"parameters\":{\"add_tags\":[],\"remove_tags\":[68]}}"
# 4. Re-count; expect the ~1,129 to drop substantially.
curl -s -H "Authorization: Token $T" "$P/api/documents/?tags__id__all=68" | jq .count

paperless-reocr --exclude-doctype 15 is a defensive backstop if any chat logs remain tagged; it also skips born-digital text originals automatically.

Phase 2 — Correspondent collapse (metadata-only; content-independent)#

Run before re-OCR: it never reads document content, and the alias map it produces feeds the Phase 5 backfill’s deterministic matching.

sudo -E paperless-correspondents-propose --llm     # writes correspondent-remap.proposed.json
                                                   # + correspondents-alias-patch.json
# REVIEW/EDIT the proposal: resolve every `review` entry into merges/to_unknown/keep,
# confirm merge directions (e.g. Jon vs John Craton), prune anything wrong. Then:
sudo mv "$STATE_DIR/correspondent-remap.proposed.json" "$STATE_DIR/correspondent-remap.approved.json"
sudo -E paperless-correspondents-apply             # DRY-RUN: prints every migrate/delete + counts
sudo -E paperless-correspondents-apply --apply     # migrate docs, delete emptied variants

Verify: total doc count unchanged; no correspondent names contain (; the prompt-leak/placeholder entries are gone; spot-check one migrated doc per big group in the UI.

Then fold the extracted handles/emails into the sops alias map so future ingests still auto-match, and regenerate the native match rules:

sudo cat "$STATE_DIR/correspondents-alias-patch.json"   # merge each add_aliases list into
sops secrets.yaml                                        #   the paperless.correspondents key
# deploy (nixos-rebuild switch) so sops-nix re-renders /run/secrets/paperless/correspondents
sudo paperless-correspondents-sync --apply               # regenerate match/matching_algorithm

Phase 3 — Validate the model on a hard sample#

The model is decided (richardyoung/olmocr2:7b-q8, baked into the target), but validate it on a fixed sample before committing 1,000+ docs — the archive’s cursive is messier than any benchmark, and olmocr2 is English-cursive-strong but weakest on the French/Japanese minority. Pick 12–15 real doc ids covering the hard cases: dense cursive from 2–3 hands, one French and one Japanese page if present, a mixed print+hand doc, a known minicpm-v narration victim (e.g. doc 75), a faint/pencil page.

export IDS="75,102,…"          # your fixed sample
sudo -E paperless-reocr --dry-run --ids "$IDS"   # OLLAMA_HOST is the M4 (see Preconditions)

# Skim the proposals — each record carries model / lint_pages / empty_pages /
# illegible_ratio; the report flags high-[illegible], scrubbed, and empty pages:
jq -c '{id, chars_after, lint_pages, empty_pages, illegible_ratio,
        text:.new_content[0:400]}' "$STATE_DIR/reocr-proposals.jsonl"

Judge against the actual scans in the UI: verbatim fidelity on the messiest hand, no fluent hallucination on faint pages (must degrade to [illegible], not prose), zero narration after lint. Watch the French/Japanese pages specifically — if one comes back garbled, that’s the model, not the pipeline; note those ids and re-run just them later against a multilingual VLM (e.g. qwen3-vl:8b, pulled on-demand) via --ids, rather than switching the whole campaign.

Phase 4 — Re-OCR (vision, hand-written subset, redo-all)#

REOCR_MARKER_TAG=reocr-v2 re-OCRs the whole tag, including the 361 docs already done with minicpm-v (their large tesseract text trips the ⚠️ prior text flag — expected). After the campaign, reocr minus reocr-v2 = stragglers.

# On ereshkigal the target bakes curate.visionModel=olmocr2 + visionNumCtx=16384,
# so VISION_MODEL/VISION_NUM_CTX need no export — only OLLAMA_HOST (Preconditions).
export REOCR_MARKER_TAG=reocr-v2

sudo -E paperless-reocr --dry-run --limit 25   # review new_content + lint/illegible/empty flags
sudo -E paperless-reocr --apply --limit 20     # pilot; eyeball in the UI
sudo -E paperless-reocr --apply --limit 150    # chunked evening runs; checkpoints resume
sudo -E paperless-reocr --apply --retry-errors # mop up

Budget: ~15–30 s/page on the M4 at 16k ctx; ~1,100 docs × 2–3 pages ≈ 10–30 h, so plan 4–8 chunked sessions with the laptop docked and caffeinate running.

Phase 5 — Titles + summaries/metadata backfill#

Now that content is readable, regenerate titles (the core fix — tesseract garbage titles made “love letters” indistinguishable) and refresh summaries/tags. The title is composed deterministically from a validated date + the matched correspondent + an LLM subject phrase (never the model’s own title text):

export OLLAMA_MODEL=qwen2.5:7b-instruct-q4_K_M   # or the M4's 14b for a quality bump
unset VISION_MODEL VISION_NUM_CTX

sudo -E paperless-backfill --dry-run --limit 25 --tag hand-written --retitle --redate
#   review report.md's "Proposed titles (OLD -> NEW)" section, then:
sudo -E paperless-backfill --apply --tag hand-written --retitle --redate --overwrite-summaries
sudo -E paperless-backfill --apply --tag hand-written --retitle --retry-errors

--redate also fixes created to the date written in the letter (handwritten docs carry scan-ish dates because Paperless parsed tesseract garbage) — only when a valid in-document date was found; otherwise the title falls back to the existing created and is flagged in the report.

Optional cleanup of the unmatched pile, now that the alias map is richer:

sudo -E paperless-backfill --apply --correspondent Unknown --overwrite-correspondent

Phase 6 — RAG re-index (mandatory)#

Correspondent renames don’t bump doc modified and chunk vectors embed the old correspondent name, so the hourly incremental indexer can’t heal the collapse — a --full re-embed is required after the campaign:

sudo paperless-index --full    # ereshkigal-local CPU embeddings; well under an hour

The hourly paperless-index timer can stay enabled throughout (harmless double-embeds of docs edited mid-campaign).

Phase 7 — Restore normal operation#

sudo systemctl start paperless-curate-catchup.timer
unset OLLAMA_HOST OLLAMA_MODEL STATE_DIR REOCR_MARKER_TAG VISION_MODEL VISION_NUM_CTX IDS
# Once (reocr − reocr-v2) is empty, retire the old marker tag in the UI.
journalctl -u paperless-curate-catchup -u paperless-index --since yesterday   # next morning

Notes#

  • Run directly against the M4. A proxy-routed campaign fails the pre-flight (or checkpoints per-doc errors after a failover) — it cannot silently half-apply.
  • minicpm-v stays the pipeline VLM on both proxy upstreams for paperless-gpt’s on-ingest OCR; only paperless-reocr uses the campaign model, and only M4-direct.
  • A present model can still return empty (context overflow). The Phase 3 dry-run is the end-to-end check: non-empty verbatim text with prompt_eval_count well under VISION_NUM_CTX.