paperless-curate#

paperless-curate is a toolset (pkgs/paperless-curate/) that curates the Paperless-NGX archive using the Paperless REST API and local Ollama inference. Where paperless-gpt tags documents as they arrive, these tools do bulk, review-gated work over the existing archive: summaries, controlled-vocabulary tags, correspondent assignment, and re-OCR of hand-written scans.

All inference is local (mokou’s Ollama). The archive holds financial, medical, and legal material, so nothing leaves the network.

Tools#

On ereshkigal the tools are wrapped (curateWrapped in the module) with this host’s environment baked in via --set-default, so a bare sudo paperless-<tool> works without exporting anything (sudo strips the caller’s env). Every variable stays overridable on the command line.

Shared conventions#

• Dry-run → apply. Every mutating tool defaults to a dry-run that writes a proposals file + a human-readable report and makes no changes. Review, then re-run with --apply. The apply phase reads the (possibly hand-edited) proposals back, so you can prune or correct before committing.
• Resumable checkpoints. Progress is appended to /var/lib/paperless-curate/checkpoint.jsonl; a re-run skips already-done docs. --retry-errors re-attempts ones previously recorded as errors.
• Protected tags. Docs tagged reviewed or manual are skipped unless --include-protected is passed.

Backfill (summaries / tags / correspondents)#

paperless-backfill fetches each document’s OCR text and, in one LLM call, produces a short summary and a set of controlled-vocabulary tags; it also assigns a correspondent by deterministic alias matching (not the LLM).

# dry-run over the whole archive → proposals.jsonl + report.md, no writes
sudo paperless-backfill

# apply; non-destructive defaults (won't overwrite an existing Summary/correspondent)
sudo paperless-backfill --apply

# just the newly-arrived docs missing a Summary (the nightly catch-up uses this)
sudo paperless-backfill --apply --missing-only

Useful flags: --limit N, --since YYYY-MM-DD, --tag NAME (only docs carrying a tag), --correspondent NAME (only docs assigned to a correspondent — e.g. --correspondent Unknown to re-attempt just the unmatched pile against a freshly-enriched alias map), --overwrite-summaries, --overwrite-correspondent, --reconcile-tags (also remove tags not in the vocabulary). An optional nightly timer (docPipeline.curate.catchup.enable) runs --missing-only --apply.

Retitling (--retitle / --redate)#

Tesseract garbage on cursive bubbles into the document title, making one “love letter” indistinguishable from the next. After re-OCR, --retitle fixes this. The LLM only supplies a short subject phrase and a candidate date; the title is composed deterministically so a hallucinated name or date can’t land in it:

<validated-date-or-created> letter from <matched-correspondent> — <subject>

The date must match YYYY-MM-DD with a plausible year, else it falls back to the doc’s existing created (flagged in the report). The correspondent comes from deterministic alias matching, not the model. --redate (implies --retitle) additionally sets the doc’s created to the validated in-document date — the real date a letter was written, versus the scan-ish date Tesseract left behind. Both are off by default (the nightly catch-up never retitles); the report’s “Proposed titles (OLD → NEW)” section is the review surface. RETITLE_NOUN (default letter) sets the noun, or RETITLE_NOUN="" drops it for non-letter runs.

Correspondent collapse#

The correspondent list accretes duplicates (case variants, Name (handle) parentheticals, maiden/married names), raw emails/screen-names, and LLM-leaked garbage. The propose/apply pair collapses it into clean person/org names, modeled on the tag-consolidation flow:

sudo paperless-correspondents-propose --llm   # writes correspondent-remap.proposed.json
                                               # + correspondents-alias-patch.json
# review/edit: resolve every `review` entry (emails only you can identify) into
# merges/to_unknown/keep, confirm merge directions, then rename (safety gate):
sudo mv …/correspondent-remap.proposed.json …/correspondent-remap.approved.json
sudo paperless-correspondents-apply            # dry-run: prints migrate/delete + counts
sudo paperless-correspondents-apply --apply    # migrate docs, delete emptied correspondents

Propose runs deterministic passes first (strip parentheticals into handle candidates, detect garbage/placeholders, look names+handles up in the sops alias map, case-fold grouping), and --llm only maps the ambiguous residue (bare names, near-miss spellings) onto the canonical list — never inventing a name. Buckets: merges, to_unknown (garbage/single-token names default here), review (must be resolved before apply), keep. The owner and Unknown are guarded — never renamed or deleted. Apply migrates docs in 100-doc batches via bulk_edit set_correspondent and only deletes a correspondent after re-reading its document_count == 0 (the FK is SET_NULL, so deleting a non-empty one would silently orphan its docs).

The emitted correspondents-alias-patch.json lists the handles/emails/old names to fold (manually) into the sops paperless/correspondents map; after deploying that, run paperless-correspondents-sync --apply so future ingests still auto-match, and paperless-index --full so the RAG index picks up the renames (a bulk set_correspondent does not bump doc modified, so the incremental indexer can’t see it). Full sequence: the re-processing runbook.

Tag consolidation#

CREATE_NEW_TAGS=true on paperless-gpt tends to sprawl tags. To consolidate:

paperless-tags-propose --llm      # writes tag-remap.proposed.json (+ report)
# review, then rename the file to tag-remap.approved.json (safety gate)
paperless-tags-apply              # creates canonical tags, migrates docs, deletes orphans

The proposal is constrained to the TAG_VOCAB_FILE vocabulary (correspondents.nix tagVocabulary); the LLM can only map an existing tag to a vocabulary term or KEEP, never invent one.

Re-OCR hand-written scans#

Paperless runs Tesseract on every ingest (PAPERLESS_OCR_MODE=redo), which is tuned for print and mangles cursive/handwriting. paperless-reocr re-reads the original scan with a vision model and writes the corrected text back into the document’s searchable content. Target docs are selected by tag (default hand-written).

Model: minicpm-v (on-ingest) / olmocr2 (re-OCR campaigns)#

On-arrival OCR uses minicpm-v as its vision model — docPipeline.ollamaModel (paperless-gpt image OCR). minicpm-v (~5.5 GB, ~615 image tokens per page) fits mokou’s 8 GB GTX 1080 on-GPU, so it satisfies the failover proxy’s model-parity contract (a proxied request that fails over to mokou must find the same model).

For a re-OCR campaign (see the re-processing runbook), curate.visionModel on ereshkigal is set to richardyoung/olmocr2:7b-q8 — an OCR-tuned Qwen2.5-VL-7B that transcribes cursive far better. At ~9.5 GB (8.1 GB weights + a 1.4 GB vision projector) it lives on the M4 only — it would CPU-spill on the 1080 and hit the qwen2.5vl empty-OCR failure — so it is not in the proxy-parity set: campaign runs point OLLAMA_HOST directly at the M4 and raise VISION_NUM_CTX to 16384 (curate.visionNumCtx; Qwen2.5-VL vision tokens overflow smaller contexts). paperless-gpt on-ingest keeps minicpm-v via ollamaModel, so proxy parity is preserved. Temperature is pinned to 0 (VISION_TEMPERATURE) so the model transcribes rather than “improves” cursive into fluent-but-wrong prose.

Either way, paperless-reocr scrubs meta-narration and markdown fences from each page (an ambiguous scan that produces “The image appears to be blank…” becomes [unreadable page], not document content), and the report flags scrubbed pages, empty pages, and a high [illegible] ratio.

minicpm-v replaced qwen2.5vl:7b, which was a poor fit on the 1080:

• At ~11 GB it does not fit, so Ollama spilled it almost entirely to CPU (ollama ps showed 98%/2% CPU/GPU) — extremely slow.
• Its per-page vision tokens overflow the context window, so Ollama truncated the image and the model returned empty OCR text.

ollama_chat_vision also sets num_ctx (VISION_NUM_CTX, default 8192) and keep_alive (VISION_KEEP_ALIVE, default 10m) so the model stays resident between pages instead of reloading each call.

Workflow#

# 1. dry-run — build proposals for review (nothing is written to Paperless)
sudo paperless-reocr --tag hand-written

# 2. review /var/lib/paperless-curate/reocr-report.md
#    it shows "tesseract N → vision N chars" per doc and flags ⚠️ where tesseract
#    already had substantial text (likely a printed/mixed doc the VLM may
#    hallucinate). Delete those lines from reocr-proposals.jsonl, and untag any
#    genuinely mis-tagged printed docs from `hand-written` in Paperless.

# 3. apply ONLY the reviewed docs (won't re-OCR anything you pruned)
sudo paperless-reocr --tag hand-written --apply --only-proposed

# 4. refresh metadata from the corrected text
sudo paperless-backfill --apply --overwrite-summaries --overwrite-correspondent --tag hand-written

Useful flags: --limit N, --since DATE, --ids CSV (re-OCR exactly these doc ids — used by the bake-off), --exclude-doctype X (skip a document type, e.g. the chat-log type, by name or id; repeatable), --dpi N (default 200; try 300 for hard pages), --model NAME (override the vision model), --max-pages N (default 30; skips book-length scans), --mark/--no-mark (the reocr marker tag, default on; set REOCR_MARKER_TAG=reocr-v2 to re-run a whole tag including already-done docs).

Limitations#

• Only the searchable content is corrected. The archive PDF’s embedded text layer stays as Tesseract produced it — Paperless does not regenerate it from an edited content. This is fine for search, summaries, and RAG, which is the goal.
• The hand-written tag can include printed or mixed documents where Tesseract was already accurate and the VLM may hallucinate. Review the ⚠️ flags and prune.

Configuration#

The wrapper bakes in host defaults; the ones most relevant here:

See Paperless-NGX / Document Pipeline for the container stack and paperless-gpt for on-arrival tagging.