Database as queue
Registration, not processing, enqueues work. Each stage asks the database what is incomplete, which makes the pipeline resumable and idempotent without a separate message bus.
Technical appendix
The Archive architecture
The public page shows what the system can do. This appendix explains the engineering shape: how scanned magazines become searchable, corrected, cited source records.
Core principle
Extraction != Truth. Extraction -> Review -> Truth.
A model reading a page produces a claim, not a fact. The pipeline is built so that extraction, correction, canonicalization, and retrieval remain separate steps.
Pipeline
Acquire/
Register/
Extract/
Gap-fill/
Enrich/
Correction gate/
Canonicalize/
Quality gates/
Retrieve
Two-layer autonomy
A deterministic supervisor handles waiting, quota checks, locks, retries, and batch launching. A separate LLM monitor handles judgment tasks: quality checks, summaries, classifications, and finish decisions.
Correction gate
A second model pass does not overwrite canonical state. It emits atomic deltas. Additive normalized enrichments can auto-promote; removals, replacements, parse failures, and large reclassifications go to review.
Model routing
Cheap broad passes handle routine page extraction. Stronger vision models are reserved for skipped pages, high-value spreads, and deep dives. The schema is model-independent, so new models can rerun the corpus without rebuilding the product surface.
Retrieval evaluation
Retrieval is measured with a golden set rather than assumed to work. The current internal prototype reaches 93% hit@10. Misses are labeled by type, turning evaluation into a roadmap for data coverage, aliases, query rewriting, and retrieval strategy.
Cloud workload
The workload is naturally cloud-shaped: batch OCR and vision processing, structured extraction, queue-backed workers, embeddings, hybrid search indexes, evaluation sweeps, review queues, and eventually a cited answer layer.
Trajectory
| Phase | The system becomes | A user can | Status |
|---|---|---|---|
| Archive | A registered pile of scans | look at original page images | done |
| Structured index | Per-page metadata | filter by game, person, publication, content type, and tag | done |
| Refinement | Corrected canonical metadata | trust what a page says it is | done |
| Hybrid retrieval | A searchable knowledge base | search across magazines at once | here now |
| Canonical entity graph | Alias-aware identity across publications | search one name and follow the same entity across scripts and magazines | next |
| Answer layer | Cited synthesis over retrieved pages | ask a question and get an answer with issue and page citations | next |
| Public research interface | A hosted research product | research Japanese game and otaku history no normal search engine indexes | future |