Commit Graph

4 Commits

Author SHA1 Message Date
Tarik Moussa
5dfa6e4bae feat(graph): paper_identifiers table — multi-ID alias support
Fixes preprint/journal duplicate problem: arXiv papers were ingested
under their preprint OpenAlex ID, but corpus citations reference the
published-version OpenAlex ID → those citations were invisible in the
graph (appeared as dangling).

Changes:
- infra/schema.sql: CREATE TABLE paper_identifiers (paper_id, openalex_id)
  with UNIQUE index; seeded from papers.openalex_id on migration
- codex/graph.py: build_citation_graph LEFT JOINs paper_identifiers as
  a second resolution path: COALESCE(p.id, pi.paper_id, c.cited_id)
- codex/ingest.py: every ingest inserts openalex_id into paper_identifiers
  (ON CONFLICT DO NOTHING) — aliases can be added manually alongside

Live DB: 35 rows seeded + 4 published-version aliases added:
  math/0503219 → W2163787581 (DCG 2007)
  math/0306167 → W2136126748 (Commun Analysis Geom 2004)
  math/0203250 → W1567166970 (Trans AMS 2003)
  1005.2698    → W1511400044 (Geom & Topol 2015)

Effect: dangling 574→570; Bobenko+Springborn 2007 now IN-KB rank 9.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-15 07:42:21 +02:00
9647897173 feat(F-16): chunk quality gate + section classification
3-signal quality filter (length/alpha-ratio/bib-score) + rule-based section classifier + retroactive CLI pass. 35 new tests, 287 total green.
2026-06-15 01:22:44 +00:00
Tarik Moussa
1df9be6563 feat(F-09): rich parsing — formula + figure extraction
- codex/parsing/mathpix.py: pix2tex (local, CPU) primary + MathPix API
  optional; bbox heuristic h>15px, math-char-count>5; singleton model cache
- codex/parsing/figures.py: pymupdf embedded-image extraction → PNG;
  caption detection via proximity + "Figure/Fig./Abbildung" prefix
- codex/models.py: FormulaChunk + FigureChunk dataclasses (R-10/R-11)
- codex/ingest.py: --rich flag wires formula+figure extraction post-ingest
- codex/cli.py: search_app sub-typer (paper + formula subcommands),
  --rich flag on ingest; wiki_app from F-12 preserved intact
- codex/config.py: mathpix_app_id/key, pix2tex_fallback, figures_dir
- infra/schema.sql: formulas + figures tables with HNSW pgvector indexes
- pyproject.toml: pymupdf>=1.24, pix2tex>=0.1.4
- tests/parsing/test_mathpix.py + test_figures.py: 31 tests (mock pix2tex
  + MathPix HTTP, real pymupdf on synthetic PDF)

Gate: 158 passed, ruff clean, mypy clean (20 files)
Requirements: R-10 R-11 R-12 R-13 R-14 → done

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-14 01:16:24 +02:00
Tarik Moussa
e7ac7766a5 feat: initial project scaffold
pyproject.toml (Python 3.12, uv), codex/ package (config, db, models),
infra/ (docker-compose + schema), .env.example, .gitignore, README.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 11:19:10 +02:00