user2595/codex-py
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
47 Commits 5 Branches 0 Tags
5025ff3a882923e48d5409e5b2deccd26ad43382
Commit Graph

4 Commits

Author SHA1 Message Date
Tarik Moussa
5025ff3a88 fix(ingest): support .txt sources + fix citing_id FK violation 
Two bugs exposed during first batch ingest run after D-03 fix:

1. `.txt` files were silently skipped ("Unbekannter Dateityp") — added
   plain-text read path alongside .tex and .pdf.

2. After D-03, OpenAlex resolves arXiv IDs → paper.id uses the DOI
   (https://doi.org/…) but fetch_citations emits citing_id=openalex_id
   (https://openalex.org/W…). FK constraint citations_citing_id_fkey
   → papers.id violated. Fix: rewrite citing_id to paper.id after
   fetching from OpenAlex.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-14 15:54:05 +02:00
Tarik Moussa
cd42e9abc8 fix(F-09): address review-gate findings — resource leak + idempotency 
- mathpix.py + figures.py: wrap fitz page-loop in try/finally so
  doc.close() is guaranteed even on exception (no file-handle leak)
- ingest.py: DELETE FROM formulas/figures WHERE paper_id before re-insert
  so --rich re-ingest is idempotent (BIGSERIAL has no natural UNIQUE key;
  ON CONFLICT DO NOTHING was a no-op)

Gate: 158 passed, ruff clean, mypy clean

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-14 02:23:21 +02:00
Tarik Moussa
1df9be6563 feat(F-09): rich parsing — formula + figure extraction 
- codex/parsing/mathpix.py: pix2tex (local, CPU) primary + MathPix API
  optional; bbox heuristic h>15px, math-char-count>5; singleton model cache
- codex/parsing/figures.py: pymupdf embedded-image extraction → PNG;
  caption detection via proximity + "Figure/Fig./Abbildung" prefix
- codex/models.py: FormulaChunk + FigureChunk dataclasses (R-10/R-11)
- codex/ingest.py: --rich flag wires formula+figure extraction post-ingest
- codex/cli.py: search_app sub-typer (paper + formula subcommands),
  --rich flag on ingest; wiki_app from F-12 preserved intact
- codex/config.py: mathpix_app_id/key, pix2tex_fallback, figures_dir
- infra/schema.sql: formulas + figures tables with HNSW pgvector indexes
- pyproject.toml: pymupdf>=1.24, pix2tex>=0.1.4
- tests/parsing/test_mathpix.py + test_figures.py: 31 tests (mock pix2tex
  + MathPix HTTP, real pymupdf on synthetic PDF)

Gate: 158 passed, ruff clean, mypy clean (20 files)
Requirements: R-10 R-11 R-12 R-13 R-14 → done

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-14 01:16:24 +02:00
Tarik Moussa
5344975bb1 feat(ingest): end-to-end idempotent ingest pipeline 
Implements codex/ingest.py with:
- OpenAlex primary / Semantic Scholar fallback metadata fetch
- Abstract dense embedding (zero-vector for missing abstracts)
- Idempotent paper upsert (ON CONFLICT DO UPDATE)
- Source file parsing (.tex via latex_to_text, .pdf via nougat + grobid)
- Chunk deletion + bulk re-insert with dense embeddings
- Citation merge from OpenAlex/S2 API and GROBID PDF refs (dedup via set)
- 7 tests covering basic, tex, pdf, not-found, idempotent, arxiv-fallback,
  and no-abstract scenarios (all mocked, offline)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-05 07:04:12 +02:00