Tarik Moussa 5344975bb1 feat(ingest): end-to-end idempotent ingest pipeline ...

Implements codex/ingest.py with:
- OpenAlex primary / Semantic Scholar fallback metadata fetch
- Abstract dense embedding (zero-vector for missing abstracts)
- Idempotent paper upsert (ON CONFLICT DO UPDATE)
- Source file parsing (.tex via latex_to_text, .pdf via nougat + grobid)
- Chunk deletion + bulk re-insert with dense embeddings
- Citation merge from OpenAlex/S2 API and GROBID PDF refs (dedup via set)
- 7 tests covering basic, tex, pdf, not-found, idempotent, arxiv-fallback,
  and no-abstract scenarios (all mocked, offline)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-06-05 07:04:12 +02:00

0 lines

0 B

Python

Raw Blame History

The file is empty.