Tarik Moussa
b61d84d4bd
feat(F-16): chunk quality gate + section classification
...
- codex/quality.py: 3-signal filter (length / alpha-ratio / bib-score)
+ rule-based section classifier + run_quality_pass retroactive DB pass
- codex/ingest.py: promote quality imports to module level; apply
filter_chunks before embedding; store section column in chunks INSERT
- codex/config.py: CHUNK_MIN_CHARS / CHUNK_MIN_ALPHA_RATIO / CHUNK_MAX_BIB_SCORE
- infra/schema.sql: ALTER TABLE chunks ADD COLUMN IF NOT EXISTS section TEXT
- .env.example: document F-16 quality thresholds
- codex/cli.py: quality run sub-command (scope by --paper-id or all papers)
- tests/quality/: 35 new tests covering all quality functions
- tests/ingest/: patch filter_chunks in source-path tests to isolate ingest
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-06-15 03:22:11 +02:00
Tarik Moussa
b87087d3c6
fix(ingest): D-04 bibkey heuristic + D-05a/b/c regression tests + __main__
...
D-04: add _make_bibkey(paper) — ConcatSurnames+Year, ≤3 authors full list,
>3 authors FirstEtAlYear. Applied in ingest_paper when paper.bibkey is
None. ON CONFLICT now fills NULL bibkeys via COALESCE(papers.bibkey,
EXCLUDED.bibkey) while preserving manually-set values.
D-05b: add codex/__main__.py so `python -m codex` dispatches to the CLI.
D-05c: fix test_ingest.py:75 — rename var to raw_api_citations (the raw
OpenAlex payload with citing_id=openalex_id) and add regression
assertion: citing_id written to DB must equal paper.id (DOI), not
paper.openalex_id.
D-05a (proxy): add TestEntryPoint.test_python_m_codex_help — runs
`sys.executable -m codex --help` via subprocess, covering the
non-uv-run entry path. 253 tests green, ruff+mypy clean.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-06-15 03:07:06 +02:00
Tarik Moussa
1df9be6563
feat(F-09): rich parsing — formula + figure extraction
...
- codex/parsing/mathpix.py: pix2tex (local, CPU) primary + MathPix API
optional; bbox heuristic h>15px, math-char-count>5; singleton model cache
- codex/parsing/figures.py: pymupdf embedded-image extraction → PNG;
caption detection via proximity + "Figure/Fig./Abbildung" prefix
- codex/models.py: FormulaChunk + FigureChunk dataclasses (R-10/R-11)
- codex/ingest.py: --rich flag wires formula+figure extraction post-ingest
- codex/cli.py: search_app sub-typer (paper + formula subcommands),
--rich flag on ingest; wiki_app from F-12 preserved intact
- codex/config.py: mathpix_app_id/key, pix2tex_fallback, figures_dir
- infra/schema.sql: formulas + figures tables with HNSW pgvector indexes
- pyproject.toml: pymupdf>=1.24, pix2tex>=0.1.4
- tests/parsing/test_mathpix.py + test_figures.py: 31 tests (mock pix2tex
+ MathPix HTTP, real pymupdf on synthetic PDF)
Gate: 158 passed, ruff clean, mypy clean (20 files)
Requirements: R-10 R-11 R-12 R-13 R-14 → done
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-06-14 01:16:24 +02:00
Tarik Moussa
5344975bb1
feat(ingest): end-to-end idempotent ingest pipeline
...
Implements codex/ingest.py with:
- OpenAlex primary / Semantic Scholar fallback metadata fetch
- Abstract dense embedding (zero-vector for missing abstracts)
- Idempotent paper upsert (ON CONFLICT DO UPDATE)
- Source file parsing (.tex via latex_to_text, .pdf via nougat + grobid)
- Chunk deletion + bulk re-insert with dense embeddings
- Citation merge from OpenAlex/S2 API and GROBID PDF refs (dedup via set)
- 7 tests covering basic, tex, pdf, not-found, idempotent, arxiv-fallback,
and no-abstract scenarios (all mocked, offline)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-06-05 07:04:12 +02:00
