feat(F-09): rich parsing — formula + figure extraction
- codex/parsing/mathpix.py: pix2tex (local, CPU) primary + MathPix API optional; bbox heuristic h>15px, math-char-count>5; singleton model cache - codex/parsing/figures.py: pymupdf embedded-image extraction → PNG; caption detection via proximity + "Figure/Fig./Abbildung" prefix - codex/models.py: FormulaChunk + FigureChunk dataclasses (R-10/R-11) - codex/ingest.py: --rich flag wires formula+figure extraction post-ingest - codex/cli.py: search_app sub-typer (paper + formula subcommands), --rich flag on ingest; wiki_app from F-12 preserved intact - codex/config.py: mathpix_app_id/key, pix2tex_fallback, figures_dir - infra/schema.sql: formulas + figures tables with HNSW pgvector indexes - pyproject.toml: pymupdf>=1.24, pix2tex>=0.1.4 - tests/parsing/test_mathpix.py + test_figures.py: 31 tests (mock pix2tex + MathPix HTTP, real pymupdf on synthetic PDF) Gate: 158 passed, ruff clean, mypy clean (20 files) Requirements: R-10 R-11 R-12 R-13 R-14 → done Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -88,3 +88,37 @@ CREATE INDEX code_links_paper_idx ON code_links (paper_id);
|
||||
-- GROUP BY cited_id
|
||||
-- ORDER BY pull DESC
|
||||
-- LIMIT 20;
|
||||
|
||||
-- ---------------------------------------------------------------------
|
||||
-- F-09 Rich Parsing: formulas + figures
|
||||
-- ---------------------------------------------------------------------
|
||||
CREATE TABLE IF NOT EXISTS formulas (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
paper_id TEXT REFERENCES papers(id) ON DELETE CASCADE,
|
||||
page INT,
|
||||
raw_latex TEXT NOT NULL,
|
||||
context TEXT,
|
||||
eq_label TEXT,
|
||||
embedding vector(1024)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS formulas_emb_idx
|
||||
ON formulas USING hnsw (embedding vector_cosine_ops);
|
||||
CREATE INDEX IF NOT EXISTS formulas_paper_idx ON formulas (paper_id);
|
||||
CREATE INDEX IF NOT EXISTS formulas_fts_idx
|
||||
ON formulas USING gin (
|
||||
to_tsvector('english', raw_latex || ' ' || coalesce(context, ''))
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS figures (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
paper_id TEXT REFERENCES papers(id) ON DELETE CASCADE,
|
||||
page INT,
|
||||
caption TEXT,
|
||||
image_path TEXT,
|
||||
embedding vector(1024)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS figures_emb_idx
|
||||
ON figures USING hnsw (embedding vector_cosine_ops);
|
||||
CREATE INDEX IF NOT EXISTS figures_paper_idx ON figures (paper_id);
|
||||
|
||||
Reference in New Issue
Block a user