feat(F-16): chunk quality gate + section classification

- codex/quality.py: 3-signal filter (length / alpha-ratio / bib-score)
  + rule-based section classifier + run_quality_pass retroactive DB pass
- codex/ingest.py: promote quality imports to module level; apply
  filter_chunks before embedding; store section column in chunks INSERT
- codex/config.py: CHUNK_MIN_CHARS / CHUNK_MIN_ALPHA_RATIO / CHUNK_MAX_BIB_SCORE
- infra/schema.sql: ALTER TABLE chunks ADD COLUMN IF NOT EXISTS section TEXT
- .env.example: document F-16 quality thresholds
- codex/cli.py: quality run sub-command (scope by --paper-id or all papers)
- tests/quality/: 35 new tests covering all quality functions
- tests/ingest/: patch filter_chunks in source-path tests to isolate ingest

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-15 03:22:11 +02:00
parent 1a9afb4433
commit b61d84d4bd
9 changed files with 529 additions and 17 deletions

View File

@@ -27,3 +27,8 @@ OPENALEX_MAILTO=you@example.com
# Nougat HTTP-Server (Jetson: http://192.168.178.103:8080 | lokal: http://localhost:8080)
# Used by `codex ingest <id> --rich` for full-PDF Mathpix Markdown output.
NOUGAT_URL=http://localhost:8080
# F-16 Chunk Quality Gate thresholds (all optional — defaults shown)
CHUNK_MIN_CHARS=60 # Discard chunks shorter than this many characters
CHUNK_MIN_ALPHA_RATIO=0.40 # Discard chunks with < 40% alphabetic characters
CHUNK_MAX_BIB_SCORE=0.70 # Discard chunks scoring above this bibliography threshold