feat(F-16): chunk quality gate + section classification
- codex/quality.py: 3-signal filter (length / alpha-ratio / bib-score) + rule-based section classifier + run_quality_pass retroactive DB pass - codex/ingest.py: promote quality imports to module level; apply filter_chunks before embedding; store section column in chunks INSERT - codex/config.py: CHUNK_MIN_CHARS / CHUNK_MIN_ALPHA_RATIO / CHUNK_MAX_BIB_SCORE - infra/schema.sql: ALTER TABLE chunks ADD COLUMN IF NOT EXISTS section TEXT - .env.example: document F-16 quality thresholds - codex/cli.py: quality run sub-command (scope by --paper-id or all papers) - tests/quality/: 35 new tests covering all quality functions - tests/ingest/: patch filter_chunks in source-path tests to isolate ingest Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -27,3 +27,8 @@ OPENALEX_MAILTO=you@example.com
|
||||
# Nougat HTTP-Server (Jetson: http://192.168.178.103:8080 | lokal: http://localhost:8080)
|
||||
# Used by `codex ingest <id> --rich` for full-PDF Mathpix Markdown output.
|
||||
NOUGAT_URL=http://localhost:8080
|
||||
|
||||
# F-16 Chunk Quality Gate thresholds (all optional — defaults shown)
|
||||
CHUNK_MIN_CHARS=60 # Discard chunks shorter than this many characters
|
||||
CHUNK_MIN_ALPHA_RATIO=0.40 # Discard chunks with < 40% alphabetic characters
|
||||
CHUNK_MAX_BIB_SCORE=0.70 # Discard chunks scoring above this bibliography threshold
|
||||
|
||||
Reference in New Issue
Block a user