feat(F-16): chunk quality gate + section classification #10

user2595 · 2026-06-15T01:22:36Z

user2595 commented

2026-06-15 01:22:36 +00:00

Summary

codex/quality.py (new): 3-signal quality predicate (chunk_min_chars / chunk_min_alpha_ratio / chunk_max_bib_score), rule-based section classifier (abstract / intro / theorem / proof / bibliography / body), run_quality_pass for retroactive DB back-fill
codex/ingest.py: promote quality imports to module level; filter_chunks applied before embedding; section column stored in chunk INSERT
codex/config.py: three new CHUNK_* settings (all env-overridable, defaults: 60 / 0.40 / 0.70)
infra/schema.sql: ALTER TABLE chunks ADD COLUMN IF NOT EXISTS section TEXT
.env.example: document F-16 thresholds
codex/cli.py: codex quality run [--paper-id ID] — retroactive quality pass over DB
tests/quality/: 35 new tests (bib-score, quality predicate, section classifier, filter, DB pass)
tests/ingest/: patch filter_chunks in source-path tests to isolate ingest from quality gate

Test plan

uv run pytest tests/quality/ — 35 passed
uv run pytest --ignore=tests/integration — 287 passed
uv run ruff check . && uv run ruff format --check . — clean
uv run mypy codex/ — no issues (24 source files)
Apply schema migration on live DB: psql papers -c "ALTER TABLE chunks ADD COLUMN IF NOT EXISTS section TEXT;"
Smoke: uv run codex quality run --paper-id 2301.07041

🤖 Generated with Claude Code

## Summary - **`codex/quality.py`** (new): 3-signal quality predicate (`chunk_min_chars` / `chunk_min_alpha_ratio` / `chunk_max_bib_score`), rule-based section classifier (abstract / intro / theorem / proof / bibliography / body), `run_quality_pass` for retroactive DB back-fill - **`codex/ingest.py`**: promote quality imports to module level; `filter_chunks` applied before embedding; `section` column stored in chunk INSERT - **`codex/config.py`**: three new `CHUNK_*` settings (all env-overridable, defaults: 60 / 0.40 / 0.70) - **`infra/schema.sql`**: `ALTER TABLE chunks ADD COLUMN IF NOT EXISTS section TEXT` - **`.env.example`**: document F-16 thresholds - **`codex/cli.py`**: `codex quality run [--paper-id ID]` — retroactive quality pass over DB - **`tests/quality/`**: 35 new tests (bib-score, quality predicate, section classifier, filter, DB pass) - **`tests/ingest/`**: patch `filter_chunks` in source-path tests to isolate ingest from quality gate ## Test plan - [x] `uv run pytest tests/quality/` — 35 passed - [x] `uv run pytest --ignore=tests/integration` — 287 passed - [x] `uv run ruff check . && uv run ruff format --check .` — clean - [x] `uv run mypy codex/` — no issues (24 source files) - [ ] Apply schema migration on live DB: `psql papers -c "ALTER TABLE chunks ADD COLUMN IF NOT EXISTS section TEXT;"` - [ ] Smoke: `uv run codex quality run --paper-id 2301.07041` 🤖 Generated with [Claude Code](https://claude.ai/claude-code)

user2595 added 1 commit 2026-06-15 01:22:37 +00:00

feat(F-16): chunk quality gate + section classification b61d84d4bd

- codex/quality.py: 3-signal filter (length / alpha-ratio / bib-score)
  + rule-based section classifier + run_quality_pass retroactive DB pass
- codex/ingest.py: promote quality imports to module level; apply
  filter_chunks before embedding; store section column in chunks INSERT
- codex/config.py: CHUNK_MIN_CHARS / CHUNK_MIN_ALPHA_RATIO / CHUNK_MAX_BIB_SCORE
- infra/schema.sql: ALTER TABLE chunks ADD COLUMN IF NOT EXISTS section TEXT
- .env.example: document F-16 quality thresholds
- codex/cli.py: quality run sub-command (scope by --paper-id or all papers)
- tests/quality/: 35 new tests covering all quality functions
- tests/ingest/: patch filter_chunks in source-path tests to isolate ingest

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

user2595 merged commit 9647897173 into main

2026-06-15 01:22:45 +00:00

user2595 deleted branch feat/F-16-chunk-quality

2026-06-15 01:22:45 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: user2595/codex-py#10