fix(review): address PR #16 code-review findings

#1 ingest: normalize the pinned caller id (_norm_cited_id) so a non-bare DOI caller cannot store a URL-form papers.id that defeats DQ-5 idempotency / the startswith(10.) recovery gates. #2 quality.section_label: collapse to a controlled bucket only for an EXACT canonical heading; descriptive titles ('Abstract Nonsense...') keep their real title instead of being mislabelled. #4 ra_grobid_backfill: release the read connection before the slow GROBID network loop, fresh connection for the write (no idle-in-transaction across the loop over the flaky tunnel).

#5/#10 tex: flatten_inputs strips unresolved input/include at the depth cap (no literal leak on cycles); _norm_texkey strips only a single leading ./ . #6/#7 arxiv.fetch_source: keep non-.tex members resolvable for input; pick primary on an UN-commented documentclass line. #13 is_arxiv_id: also exclude http:// and arXiv-DOI forms. Tests added/updated for each.

Left as deliberate decisions: #3 (pre-section text drop is pre-existing in extract_sections; abstract stored separately), #8 (script normalizer is intentionally self-contained, already documented), #9/#11/#12 (no current trigger / tightening the gzip heuristic would reject valid old LaTeX like documentstyle / title truncation is by design on a write-only column). ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-26 22:50:23 +02:00
parent 9e5184385a
commit b2f01ce010
11 changed files with 190 additions and 75 deletions

View File

@@ -272,16 +272,25 @@ class TestCleanTitle:
class TestSectionLabel:
def test_title_maps_to_controlled_bucket(self):
# Headings matching the controlled vocab keep the canonical bucket.
def test_bare_canonical_heading_maps_to_bucket(self):
# Only an EXACT canonical heading collapses to its controlled bucket.
assert section_label("Introduction", "body text") == "intro"
assert section_label("Proof of Theorem 3", "body text") == "proof"
assert section_label("Proof", "body text") == "proof"
assert section_label("References", "body text") == "bibliography"
assert section_label("Abstract", "body text") == "abstract"
def test_descriptive_title_stored_verbatim(self):
# R-F win: a non-vocab heading is stored as its cleaned title, not "body".
# R-F: a heading that merely STARTS with a keyword keeps its real title,
# not the bucket — fixes classify_section prefix-match over-bucketing.
assert section_label("Preliminaries", "body text") == "preliminaries"
assert section_label("3.1 Discrete Conformal Maps", "x") == "discrete conformal maps"
assert section_label("Proof of Theorem 3", "x") == "proof of theorem 3"
assert section_label("Abstract Nonsense and Categories", "x") == (
"abstract nonsense and categories"
)
assert section_label("Introduction to Operator Algebras", "x") == (
"introduction to operator algebras"
)
def test_none_or_blank_title_falls_back_to_content(self):
# .pdf/.txt path: unchanged content-based classification.