fix(review): address PR #16 code-review findings
#1 ingest: normalize the pinned caller id (_norm_cited_id) so a non-bare DOI caller cannot store a URL-form papers.id that defeats DQ-5 idempotency / the startswith(10.) recovery gates. #2 quality.section_label: collapse to a controlled bucket only for an EXACT canonical heading; descriptive titles ('Abstract Nonsense...') keep their real title instead of being mislabelled. #4 ra_grobid_backfill: release the read connection before the slow GROBID network loop, fresh connection for the write (no idle-in-transaction across the loop over the flaky tunnel). #5/#10 tex: flatten_inputs strips unresolved input/include at the depth cap (no literal leak on cycles); _norm_texkey strips only a single leading ./ . #6/#7 arxiv.fetch_source: keep non-.tex members resolvable for input; pick primary on an UN-commented documentclass line. #13 is_arxiv_id: also exclude http:// and arXiv-DOI forms. Tests added/updated for each. Left as deliberate decisions: #3 (pre-section text drop is pre-existing in extract_sections; abstract stored separately), #8 (script normalizer is intentionally self-contained, already documented), #9/#11/#12 (no current trigger / tightening the gzip heuristic would reject valid old LaTeX like documentstyle / title truncation is by design on a write-only column). ruff + mypy clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -207,12 +207,14 @@ def ingest_paper(
|
||||
raise ValueError(f"Paper not found: {paper_id}")
|
||||
|
||||
# Canonical identity is the caller-supplied id — what the CLI / ingest scripts
|
||||
# use and what papers.id is contracted to hold (arXiv id or DOI). OpenAlex's
|
||||
# fetch_paper fills paper.id with the URL-form doi/id (audit C-7); since DQ-5
|
||||
# `openalex._canonical_id` already normalizes that to the bare form, this pin
|
||||
# is belt-and-suspenders and keeps re-ingest idempotent regardless. The
|
||||
# OpenAlex work id is retained as openalex_id and in paper_identifiers.
|
||||
paper.id = paper_id.strip()
|
||||
# use and what papers.id is contracted to hold (arXiv id or DOI). Pin it to the
|
||||
# caller's id (audit C-7), but normalize it the same way cited-ids are (strip a
|
||||
# https://doi.org/ or doi: prefix, lower-case DOIs) so a non-bare caller cannot
|
||||
# store a URL-form papers.id that would defeat DQ-5's ON CONFLICT (id) idempotency
|
||||
# or the `paper.id.startswith("10.")` recovery gates below. arXiv/W ids pass
|
||||
# through unchanged. The OpenAlex work id is retained as openalex_id / in
|
||||
# paper_identifiers.
|
||||
paper.id = _norm_cited_id(paper_id.strip())
|
||||
|
||||
# DQ-2: OpenAlex often has the paper but no abstract (it cannot redistribute
|
||||
# some publishers' abstracts). Supplement from S2, then Crossref (uses the
|
||||
|
||||
Reference in New Issue
Block a user