Data-quality roadmap: R-A–R-F + DQ-5 fixes (citations 590→1075, section signal 10%→95%) #16

Merged
user2595 merged 31 commits from integration/roadmap into main 2026-06-27 07:12:31 +00:00
Owner

Consolidates the data-quality roadmap (developed off audit/loop-1 on five feature branches) and reconciles it with the audit-remediation work already on main (#12–15).

What's included

  • R-A — GROBID reference backfill on PDFs + native arm64 GROBID on the Jetson; citing coverage 27/29 → 29/29.
  • R-B — Crossref references as a third citation fallback (OpenAlex-empty → S2 → Crossref).
  • R-C — section-aware multi-file .tex ingest (\input/\include flattening + per-\section chunking) + legacy bare-gzip source handling.
  • R-Dgraph report warns on low citing-paper coverage, not just paper count.
  • R-F — richer section labels: store the real \section title (hybrid with the controlled vocab); section signal 10% → 95% non-body.
  • DQ-5 — DOI + arXiv-DOI re-ingest idempotency (openalex._canonical_id).
  • Infra: tmux-on-Jetson SSH-stability TODO; on-demand GROBID.

Reconciliation with main (#12–15)

Merged origin/main in and resolved 4 conflicts:

  • cli.py — R-D citing-coverage warning and main's small-corpus JSON / hub-title changes (both kept).
  • ingest.py — C-7 (paper.id = caller's bare id) pinned first, then DQ-2 abstract recovery.
  • test_ingest.py / test_openalex.py — both branches' tests kept; fixed an auto-merged contradictory paper.id assertion.
  • C-7 ↔ DQ-5 overlap (both canonicalise papers.id): kept both layers — openalex._canonical_id normalises fetch_paper's id (DQ-5); ingest pins to the caller's bare id (C-7); they converge on the bare canonical id.

Corpus impact (live Jetson DB)

  • Citations 590 → 1075; citing coverage 29/29; section signal 10% → 95% non-body.

Validation

  • 402 tests pass; ruff, ruff-format, mypy all clean on the reconciled branch. Conflict-free / mergeable into main.

Full record: docs/audit/DATA-QUALITY-2026-06-15.md (DQ-1..DQ-5 + R-A..R-F resolution blocks) and docs/SESSION-2026-06-17-open-items.md.

🤖 Generated with Claude Code

Consolidates the data-quality roadmap (developed off `audit/loop-1` on five feature branches) and **reconciles it with the audit-remediation work already on `main` (#12–15)**. ## What's included - **R-A** — GROBID reference backfill on PDFs + native arm64 GROBID on the Jetson; citing coverage 27/29 → **29/29**. - **R-B** — Crossref references as a third citation fallback (OpenAlex-empty → S2 → Crossref). - **R-C** — section-aware multi-file `.tex` ingest (`\input`/`\include` flattening + per-`\section` chunking) + legacy bare-gzip source handling. - **R-D** — `graph report` warns on low *citing-paper* coverage, not just paper count. - **R-F** — richer section labels: store the real `\section` title (hybrid with the controlled vocab); section signal **10% → 95%** non-`body`. - **DQ-5** — DOI + arXiv-DOI re-ingest idempotency (`openalex._canonical_id`). - Infra: tmux-on-Jetson SSH-stability TODO; on-demand GROBID. ## Reconciliation with main (#12–15) Merged `origin/main` in and resolved 4 conflicts: - `cli.py` — R-D citing-coverage warning **and** main's small-corpus JSON / hub-title changes (both kept). - `ingest.py` — C-7 (`paper.id` = caller's bare id) pinned first, then DQ-2 abstract recovery. - `test_ingest.py` / `test_openalex.py` — both branches' tests kept; fixed an auto-merged contradictory `paper.id` assertion. - **C-7 ↔ DQ-5 overlap** (both canonicalise `papers.id`): kept both layers — `openalex._canonical_id` normalises `fetch_paper`'s id (DQ-5); ingest pins to the caller's bare id (C-7); they converge on the bare canonical id. ## Corpus impact (live Jetson DB) - Citations 590 → **1075**; citing coverage **29/29**; section signal **10% → 95%** non-`body`. ## Validation - **402 tests pass**; ruff, ruff-format, mypy all clean on the reconciled branch. **Conflict-free / mergeable** into `main`. Full record: `docs/audit/DATA-QUALITY-2026-06-15.md` (DQ-1..DQ-5 + R-A..R-F resolution blocks) and `docs/SESSION-2026-06-17-open-items.md`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
user2595 added 28 commits 2026-06-22 09:42:48 +00:00
`codex search paper` crashed on the live DB with "operator does not exist:
vector <-> double precision[]". The pgvector `<->` operator needs its RHS
typed as vector; peer call sites (mcp_server.py, wiki.py) cast but the CLI
did not. Mocked unit tests never exercise real pgvector, so it went unseen.

Add the cast to both `<->` occurrences and a regression guard asserting the
SQL contains `::vector`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
12/29 papers had zero citations: OpenAlex indexes arXiv preprints and theses
without a parsed reference list (verified referenced_works_count=0 server-side
for all 11 with an openalex_id). Not an ingest bug — a source-coverage limit.

- ingest.py: when OpenAlex returns no references, fall back to a Semantic
  Scholar reference supplement (_s2_reference_supplement); citing_id rewritten
  to canonical papers.id, DOI cited-ids lowercased. Removed a now-redundant
  discarded S2 probe in the OpenAlex-404 arXiv branch.
- semanticscholar.py: fix TypeError on S2's HTTP-200 {"data": null} no-refs
  responses (.get("data", []) returns None when the key is present-but-null).
- tests: regression test for the OpenAlex-empty -> S2 path.
- Live DB backfilled idempotently: +330 citations (590->920), zero-out-edge
  papers 12->2, citing coverage 17->27/29. Only the 2 TU-Berlin depositonce
  theses remain (no references in any source).
- docs: DATA-QUALITY-2026-06-15.md — DQ-1 resolution, DQ-4 result, and a
  free-source acquisition roadmap (R-A..R-E).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three papers lacked abstracts and 1911.00966 was fully degraded (OpenAlex 404
-> empty `Paper(id, title="")` stub). Recovery sources verified, then wired in:

- arxiv.fetch_metadata: resolve an arXiv id to a Paper via the Atom export API
  (authoritative for preprints). Used as an ingest fallback on OpenAlex 404,
  replacing the empty stub.
- crossref.py (new): fetch_abstract(doi), JATS-stripped, Crossref Polite Pool.
- semanticscholar.fetch_abstract: fetch just the abstract field.
- ingest.py: _recover_abstract() supplements an empty abstract from S2 then
  Crossref *before* embedding, so the paper gets a real (non-zero) vector.

Live DB backfill: 1911.00966 fully recovered from arXiv (bibkey
PinkallSpringborn2019, 384-char abstract, its 10 chunks now visible to chunk
search); 10.1007/s00454-019-00132-8 abstract from S2 (1186 chars). Book chapter
10.1007/978-3-642-17413-1_7 has no abstract in OpenAlex/S2/Crossref -> documented
limit. Corpus now has 1 paper without an abstract.

Also documented DQ-5 (not fixed): ingest_paper is not idempotent for DOI papers
(openalex returns full-URL id vs the bare canonical id) -> re-ingest trips
papers_openalex_id_key. Blocks roadmap R-A/R-C; needs a careful _map_paper fix.

Tests: arxiv/crossref/s2 fetchers + ingest recovery paths (342 passing).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-ran chunk_text + is_quality_chunk on all 29 source .txt files vs stored
chunks. Coverage 109-114% everywhere (the >100% is the 64/512-word overlap, i.e.
full retention); stored==kept for every paper. Only 18 chunks dropped corpus-wide,
all low_alpha = garbled OCR tables in 2 scanned-book sources (0-387-29555-0_13,
depositonce-5415) — no real math content lost. Head/tail aligned (no truncation).

Closes the data-quality assessment axis: DQ-1 (citations), DQ-2 (metadata),
DQ-3 (fidelity), DQ-4 (retrieval) all resolved. DQ-5 (re-ingest idempotency)
remains, tracked separately.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
openalex._map_paper set Paper.id from the raw `doi` field, which OpenAlex
returns as a full URL (https://doi.org/10.x). The canonical papers.id (M-1
migration) is the bare, lower-cased DOI, so re-ingesting a DOI paper missed
ON CONFLICT (id) and tripped the separate papers_openalex_id_key constraint
(UniqueViolation).

Fix: _normalize_doi() strips https://doi.org//http://doi.org//doi: and
lower-cases; applied in the DOI branch of _map_paper only (W-id fallback
untouched). Also un-breaks _s2_id_for/Crossref recovery, which the URL form
silently defeated. Corrected the _SAMPLE_WORK fixture to the URL form (the bare
fixture hid the bug) + added _normalize_doi and re-ingest idempotency regression
tests.

Live verification: re-ingest of 10.1007/s00454-019-00132-8 (previously crashed)
now UPDATEs in place — single row, added_at unchanged, no stray row, citations
45->45. Unblocks roadmap R-A/R-C. Full suite 348 green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Clarify that DQ-3's 'VERIFIED CLEAN' covers the ingest pipeline (txt -> chunks)
only — it treats the .txt as ground truth. The upstream PDF/source -> .txt
extraction (done outside codex-py) is unmeasured and shown to be lossy (DQ-3's
dropped chunks were garbled OCR tables already present in the .txt). Add DQ-6 to
measure that layer; 36 source PDFs are available. Note R-A/R-C (re-ingest from
PDF/.tex) would largely sidestep it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GROBID extracts a cited DOI verbatim from PDF reference text, so it can be mixed-case, a https://doi.org/ URL, or doi:-prefixed. The PDF citation branch inserted it unchanged, while every other path (S2 supplement, OpenAlex, papers.id itself) uses the bare lower-cased DOI. Once R-A runs GROBID reference extraction over arXiv PDFs, those un-normalized cited-ids would never equal a bare-lowercase papers.id/cited_id — dangling the cross-citations and splitting one reference into case-/URL-variant graph nodes.

Extend _norm_cited_id to strip https://doi.org//http://doi.org//doi: prefixes and lower-case (mirroring openalex._normalize_doi), making it the single canonical cited-id normalizer, and route the GROBID branch through it. arXiv ids and S2 paperIds still pass through untouched.

Add test_ingest_pdf_grobid_citations_normalize_doi covering mixed-case, URL-form, and doi:-prefixed DOIs plus an arXiv-only ref.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
wiki/index.md is regenerated by `codex wiki compile` from wiki/concepts.yaml
(the tracked source), so the generated index should not be version-controlled.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Enable roadmap R-A (GROBID reference extraction) end-to-end and run it
against local Jetson infra instead of a Mac/Rosetta emulation.

Backfill:
- Add scripts/ra_grobid_backfill.py: references-only, idempotent citation
  backfill (dry-run default). Deliberately not ingest_paper(source_path=pdf),
  which re-OCRs and replaces the DQ-3-clean .txt chunks; this touches only the
  citations table (ON CONFLICT DO NOTHING). 7 tests.
- Make extract_references timeout configurable (large theses exceed the 60s
  default, especially against a slower GROBID).

Jetson infra:
- Bump grobid/grobid 0.8.2 -> 0.9.0-crf. The -crf tag is the only GROBID
  variant published as an arm64 multi-arch manifest; every 0.8.x tag and the
  full deep-learning image are amd64-only, which is why GROBID never ran on the
  aarch64 Jetson. Enable the 4g memory cap + init/ulimits per GROBID docs.
- Add docs/infra/grobid-jetson.md runbook: arm64 image rationale, the two host
  prerequisites (docker-group membership + the Compose v2 CLI plugin, both
  missing on the Jetson), service-scoped deploy, and the GET /api/isalive check.
  Verified live 2026-06-17: native arm64 pull, isalive=true, end-to-end
  extract_references on lutz-2024-thesis.pdf = 116 refs (101 with DOI/arXiv).

docs(audit): mark R-A core done (citing coverage 27/29 -> 29/29; edges 920 -> 1022).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add crossref.fetch_references(doi), which parses message.reference[].DOI into Citations (DOI-less book/unstructured refs are skipped — they cannot join the graph). Wire it as the third leg of the ingest citation fallback chain: OpenAlex-empty -> S2 -> Crossref, for DOI papers only (Crossref's works endpoint is DOI-keyed). Cited DOIs are normalized to the canonical bare lower-case form via _norm_cited_id.

As a fallback it fires only when OpenAlex and S2 both return no references, so it adds a forward-looking third source without changing the already-covered corpus.

Tests: fetch_references unit tests (DOI edges / unstructured skipped / no refs / 404) and ingest tests for both fallback directions. Full suite 361 passed; ruff + mypy clean. Live-validated: 33/40/24 refs for Springer/ACM/Wiley DOIs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GROBID is only needed during reference extraction (ra_grobid_backfill.py),
not for normal corpus operations (search, embeddings, wiki). Idle it still
holds ~3.5GB on the 8GB Jetson, so keep it stopped by default.

- infra/docker-compose.yml: grobid restart unless-stopped -> "no" (no auto-start
  on boot; db keeps unless-stopped). Start/stop papers-grobid around an R-A run.
- docs/infra/grobid-jetson.md: add "Run on-demand" section with the start/stop
  workflow, and record the refextract head-to-head evaluation on the Jetson that
  led to keeping GROBID: far worse DOI recall on this math corpus (lutz thesis
  101 usable IDs vs refextract 2) and slower per PDF despite a smaller footprint.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Flatten multi-file arXiv LaTeX (\input/\include) in arxiv.fetch_source so the full body is assembled rather than just the primary file's include skeleton, and add section-aware chunking (tex.chunk_sections) so .tex chunks never span a \section boundary and are labelled by their real heading. The ingest .tex path now produces (section, chunk) pairs, quality-gated together so labels stay aligned; the stored 'section' column gains genuine signal instead of mostly 'body' (serves DQ-3 fidelity + audit R-12).

Adds scripts/rc_tex_reingest.py to re-ingest arXiv papers from .tex (dry-run by default; replaces chunks). Tests: flatten_inputs, chunk_sections, multi-file fetch_source, section-label ingest, is_arxiv_id. Full suite 365 passed; ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Old arXiv submissions (e.g. legacy math/* papers) are served as a bare gzipped .tex rather than a .tar.gz, which tarfile.open rejected with 'invalid header'. Fall back to gunzipping the body directly when the tar parse fails, so R-C covers those papers too (recovers math/0001176 + math/0306167). Adds a regression test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
OpenAlex returns arXiv works under the arXiv DOI (10.48550/arXiv.<id>); _normalize_doi (DQ-5) left it as 10.48550/arxiv.<id>, but the corpus keys arXiv papers on the BARE id (2305.10988 / math/0603097). So re-ingesting an arXiv paper by its bare id missed INSERT ... ON CONFLICT (id) and tripped papers_openalex_id_key — the arXiv half of DQ-5, explicitly deferred there and surfaced again driving the R-C .tex re-ingest.

Add _canonical_id() (normalize DOI, then strip the 10.48550/arxiv. prefix) and use it in _map_paper. Regression tests (modern + legacy arXiv DOI, regular DOI unchanged, fetch_paper bare id). Confirmed live: re-ingesting 2305.10988 now UPDATEs in place. Full suite 370 passed; ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add graph.citing_coverage() (papers with >=1 out-edge / total) and a graph_min_citing_coverage setting (default 0.8). codex graph report now surfaces 'Citing coverage: N/M papers with out-edges (P%)' in text and JSON, and warns when the share is below threshold -- low citing coverage starves PageRank/coupling even when the paper count looks healthy (the DQ-1 sub-item). Separate from the existing total-count graph_min_corpus_size warning.

Tests: citing_coverage unit tests + CLI tests (line shown, warning fires/suppressed, JSON field). Full suite green; ruff + mypy clean. Live: graph report shows 29/29 (100%), no warning.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add quality.section_label(title, content): for section-aware .tex ingest, keep the controlled bucket (intro/theorem/proof/abstract/bibliography) when the real \section heading maps to one, else store the cleaned title verbatim (e.g. 'preliminaries', 'rigidity') instead of collapsing to 'body'. Math papers title most sections descriptively, so R-C's vocab-only mapping left ~90% as 'body'; this lifts the section signal toward near-full. .pdf/.txt/run_quality_pass keep content-based classify_section unchanged. Safe because the section column is write-only (no consumer filters on the vocab).

New _clean_title (strip LaTeX/numbering, lower-case, truncate). Tests: section_label bucket/verbatim/fallback paths + _clean_title; .tex ingest stores a descriptive heading as its title. Full suite 377 passed; ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Polish _clean_title for R-F: strip accent/escape macros (backslash-quote o to o, backslash-amp), convert tilde ties to spaces, and truncate at a word boundary so stored section titles read cleanly (mobius invariance, colin de verdiere). Tests added; ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Mac->Jetson SSH tunnel dropped repeatedly during multi-minute live writes (re-ingest, embed loops, sweeps), aborting them partway. Document the fix: run long ops on the Jetson inside tmux (DB is localhost, no tunnel; survives SSH drops), use autossh for interactive tunnels, and optionally make backfill writes per-item resilient.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Consolidates the session's loose ends: Jetson offline (blocker), R-F accent re-clean pending, 5 feature branches unmerged to main (+ merge order), tmux-on-Jetson infra TODO, and documented known limits.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
user2595 added 1 commit 2026-06-22 13:12:32 +00:00
Resolved 4 conflicts: cli.py (R-D citing-coverage warning + main's small-corpus JSON/hub-title changes both kept); ingest.py (C-7 paper.id=caller-id pin first, then DQ-2 abstract recovery); test_ingest.py + test_openalex.py (kept both branches' tests).

C-7/DQ-5 overlap (both canonicalise papers.id): kept both layers — openalex._canonical_id normalizes fetch_paper's id (DQ-5); ingest pins papers.id to the caller's bare id (C-7); they converge on the bare canonical id. Fixed the auto-merged test_fetch_paper_success that had contradictory bare/URL assertions. 402 tests pass; ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
user2595 added 1 commit 2026-06-26 20:51:23 +00:00
#1 ingest: normalize the pinned caller id (_norm_cited_id) so a non-bare DOI caller cannot store a URL-form papers.id that defeats DQ-5 idempotency / the startswith(10.) recovery gates. #2 quality.section_label: collapse to a controlled bucket only for an EXACT canonical heading; descriptive titles ('Abstract Nonsense...') keep their real title instead of being mislabelled. #4 ra_grobid_backfill: release the read connection before the slow GROBID network loop, fresh connection for the write (no idle-in-transaction across the loop over the flaky tunnel).

#5/#10 tex: flatten_inputs strips unresolved input/include at the depth cap (no literal leak on cycles); _norm_texkey strips only a single leading ./ . #6/#7 arxiv.fetch_source: keep non-.tex members resolvable for input; pick primary on an UN-commented documentclass line. #13 is_arxiv_id: also exclude http:// and arXiv-DOI forms. Tests added/updated for each.

Left as deliberate decisions: #3 (pre-section text drop is pre-existing in extract_sections; abstract stored separately), #8 (script normalizer is intentionally self-contained, already documented), #9/#11/#12 (no current trigger / tightening the gzip heuristic would reject valid old LaTeX like documentstyle / title truncation is by design on a write-only column). ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
user2595 added 1 commit 2026-06-27 07:09:41 +00:00
#1: assert a URL-form / mixed-case caller id is normalized to the bare canonical form in the papers upsert (the fix's whole point; existing DQ-5 test only used a bare caller). #4: assert ra_grobid_backfill.main() opens two separate connections (read released before the GROBID loop, fresh one for the write) rather than holding one across the loop. Both pass; ruff clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
user2595 merged commit 5aa2f7db40 into main 2026-06-27 07:12:31 +00:00
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: user2595/codex-py#16
No description provided.