Flatten multi-file arXiv LaTeX (\input/\include) in arxiv.fetch_source so the full body is assembled rather than just the primary file's include skeleton, and add section-aware chunking (tex.chunk_sections) so .tex chunks never span a \section boundary and are labelled by their real heading. The ingest .tex path now produces (section, chunk) pairs, quality-gated together so labels stay aligned; the stored 'section' column gains genuine signal instead of mostly 'body' (serves DQ-3 fidelity + audit R-12).
Adds scripts/rc_tex_reingest.py to re-ingest arXiv papers from .tex (dry-run by default; replaces chunks). Tests: flatten_inputs, chunk_sections, multi-file fetch_source, section-label ingest, is_arxiv_id. Full suite 365 passed; ruff + mypy clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GROBID extracts a cited DOI verbatim from PDF reference text, so it can be mixed-case, a https://doi.org/ URL, or doi:-prefixed. The PDF citation branch inserted it unchanged, while every other path (S2 supplement, OpenAlex, papers.id itself) uses the bare lower-cased DOI. Once R-A runs GROBID reference extraction over arXiv PDFs, those un-normalized cited-ids would never equal a bare-lowercase papers.id/cited_id — dangling the cross-citations and splitting one reference into case-/URL-variant graph nodes.
Extend _norm_cited_id to strip https://doi.org//http://doi.org//doi: prefixes and lower-case (mirroring openalex._normalize_doi), making it the single canonical cited-id normalizer, and route the GROBID branch through it. arXiv ids and S2 paperIds still pass through untouched.
Add test_ingest_pdf_grobid_citations_normalize_doi covering mixed-case, URL-form, and doi:-prefixed DOIs plus an arXiv-only ref.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three papers lacked abstracts and 1911.00966 was fully degraded (OpenAlex 404
-> empty `Paper(id, title="")` stub). Recovery sources verified, then wired in:
- arxiv.fetch_metadata: resolve an arXiv id to a Paper via the Atom export API
(authoritative for preprints). Used as an ingest fallback on OpenAlex 404,
replacing the empty stub.
- crossref.py (new): fetch_abstract(doi), JATS-stripped, Crossref Polite Pool.
- semanticscholar.fetch_abstract: fetch just the abstract field.
- ingest.py: _recover_abstract() supplements an empty abstract from S2 then
Crossref *before* embedding, so the paper gets a real (non-zero) vector.
Live DB backfill: 1911.00966 fully recovered from arXiv (bibkey
PinkallSpringborn2019, 384-char abstract, its 10 chunks now visible to chunk
search); 10.1007/s00454-019-00132-8 abstract from S2 (1186 chars). Book chapter
10.1007/978-3-642-17413-1_7 has no abstract in OpenAlex/S2/Crossref -> documented
limit. Corpus now has 1 paper without an abstract.
Also documented DQ-5 (not fixed): ingest_paper is not idempotent for DOI papers
(openalex returns full-URL id vs the bare canonical id) -> re-ingest trips
papers_openalex_id_key. Blocks roadmap R-A/R-C; needs a careful _map_paper fix.
Tests: arxiv/crossref/s2 fetchers + ingest recovery paths (342 passing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
12/29 papers had zero citations: OpenAlex indexes arXiv preprints and theses
without a parsed reference list (verified referenced_works_count=0 server-side
for all 11 with an openalex_id). Not an ingest bug — a source-coverage limit.
- ingest.py: when OpenAlex returns no references, fall back to a Semantic
Scholar reference supplement (_s2_reference_supplement); citing_id rewritten
to canonical papers.id, DOI cited-ids lowercased. Removed a now-redundant
discarded S2 probe in the OpenAlex-404 arXiv branch.
- semanticscholar.py: fix TypeError on S2's HTTP-200 {"data": null} no-refs
responses (.get("data", []) returns None when the key is present-but-null).
- tests: regression test for the OpenAlex-empty -> S2 path.
- Live DB backfilled idempotently: +330 citations (590->920), zero-out-edge
papers 12->2, citing coverage 17->27/29. Only the 2 TU-Berlin depositonce
theses remain (no references in any source).
- docs: DATA-QUALITY-2026-06-15.md — DQ-1 resolution, DQ-4 result, and a
free-source acquisition roadmap (R-A..R-E).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixes preprint/journal duplicate problem: arXiv papers were ingested
under their preprint OpenAlex ID, but corpus citations reference the
published-version OpenAlex ID → those citations were invisible in the
graph (appeared as dangling).
Changes:
- infra/schema.sql: CREATE TABLE paper_identifiers (paper_id, openalex_id)
with UNIQUE index; seeded from papers.openalex_id on migration
- codex/graph.py: build_citation_graph LEFT JOINs paper_identifiers as
a second resolution path: COALESCE(p.id, pi.paper_id, c.cited_id)
- codex/ingest.py: every ingest inserts openalex_id into paper_identifiers
(ON CONFLICT DO NOTHING) — aliases can be added manually alongside
Live DB: 35 rows seeded + 4 published-version aliases added:
math/0503219 → W2163787581 (DCG 2007)
math/0306167 → W2136126748 (Commun Analysis Geom 2004)
math/0203250 → W1567166970 (Trans AMS 2003)
1005.2698 → W1511400044 (Geom & Topol 2015)
Effect: dangling 574→570; Bobenko+Springborn 2007 now IN-KB rank 9.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
D-04: add _make_bibkey(paper) — ConcatSurnames+Year, ≤3 authors full list,
>3 authors FirstEtAlYear. Applied in ingest_paper when paper.bibkey is
None. ON CONFLICT now fills NULL bibkeys via COALESCE(papers.bibkey,
EXCLUDED.bibkey) while preserving manually-set values.
D-05b: add codex/__main__.py so `python -m codex` dispatches to the CLI.
D-05c: fix test_ingest.py:75 — rename var to raw_api_citations (the raw
OpenAlex payload with citing_id=openalex_id) and add regression
assertion: citing_id written to DB must equal paper.id (DOI), not
paper.openalex_id.
D-05a (proxy): add TestEntryPoint.test_python_m_codex_help — runs
`sys.executable -m codex --help` via subprocess, covering the
non-uv-run entry path. 253 tests green, ruff+mypy clean.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs exposed during first batch ingest run after D-03 fix:
1. `.txt` files were silently skipped ("Unbekannter Dateityp") — added
plain-text read path alongside .tex and .pdf.
2. After D-03, OpenAlex resolves arXiv IDs → paper.id uses the DOI
(https://doi.org/…) but fetch_citations emits citing_id=openalex_id
(https://openalex.org/W…). FK constraint citations_citing_id_fkey
→ papers.id violated. Fix: rewrite citing_id to paper.id after
fetching from OpenAlex.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- mathpix.py + figures.py: wrap fitz page-loop in try/finally so
doc.close() is guaranteed even on exception (no file-handle leak)
- ingest.py: DELETE FROM formulas/figures WHERE paper_id before re-insert
so --rich re-ingest is idempotent (BIGSERIAL has no natural UNIQUE key;
ON CONFLICT DO NOTHING was a no-op)
Gate: 158 passed, ruff clean, mypy clean
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
