feat(ingest): recover missing metadata + abstracts from arXiv/S2/Crossref (DQ-2)

Three papers lacked abstracts and 1911.00966 was fully degraded (OpenAlex 404
-> empty `Paper(id, title="")` stub). Recovery sources verified, then wired in:

- arxiv.fetch_metadata: resolve an arXiv id to a Paper via the Atom export API
  (authoritative for preprints). Used as an ingest fallback on OpenAlex 404,
  replacing the empty stub.
- crossref.py (new): fetch_abstract(doi), JATS-stripped, Crossref Polite Pool.
- semanticscholar.fetch_abstract: fetch just the abstract field.
- ingest.py: _recover_abstract() supplements an empty abstract from S2 then
  Crossref *before* embedding, so the paper gets a real (non-zero) vector.

Live DB backfill: 1911.00966 fully recovered from arXiv (bibkey
PinkallSpringborn2019, 384-char abstract, its 10 chunks now visible to chunk
search); 10.1007/s00454-019-00132-8 abstract from S2 (1186 chars). Book chapter
10.1007/978-3-642-17413-1_7 has no abstract in OpenAlex/S2/Crossref -> documented
limit. Corpus now has 1 paper without an abstract.

Also documented DQ-5 (not fixed): ingest_paper is not idempotent for DOI papers
(openalex returns full-URL id vs the bare canonical id) -> re-ingest trips
papers_openalex_id_key. Blocks roadmap R-A/R-C; needs a careful _map_paper fix.

Tests: arxiv/crossref/s2 fetchers + ingest recovery paths (342 passing).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-17 00:12:59 +02:00
parent 51dc74470c
commit ff03443af4
9 changed files with 521 additions and 20 deletions

View File

@@ -107,6 +107,24 @@ def fetch_references(paper_id: str) -> list[Citation]:
return citations
def fetch_abstract(paper_id: str) -> str | None:
"""Fetch just the abstract for a paper from Semantic Scholar.
Used as an ingest abstract supplement (DQ-2) when OpenAlex has the paper
but no abstract. ``paper_id`` should be namespaced (``arXiv:…`` / ``DOI:…``).
Returns None on 404 or when S2 has no abstract.
"""
url = f"{_BASE_GRAPH}/paper/{paper_id}"
try:
response = _get(url, params={"fields": "abstract"})
except httpx.HTTPStatusError as exc:
if exc.response.status_code == 404:
return None
raise
abstract = response.json().get("abstract")
return abstract or None
def fetch_recommendations(paper_id: str, limit: int = 20) -> list[str]:
"""Fetch recommended paper IDs from Semantic Scholar.