feat(ingest): recover missing metadata + abstracts from arXiv/S2/Crossref (DQ-2)
Three papers lacked abstracts and 1911.00966 was fully degraded (OpenAlex 404 -> empty `Paper(id, title="")` stub). Recovery sources verified, then wired in: - arxiv.fetch_metadata: resolve an arXiv id to a Paper via the Atom export API (authoritative for preprints). Used as an ingest fallback on OpenAlex 404, replacing the empty stub. - crossref.py (new): fetch_abstract(doi), JATS-stripped, Crossref Polite Pool. - semanticscholar.fetch_abstract: fetch just the abstract field. - ingest.py: _recover_abstract() supplements an empty abstract from S2 then Crossref *before* embedding, so the paper gets a real (non-zero) vector. Live DB backfill: 1911.00966 fully recovered from arXiv (bibkey PinkallSpringborn2019, 384-char abstract, its 10 chunks now visible to chunk search); 10.1007/s00454-019-00132-8 abstract from S2 (1186 chars). Book chapter 10.1007/978-3-642-17413-1_7 has no abstract in OpenAlex/S2/Crossref -> documented limit. Corpus now has 1 paper without an abstract. Also documented DQ-5 (not fixed): ingest_paper is not idempotent for DOI papers (openalex returns full-URL id vs the bare canonical id) -> re-ingest trips papers_openalex_id_key. Blocks roadmap R-A/R-C; needs a careful _map_paper fix. Tests: arxiv/crossref/s2 fetchers + ingest recovery paths (342 passing). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -107,6 +107,24 @@ def fetch_references(paper_id: str) -> list[Citation]:
|
||||
return citations
|
||||
|
||||
|
||||
def fetch_abstract(paper_id: str) -> str | None:
|
||||
"""Fetch just the abstract for a paper from Semantic Scholar.
|
||||
|
||||
Used as an ingest abstract supplement (DQ-2) when OpenAlex has the paper
|
||||
but no abstract. ``paper_id`` should be namespaced (``arXiv:…`` / ``DOI:…``).
|
||||
Returns None on 404 or when S2 has no abstract.
|
||||
"""
|
||||
url = f"{_BASE_GRAPH}/paper/{paper_id}"
|
||||
try:
|
||||
response = _get(url, params={"fields": "abstract"})
|
||||
except httpx.HTTPStatusError as exc:
|
||||
if exc.response.status_code == 404:
|
||||
return None
|
||||
raise
|
||||
abstract = response.json().get("abstract")
|
||||
return abstract or None
|
||||
|
||||
|
||||
def fetch_recommendations(paper_id: str, limit: int = 20) -> list[str]:
|
||||
"""Fetch recommended paper IDs from Semantic Scholar.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user