feat(ingest): recover missing metadata + abstracts from arXiv/S2/Crossref (DQ-2)

Three papers lacked abstracts and 1911.00966 was fully degraded (OpenAlex 404
-> empty `Paper(id, title="")` stub). Recovery sources verified, then wired in:

- arxiv.fetch_metadata: resolve an arXiv id to a Paper via the Atom export API
  (authoritative for preprints). Used as an ingest fallback on OpenAlex 404,
  replacing the empty stub.
- crossref.py (new): fetch_abstract(doi), JATS-stripped, Crossref Polite Pool.
- semanticscholar.fetch_abstract: fetch just the abstract field.
- ingest.py: _recover_abstract() supplements an empty abstract from S2 then
  Crossref *before* embedding, so the paper gets a real (non-zero) vector.

Live DB backfill: 1911.00966 fully recovered from arXiv (bibkey
PinkallSpringborn2019, 384-char abstract, its 10 chunks now visible to chunk
search); 10.1007/s00454-019-00132-8 abstract from S2 (1186 chars). Book chapter
10.1007/978-3-642-17413-1_7 has no abstract in OpenAlex/S2/Crossref -> documented
limit. Corpus now has 1 paper without an abstract.

Also documented DQ-5 (not fixed): ingest_paper is not idempotent for DOI papers
(openalex returns full-URL id vs the bare canonical id) -> re-ingest trips
papers_openalex_id_key. Blocks roadmap R-A/R-C; needs a careful _map_paper fix.

Tests: arxiv/crossref/s2 fetchers + ingest recovery paths (342 passing).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

codex/ingest.py

@@ -13,7 +13,7 @@ from codex.db import get_conn
 from codex.embed import get_embedder
 from codex.models import Citation, Paper
 from codex.quality import classify_section, filter_chunks
-from codex.sources import openalex, semanticscholar
+from codex.sources import arxiv, crossref, openalex, semanticscholar
 
 logger = logging.getLogger(__name__)
 
@@ -71,6 +71,31 @@ def _norm_cited_id(cited_id: str) -> str:
     return cited_id.lower() if cited_id.startswith("10.") else cited_id
 
 
+def _recover_abstract(paper: Paper) -> str | None:
+    """Recover a missing abstract from S2, then Crossref (DQ-2 supplement).
+
+    Tried in order of coverage for this corpus: Semantic Scholar (indexes most
+    arXiv + journal abstracts), then Crossref (publisher-deposited, DOI only).
+    Network/parse failures degrade to None rather than aborting the ingest.
+    """
+    s2_id = _s2_id_for(paper.id)
+    if s2_id is not None:
+        try:
+            abstract = semanticscholar.fetch_abstract(s2_id)
+            if abstract and abstract.strip():
+                return abstract.strip()
+        except Exception:
+            logger.warning("S2 abstract recovery failed for %s", paper.id, exc_info=True)
+    if paper.id.startswith("10."):
+        try:
+            abstract = crossref.fetch_abstract(paper.id)
+            if abstract and abstract.strip():
+                return abstract.strip()
+        except Exception:
+            logger.warning("Crossref abstract recovery failed for %s", paper.id, exc_info=True)
+    return None
+
+
 def _s2_reference_supplement(paper: Paper) -> list[Citation]:
     """Fetch a paper's references from Semantic Scholar (DQ-1 supplement).
 
@@ -136,16 +161,25 @@ def ingest_paper(
             len(paper_id) > 4 and paper_id[4:5] == "." and paper_id[:4].isdigit()
         )
         if looks_like_arxiv:
-            # OpenAlex 404 on an arXiv id → keep a minimal stub. Citation
-            # recovery from S2 happens in the citation block below (DQ-1
-            # supplement), so no probe fetch is needed here.
-            paper = Paper(id=paper_id, title="")
+            # OpenAlex 404 on an arXiv id → recover authoritative metadata from
+            # the arXiv API (DQ-2); fall back to a minimal stub if arXiv also
+            # has nothing. Citation recovery from S2 happens in the citation
+            # block below (DQ-1 supplement).
+            paper = arxiv.fetch_metadata(paper_id) or Paper(id=paper_id, title="")
         else:
             raise ValueError(f"Paper not found: {paper_id}")
 
     if paper is None:
         raise ValueError(f"Paper not found: {paper_id}")
 
+    # DQ-2: OpenAlex often has the paper but no abstract (it cannot redistribute
+    # some publishers' abstracts). Supplement from S2, then Crossref, so the
+    # paper gets a real (non-zero) abstract embedding instead of a zero vector.
+    if not (paper.abstract or "").strip():
+        recovered = _recover_abstract(paper)
+        if recovered:
+            paper.abstract = recovered
+
     # Auto-generate bibkey when source has none (D-04).
     if paper.bibkey is None:
         paper.bibkey = _make_bibkey(paper)