feat(ingest): recover missing metadata + abstracts from arXiv/S2/Crossref (DQ-2)

Three papers lacked abstracts and 1911.00966 was fully degraded (OpenAlex 404
-> empty `Paper(id, title="")` stub). Recovery sources verified, then wired in:

- arxiv.fetch_metadata: resolve an arXiv id to a Paper via the Atom export API
  (authoritative for preprints). Used as an ingest fallback on OpenAlex 404,
  replacing the empty stub.
- crossref.py (new): fetch_abstract(doi), JATS-stripped, Crossref Polite Pool.
- semanticscholar.fetch_abstract: fetch just the abstract field.
- ingest.py: _recover_abstract() supplements an empty abstract from S2 then
  Crossref *before* embedding, so the paper gets a real (non-zero) vector.

Live DB backfill: 1911.00966 fully recovered from arXiv (bibkey
PinkallSpringborn2019, 384-char abstract, its 10 chunks now visible to chunk
search); 10.1007/s00454-019-00132-8 abstract from S2 (1186 chars). Book chapter
10.1007/978-3-642-17413-1_7 has no abstract in OpenAlex/S2/Crossref -> documented
limit. Corpus now has 1 paper without an abstract.

Also documented DQ-5 (not fixed): ingest_paper is not idempotent for DOI papers
(openalex returns full-URL id vs the bare canonical id) -> re-ingest trips
papers_openalex_id_key. Blocks roadmap R-A/R-C; needs a careful _map_paper fix.

Tests: arxiv/crossref/s2 fetchers + ingest recovery paths (342 passing).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

codex/sources/arxiv.py

@@ -1,6 +1,7 @@
 """arXiv API client.
 
 Provides:
+- fetch_metadata: resolve an arXiv ID to a Paper via the Atom export API.
 - fetch_source: download the .tar.gz source of a paper and extract the primary .tex file.
 - fetch_pdf_url: return the canonical PDF URL for a given arXiv ID.
 """
@@ -10,12 +11,89 @@ from __future__ import annotations
 import io
 import logging
 import tarfile
+import xml.etree.ElementTree as ET
 
 import httpx
+from tenacity import retry, retry_if_exception, stop_after_attempt, wait_exponential
+
+from codex.models import Paper
 
 logger = logging.getLogger(__name__)
 
 _BASE = "https://arxiv.org"
+_EXPORT = "https://export.arxiv.org"
+_ATOM = "{http://www.w3.org/2005/Atom}"
+
+
+def _is_retryable(exc: BaseException) -> bool:
+    if isinstance(exc, httpx.HTTPStatusError):
+        return exc.response.status_code == 429 or exc.response.status_code >= 500
+    return False
+
+
+@retry(
+    retry=retry_if_exception(_is_retryable),
+    stop=stop_after_attempt(4),
+    wait=wait_exponential(min=1, max=20),
+    before_sleep=lambda rs: logger.warning("arXiv retry %d", rs.attempt_number),
+)
+def _query(arxiv_id: str) -> httpx.Response:
+    response = httpx.get(
+        f"{_EXPORT}/api/query",
+        params={"id_list": arxiv_id},
+        timeout=30,
+        follow_redirects=True,
+    )
+    response.raise_for_status()
+    return response
+
+
+def fetch_metadata(arxiv_id: str) -> Paper | None:
+    """Resolve an arXiv ID to a Paper via the Atom export API.
+
+    Authoritative metadata source for arXiv preprints — used as an ingest
+    fallback when OpenAlex 404s on an arXiv id (DQ-2). The arXiv id (bare,
+    e.g. ``"1911.00966"`` or legacy ``"math/0603097"``) becomes ``Paper.id``;
+    ``openalex_id`` and ``bibkey`` are left for the caller to populate.
+
+    Returns
+    -------
+    Paper | None
+        Populated Paper (title, authors, year, abstract), or None if arXiv
+        has no entry for the id.
+    """
+    bare = arxiv_id[len("arxiv:") :] if arxiv_id.lower().startswith("arxiv:") else arxiv_id
+    try:
+        response = _query(bare)
+    except httpx.HTTPError:
+        logger.warning("arXiv metadata fetch failed for %s", bare, exc_info=True)
+        return None
+    try:
+        root = ET.fromstring(response.text)
+    except ET.ParseError:
+        return None
+    entry = root.find(f"{_ATOM}entry")
+    if entry is None:
+        return None
+    # An id-not-found query still returns a feed but with no <entry>.
+    title = (entry.findtext(f"{_ATOM}title") or "").strip()
+    if not title:
+        return None
+    summary = (entry.findtext(f"{_ATOM}summary") or "").strip()
+    published = entry.findtext(f"{_ATOM}published") or ""
+    year = int(published[:4]) if published[:4].isdigit() else None
+    authors = [
+        name.strip()
+        for a in entry.findall(f"{_ATOM}author")
+        if (name := a.findtext(f"{_ATOM}name")) and name.strip()
+    ]
+    return Paper(
+        id=bare,
+        title=" ".join(title.split()),
+        authors=authors,
+        year=year,
+        abstract=" ".join(summary.split()) or None,
+    )
 
 
 def fetch_source(arxiv_id: str) -> str | None: