feat(ingest): recover missing metadata + abstracts from arXiv/S2/Crossref (DQ-2)

Three papers lacked abstracts and 1911.00966 was fully degraded (OpenAlex 404
-> empty `Paper(id, title="")` stub). Recovery sources verified, then wired in:

- arxiv.fetch_metadata: resolve an arXiv id to a Paper via the Atom export API
  (authoritative for preprints). Used as an ingest fallback on OpenAlex 404,
  replacing the empty stub.
- crossref.py (new): fetch_abstract(doi), JATS-stripped, Crossref Polite Pool.
- semanticscholar.fetch_abstract: fetch just the abstract field.
- ingest.py: _recover_abstract() supplements an empty abstract from S2 then
  Crossref *before* embedding, so the paper gets a real (non-zero) vector.

Live DB backfill: 1911.00966 fully recovered from arXiv (bibkey
PinkallSpringborn2019, 384-char abstract, its 10 chunks now visible to chunk
search); 10.1007/s00454-019-00132-8 abstract from S2 (1186 chars). Book chapter
10.1007/978-3-642-17413-1_7 has no abstract in OpenAlex/S2/Crossref -> documented
limit. Corpus now has 1 paper without an abstract.

Also documented DQ-5 (not fixed): ingest_paper is not idempotent for DOI papers
(openalex returns full-URL id vs the bare canonical id) -> re-ingest trips
papers_openalex_id_key. Blocks roadmap R-A/R-C; needs a careful _map_paper fix.

Tests: arxiv/crossref/s2 fetchers + ingest recovery paths (342 passing).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/audit/DATA-QUALITY-2026-06-15.md

@@ -2,7 +2,10 @@
 
 **Status:** baseline scan done 2026-06-15. **DQ-1 RESOLVED 2026-06-16** (S2 supplement
 applied to live DB + wired into ingest). **DQ-4 MEASURED 2026-06-16** — retrieval is
-strong after fixing a P0 `codex search paper` crash. **DQ-2/DQ-3 still open.**
+strong after fixing a P0 `codex search paper` crash. **DQ-2 RESOLVED 2026-06-16**
+(1911.00966 fully recovered from arXiv; s00454 abstract from S2; book chapter is a
+documented limit; recovery wired into ingest). **DQ-3 still open. New: DQ-5**
+(`ingest_paper` not idempotent for DOI papers — blocks R-A/R-C).
 **Roadmap R-A..R-E** added 2026-06-16 (free-source acquisition levers — see Roadmap section).
 **Author:** Audit-Loop (Opus). **Intended reader:** a *cold* session with no memory
 of the audit conversation — everything needed to continue is in this file.
@@ -110,7 +113,32 @@ incomplete, or unrepresentative data. This document is that second axis — a
   supplement, or (b) documented as an inherent OpenAlex-coverage limit with the
   graph caveated accordingly.
 
-### DQ-2 — Metadata gaps: 3 no-abstract, 1 fully metadata-less · **MED**
+### DQ-2 — Metadata gaps: 3 no-abstract, 1 fully metadata-less · **RESOLVED 2026-06-16**
+
+**Resolution (what was done):**
+- **`1911.00966` fully recovered** from the arXiv Atom API (OpenAlex 404'd on it):
+  title "A discrete version of Liouville's theorem on conformal maps",
+  Pinkall & Springborn, 2019, 384-char abstract → bibkey `PinkallSpringborn2019`
+  auto-generated, abstract embedded (non-zero). It was the worst node in the
+  corpus — now `@cite`-able, in wiki grounding, and its **10 chunks are visible
+  to chunk search** (previously filtered out by `bibkey IS NOT NULL`). Citations
+  were already added in DQ-1 (27 S2 refs).
+- **`10.1007/s00454-019-00132-8` abstract recovered from S2** (1186 chars) and
+  re-embedded (targeted `UPDATE`, not full re-ingest — see DQ-5).
+- **`10.1007/978-3-642-17413-1_7` (book chapter): documented inherent limit.**
+  No abstract in OpenAlex, S2, **or Crossref** (verified). Left as the single
+  remaining zero-vector paper; user may add an abstract by hand.
+- **Corpus now: 1 paper without abstract** (the book chapter), down from 3+1.
+  Zero-vector search pollution (DQ-4 secondary finding) reduced from 2 tail
+  fillers to 1.
+- **Wired into ingest** so this self-heals: OpenAlex-404 on an arXiv id now
+  recovers metadata via `arxiv.fetch_metadata` (new); an empty abstract is
+  supplemented via `_recover_abstract` → S2 then Crossref (new
+  `codex/sources/crossref.py`). New `semanticscholar.fetch_abstract`.
+- **Files touched:** `codex/sources/arxiv.py`, `codex/sources/crossref.py` (new),
+  `codex/sources/semanticscholar.py`, `codex/ingest.py`, + tests. Full suite: 342.
+
+**Original finding (for context):**
 - **Measured:**
   - **No abstract (3):** `10.1007/978-3-642-17413-1_7`,
     `10.1007/s00454-019-00132-8`, `1911.00966`. These get a **zero-vector**
@@ -187,6 +215,30 @@ is fine (monotonic); only the absolute score label is misleading.
 - **Acceptance:** a short relevance table (query → top-5 → on/off-topic) good
   enough to trust the KB for lookups, or a list of failure modes to fix.
 
+### DQ-5 — `ingest_paper` is NOT idempotent for DOI papers · **HIGH (blocks R-A/R-C)** · found 2026-06-16
+- **Symptom:** re-ingesting an existing DOI paper crashes with
+  `UniqueViolation: papers_openalex_id_key`. Surfaced trying to re-ingest
+  `10.1007/s00454-019-00132-8` during the DQ-2 backfill.
+- **Root cause:** `openalex._map_paper` sets `Paper.id = data["doi"]`, which is the
+  **full URL** `https://doi.org/10.1007/…`. The stored canonical id is the **bare**
+  `10.1007/…` (post the M-1 canonical-id migration). So `INSERT … ON CONFLICT (id)`
+  finds no id match, attempts a fresh INSERT, and trips the *separate* unique
+  constraint on `openalex_id` (already held by the canonical-id row).
+- **Confirmed:** `openalex.fetch_paper("10.1007/s00454-019-00132-8").id ==
+  "https://doi.org/10.1007/s00454-019-00132-8"` (≠ stored bare id).
+- **Impact:** any re-ingest of a DOI paper aborts. This **blocks roadmap R-A
+  (GROBID on PDFs) and R-C (.tex ingest)** — both re-ingest existing papers. Also
+  implies *new* DOI ingests store full-URL ids, inconsistent with the bare-id
+  corpus (latent split-identity risk).
+- **Proposed fix (handle with care — M-1 incident territory):** normalize the DOI
+  in `_map_paper` to the canonical bare, lower-cased form (strip
+  `https://doi.org/`), matching what the M-1 migration produced. Add a re-ingest
+  idempotency test over a DOI paper. Did **not** fix inline this session — the
+  canonical-id derivation is exactly where the M-1 migration incident occurred, so
+  it needs its own focused change + a live re-ingest check.
+- **Workaround used for DQ-2:** targeted `UPDATE … SET abstract, abstract_emb`
+  instead of full re-ingest.
+
 ---
 
 ## Roadmap — data-acquisition levers (post-audit, all free sources)
@@ -200,6 +252,8 @@ Paywall/uni-login was explicitly considered and **deferred** (see R-E). Each ite
 is self-contained so a cold session can pick it up.
 
 ### R-A — Run GROBID reference extraction on arXiv PDFs · **HIGH lever, LOW-MED effort**
+- **⚠ Blocked by DQ-5:** re-ingesting existing DOI papers currently crashes
+  (`papers_openalex_id_key`). Fix DQ-5 first, or the GROBID re-ingest aborts.
 - **What:** the whole corpus was ingested from `.txt` (`PAPERS_DIR=…/papers/txt`),
   so the GROBID PDF path never ran. Re-ingest with a PDF `source_path` per paper
   so `codex.parsing.grobid.extract_references` parses each bibliography.
@@ -268,18 +322,21 @@ is self-contained so a cold session can pick it up.
    coverage; remedy wired into ingest).
 2. ~~**DQ-4**~~ — **DONE 2026-06-16** (P0 `search paper` crash fixed; relevance
    verified strong; surfaced the zero-vector pollution that motivates DQ-2).
-3. **DQ-2** (NEXT — recover `1911.00966`; backfill 3 missing abstracts so they
-   stop polluting paper-level search with zero-vectors), then **DQ-3** (chunk-vs-
-   source `.txt` fidelity).
-4. **Roadmap levers** (see section above), best sequenced after DQ-2/DQ-3:
-   **R-A** (GROBID on arXiv PDFs — closes the last 2 zero-citation theses, highest
-   ROI), then **R-B** (Crossref source — also recovers DQ-2 abstracts), **R-C**
-   (`.tex` ingest — serves DQ-3), **R-D** (F-15 coverage warning, quick).
-   **R-E** (paywall) is deferred.
+3. ~~**DQ-2**~~ — **DONE 2026-06-16** (1911.00966 fully recovered; s00454 abstract
+   from S2; book chapter documented; recovery wired into ingest).
+4. **DQ-5** (NEXT — `ingest_paper` not idempotent for DOI papers; **blocks R-A/R-C**;
+   careful — M-1 canonical-id territory), then **DQ-3** (chunk-vs-source fidelity).
+5. **Roadmap levers** (see section above): **R-A** (GROBID on arXiv PDFs — closes
+   the last 2 zero-citation theses; needs DQ-5 first), **R-B** (Crossref refs —
+   abstract half already shipped in DQ-2), **R-C** (`.tex` ingest — serves DQ-3;
+   needs DQ-5 first), **R-D** (F-15 coverage warning, quick). **R-E** (paywall)
+   deferred.
 
-DQ-1..DQ-4 are *data*/assessment work; the roadmap items R-A..R-D add **new code**
-(re-ingest scripts, a Crossref source module, a `.tex` path, a graph warning) on
-top of the already-remediated pipeline (see the AUDIT-* docs and PRs #12–#14).
+DQ-1..DQ-4 are *data*/assessment work; DQ-5 + roadmap R-A..R-D add **new code**
+(canonical-id fix, re-ingest scripts, Crossref references, a `.tex` path, a graph
+warning) on top of the already-remediated pipeline (see the AUDIT-* docs and PRs
+#12–#14). The DQ-1/DQ-2 ingest wiring + Crossref/arXiv abstract recovery already
+landed this session.
 
 ---