fix(openalex): normalize DOI to bare canonical id so re-ingest is idempotent (DQ-5)
openalex._map_paper set Paper.id from the raw `doi` field, which OpenAlex returns as a full URL (https://doi.org/10.x). The canonical papers.id (M-1 migration) is the bare, lower-cased DOI, so re-ingesting a DOI paper missed ON CONFLICT (id) and tripped the separate papers_openalex_id_key constraint (UniqueViolation). Fix: _normalize_doi() strips https://doi.org//http://doi.org//doi: and lower-cases; applied in the DOI branch of _map_paper only (W-id fallback untouched). Also un-breaks _s2_id_for/Crossref recovery, which the URL form silently defeated. Corrected the _SAMPLE_WORK fixture to the URL form (the bare fixture hid the bug) + added _normalize_doi and re-ingest idempotency regression tests. Live verification: re-ingest of 10.1007/s00454-019-00132-8 (previously crashed) now UPDATEs in place — single row, added_at unchanged, no stray row, citations 45->45. Unblocks roadmap R-A/R-C. Full suite 348 green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
@@ -6,8 +6,9 @@ strong after fixing a P0 `codex search paper` crash. **DQ-2 RESOLVED 2026-06-16*
|
||||
(1911.00966 fully recovered from arXiv; s00454 abstract from S2; book chapter is a
|
||||
documented limit; recovery wired into ingest). **DQ-3 RESOLVED 2026-06-16**
|
||||
(content fidelity clean — full coverage, only OCR junk dropped). **All four
|
||||
assessment items (DQ-1..DQ-4) DONE. New: DQ-5** (`ingest_paper` not idempotent
|
||||
for DOI papers — blocks R-A/R-C, tracked separately).
|
||||
assessment items (DQ-1..DQ-4) DONE. DQ-5 RESOLVED 2026-06-17** (DOI id normalized
|
||||
to bare canonical form in `openalex._map_paper`; live re-ingest of
|
||||
`10.1007/s00454-019-00132-8` confirmed idempotent → **unblocks R-A/R-C**).
|
||||
**Roadmap R-A..R-E** added 2026-06-16 (free-source acquisition levers — see Roadmap section).
|
||||
**Author:** Audit-Loop (Opus). **Intended reader:** a *cold* session with no memory
|
||||
of the audit conversation — everything needed to continue is in this file.
|
||||
@@ -239,7 +240,43 @@ is fine (monotonic); only the absolute score label is misleading.
|
||||
- **Acceptance:** a short relevance table (query → top-5 → on/off-topic) good
|
||||
enough to trust the KB for lookups, or a list of failure modes to fix.
|
||||
|
||||
### DQ-5 — `ingest_paper` is NOT idempotent for DOI papers · **HIGH (blocks R-A/R-C)** · found 2026-06-16
|
||||
### DQ-5 — `ingest_paper` is NOT idempotent for DOI papers · **RESOLVED 2026-06-17** (found 2026-06-16)
|
||||
|
||||
**Resolution (what was done):**
|
||||
- **Root-cause fix, minimal + contained:** added `openalex._normalize_doi()` and
|
||||
applied it in `_map_paper` so the DOI branch yields the canonical **bare,
|
||||
lower-cased** form (strips `https://doi.org/` / `http://doi.org/` / `doi:`;
|
||||
lower-cases — DOIs are case-insensitive, matching `ingest._norm_cited_id`). The
|
||||
no-DOI fallback (OpenAlex W-id URL) is left untouched. `_map_paper`→`fetch_paper`
|
||||
→`ingest_paper` is the only consumer chain, so blast radius is one function.
|
||||
- **Checked nothing depended on the URL form.** It was the opposite: the full-URL
|
||||
id silently *defeated* `_s2_id_for(paper.id)` and `_recover_abstract`'s
|
||||
`paper.id.startswith("10.")` (a `https://…` string is neither `10.`- nor
|
||||
`doi:`-prefixed), so S2/Crossref recovery never fired for DOI papers. The fix
|
||||
repairs those paths too.
|
||||
- **Tests (offline, mocked DB):** closed audit **T-3** — the OpenAlex fixture now
|
||||
emits the real URL-form DOI and asserts `paper.id` is the bare form; added
|
||||
`_normalize_doi` unit tests (https/http/`doi:`/upper-case/already-bare) and a
|
||||
W-id-fallback test; added ingest regression `test_ingest_doi_paper_upserts_bare
|
||||
_canonical_id` (runs the real `_map_paper` through ingest, asserts the papers
|
||||
upsert carries the **bare** id into `%(id)s` and uses `ON CONFLICT (id)`).
|
||||
Verified it has teeth (fails on the pre-fix code). Full suite **348 passed**;
|
||||
ruff + mypy --strict clean.
|
||||
- **Live idempotency confirmed (tunnel up):** re-ingested
|
||||
`10.1007/s00454-019-00132-8` against the Jetson DB → no `UniqueViolation`;
|
||||
`IngestResult(paper_id='10.1007/s00454-019-00132-8', citations_upserted=45)`.
|
||||
After-state: **UPDATE in place** — `id` and `openalex_id` unchanged, **`added_at`
|
||||
unchanged** (proves no fresh INSERT), exactly **1** row for the DOI (no URL-form
|
||||
duplicate), citations 45→45, total papers 29→29.
|
||||
- **Unblocks R-A (GROBID re-ingest) and R-C (.tex re-ingest)** — both re-ingest
|
||||
existing papers. **Files touched:** `codex/sources/openalex.py`,
|
||||
`tests/sources/test_openalex.py`, `tests/ingest/test_ingest.py`.
|
||||
- **Note (out of scope, pre-existing):** arXiv papers stored as bare ids
|
||||
(`2305.10988`) are still not round-trip idempotent if OpenAlex returns an
|
||||
arXiv-DOI/W-id instead of the bare arXiv id — a separate canonicalization
|
||||
concern, not this UniqueViolation. The DOI path (this finding) is fixed.
|
||||
|
||||
**Original finding (for context):**
|
||||
- **Symptom:** re-ingesting an existing DOI paper crashes with
|
||||
`UniqueViolation: papers_openalex_id_key`. Surfaced trying to re-ingest
|
||||
`10.1007/s00454-019-00132-8` during the DQ-2 backfill.
|
||||
@@ -276,8 +313,8 @@ Paywall/uni-login was explicitly considered and **deferred** (see R-E). Each ite
|
||||
is self-contained so a cold session can pick it up.
|
||||
|
||||
### R-A — Run GROBID reference extraction on arXiv PDFs · **HIGH lever, LOW-MED effort**
|
||||
- **⚠ Blocked by DQ-5:** re-ingesting existing DOI papers currently crashes
|
||||
(`papers_openalex_id_key`). Fix DQ-5 first, or the GROBID re-ingest aborts.
|
||||
- **✓ Unblocked (DQ-5 RESOLVED 2026-06-17):** DOI re-ingest is now idempotent, so
|
||||
the GROBID re-ingest no longer aborts on `papers_openalex_id_key`.
|
||||
- **What:** the whole corpus was ingested from `.txt` (`PAPERS_DIR=…/papers/txt`),
|
||||
so the GROBID PDF path never ran. Re-ingest with a PDF `source_path` per paper
|
||||
so `codex.parsing.grobid.extract_references` parses each bibliography.
|
||||
@@ -350,13 +387,13 @@ is self-contained so a cold session can pick it up.
|
||||
from S2; book chapter documented; recovery wired into ingest).
|
||||
4. ~~**DQ-3**~~ — **DONE 2026-06-16** (content fidelity clean; full coverage, only
|
||||
18 OCR-junk chunks dropped, no math lost). **Assessment axis DQ-1..DQ-4 complete.**
|
||||
5. **DQ-5** (NEXT — `ingest_paper` not idempotent for DOI papers; **blocks R-A/R-C**;
|
||||
careful — M-1 canonical-id territory).
|
||||
6. **Roadmap levers** (see section above): **R-A** (GROBID on arXiv PDFs — closes
|
||||
the last 2 zero-citation theses; needs DQ-5 first), **R-B** (Crossref refs —
|
||||
abstract half already shipped in DQ-2), **R-C** (`.tex` ingest — serves DQ-3;
|
||||
needs DQ-5 first), **R-D** (F-15 coverage warning, quick). **R-E** (paywall)
|
||||
deferred.
|
||||
5. ~~**DQ-5**~~ — **DONE 2026-06-17** (DOI id normalized to bare form in
|
||||
`openalex._map_paper`; live re-ingest of `10.1007/s00454-019-00132-8` idempotent
|
||||
— UPDATE in place, `added_at` unchanged; suite 348 green). **Unblocks R-A/R-C.**
|
||||
6. **Roadmap levers** (see section above — now all unblocked): **R-A** (GROBID on
|
||||
arXiv PDFs — closes the last 2 zero-citation theses), **R-B** (Crossref refs —
|
||||
abstract half already shipped in DQ-2), **R-C** (`.tex` ingest — serves DQ-3),
|
||||
**R-D** (F-15 coverage warning, quick). **R-E** (paywall) deferred.
|
||||
|
||||
DQ-1..DQ-4 are *data*/assessment work; DQ-5 + roadmap R-A..R-D add **new code**
|
||||
(canonical-id fix, re-ingest scripts, Crossref references, a `.tex` path, a graph
|
||||
|
||||
Reference in New Issue
Block a user