merge: reconcile data-quality roadmap with audit-remediation main (#12-15)

Resolved 4 conflicts: cli.py (R-D citing-coverage warning + main's small-corpus JSON/hub-title changes both kept); ingest.py (C-7 paper.id=caller-id pin first, then DQ-2 abstract recovery); test_ingest.py + test_openalex.py (kept both branches' tests).

C-7/DQ-5 overlap (both canonicalise papers.id): kept both layers — openalex._canonical_id normalizes fetch_paper's id (DQ-5); ingest pins papers.id to the caller's bare id (C-7); they converge on the bare canonical id. Fixed the auto-merged test_fetch_paper_success that had contradictory bare/URL assertions. 402 tests pass; ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-22 15:11:26 +02:00
29 changed files with 900 additions and 193 deletions

View File

@@ -1,8 +1,10 @@
# ADR-F15 — Literature Graph / Citation PageRank
**Status:** IMPL (GO after live-corpus spike 2026-06-15)
**Status:** IMPL GO after live-corpus spike 2026-06-15. See **Audit Update
(2026-06-15)** at the end: the ID-format mismatch is now resolved in code
(audit C-1, C-7) and the Spike Result numbers below predate that resolution.
**Owners:** F-15 Worker (Sonnet)
**Last updated:** 2026-06-15
**Last updated:** 2026-06-15 (audit addendum)
---
@@ -138,6 +140,12 @@ If the `citations` table is empty: graph has 0 nodes; `graph report` prints
## Known Limitation: ID-Format Mismatch
> **RESOLVED (audit 2026-06-15) — see Audit Update below.** This no longer holds:
> `build_citation_graph` resolves OpenAlex `cited_id`s to the canonical
> `papers.id` via the shared `RESOLVED_CITATIONS_SQL` (C-1) and ingest stores
> canonical ids (C-7), so inter-KB citations *are* now edges. Original text kept
> for the record.
`cited_id` uses OpenAlex IDs; `papers.id` uses DOIs. Cross-citations
between ingested papers are therefore not represented as graph edges.
This is a D-05-class issue (same root cause as the `citing_id`/`openalex_id`
@@ -155,3 +163,42 @@ small while the corpus is < 100 papers.
- R-44: `codex graph report [--top-n N] [--json]`
- R-45: graceful degradation < 5 nodes: uniform scores, warning
- R-46: **this document** GO; Bobenko+Springborn 2007 at rank 7 validates hub detection; score flatness expected at corpus size 29; increase to > 100 papers for stronger PageRank differentiation
---
## Audit Update (2026-06-15)
A post-hoc audit of the shipped F-15 code found that the live-corpus spike above
was measured on a graph the current code no longer produces:
- **ID-format mismatch resolved.** `build_citation_graph` resolves OpenAlex
`cited_id`s to the canonical `papers.id` through the shared
`RESOLVED_CITATIONS_SQL`, now also used by `codex.discover` so the two cannot
diverge (audit C-1). Ingest stores the caller's canonical id as `papers.id`
instead of OpenAlex's URL-form doi (audit C-7). Inter-KB citations are
therefore represented as edges — the "Known Limitation" above is obsolete.
- **Spike re-measured after migration (2026-06-15).** The corpus was re-ingested
onto canonical bare ids (29 papers, all bare — verified `url_form_ids = 0`), and
the graph re-run on the post-migration corpus:
| Metric | Original spike (pre-fix) | Post-migration (re-measured) |
|--------|--------------------------|------------------------------|
| Ingested papers | 29 | 29 |
| Graph nodes | 495 | 489 |
| Graph edges | 590 | 590 |
| Dangling nodes | 478 | 472 |
| Top-10 PR spread | 5.7 % | 5.4 % |
| Rank-1 hub | W1974956622 (dangling) | **Pinkall/Polthier 1993 — in-KB (resolved)** |
| Bobenko/Springborn 2007 | rank 7 | rank 7 |
The resolution fix's signature is visible: Pinkall/Polthier 1993 now resolves to
its canonical DOI and rises to **rank 1 as an in-KB hub** (previously a dangling
OpenAlex node). Bobenko/Springborn 2007 holds rank 7, and the foundational works
(Pinkall/Polthier, Yamabe, Schoen, Bobenko/Springborn) still surface. **GO
re-affirmed** on the shipping topology.
Migration footnotes: two papers initially survived in URL form because the
id-keyed upsert did not catch a `papers.openalex_id` collision, and `2305.10988`
failed on a `papers.bibkey` collision with `2310.17529` (both →
`BobenkoLutz2023`). Same upsert gap — resolved by the bibkey-retry (audit C-15)
plus a delete + re-ingest of the two stragglers; corpus is now 29/29 canonical.