openalex._map_paper set Paper.id from the raw `doi` field, which OpenAlex
returns as a full URL (https://doi.org/10.x). The canonical papers.id (M-1
migration) is the bare, lower-cased DOI, so re-ingesting a DOI paper missed
ON CONFLICT (id) and tripped the separate papers_openalex_id_key constraint
(UniqueViolation).
Fix: _normalize_doi() strips https://doi.org//http://doi.org//doi: and
lower-cases; applied in the DOI branch of _map_paper only (W-id fallback
untouched). Also un-breaks _s2_id_for/Crossref recovery, which the URL form
silently defeated. Corrected the _SAMPLE_WORK fixture to the URL form (the bare
fixture hid the bug) + added _normalize_doi and re-ingest idempotency regression
tests.
Live verification: re-ingest of 10.1007/s00454-019-00132-8 (previously crashed)
now UPDATEs in place — single row, added_at unchanged, no stray row, citations
45->45. Unblocks roadmap R-A/R-C. Full suite 348 green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-ran chunk_text + is_quality_chunk on all 29 source .txt files vs stored
chunks. Coverage 109-114% everywhere (the >100% is the 64/512-word overlap, i.e.
full retention); stored==kept for every paper. Only 18 chunks dropped corpus-wide,
all low_alpha = garbled OCR tables in 2 scanned-book sources (0-387-29555-0_13,
depositonce-5415) — no real math content lost. Head/tail aligned (no truncation).
Closes the data-quality assessment axis: DQ-1 (citations), DQ-2 (metadata),
DQ-3 (fidelity), DQ-4 (retrieval) all resolved. DQ-5 (re-ingest idempotency)
remains, tracked separately.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three papers lacked abstracts and 1911.00966 was fully degraded (OpenAlex 404
-> empty `Paper(id, title="")` stub). Recovery sources verified, then wired in:
- arxiv.fetch_metadata: resolve an arXiv id to a Paper via the Atom export API
(authoritative for preprints). Used as an ingest fallback on OpenAlex 404,
replacing the empty stub.
- crossref.py (new): fetch_abstract(doi), JATS-stripped, Crossref Polite Pool.
- semanticscholar.fetch_abstract: fetch just the abstract field.
- ingest.py: _recover_abstract() supplements an empty abstract from S2 then
Crossref *before* embedding, so the paper gets a real (non-zero) vector.
Live DB backfill: 1911.00966 fully recovered from arXiv (bibkey
PinkallSpringborn2019, 384-char abstract, its 10 chunks now visible to chunk
search); 10.1007/s00454-019-00132-8 abstract from S2 (1186 chars). Book chapter
10.1007/978-3-642-17413-1_7 has no abstract in OpenAlex/S2/Crossref -> documented
limit. Corpus now has 1 paper without an abstract.
Also documented DQ-5 (not fixed): ingest_paper is not idempotent for DOI papers
(openalex returns full-URL id vs the bare canonical id) -> re-ingest trips
papers_openalex_id_key. Blocks roadmap R-A/R-C; needs a careful _map_paper fix.
Tests: arxiv/crossref/s2 fetchers + ingest recovery paths (342 passing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
12/29 papers had zero citations: OpenAlex indexes arXiv preprints and theses
without a parsed reference list (verified referenced_works_count=0 server-side
for all 11 with an openalex_id). Not an ingest bug — a source-coverage limit.
- ingest.py: when OpenAlex returns no references, fall back to a Semantic
Scholar reference supplement (_s2_reference_supplement); citing_id rewritten
to canonical papers.id, DOI cited-ids lowercased. Removed a now-redundant
discarded S2 probe in the OpenAlex-404 arXiv branch.
- semanticscholar.py: fix TypeError on S2's HTTP-200 {"data": null} no-refs
responses (.get("data", []) returns None when the key is present-but-null).
- tests: regression test for the OpenAlex-empty -> S2 path.
- Live DB backfilled idempotently: +330 citations (590->920), zero-out-edge
papers 12->2, citing coverage 17->27/29. Only the 2 TU-Berlin depositonce
theses remain (no references in any source).
- docs: DATA-QUALITY-2026-06-15.md — DQ-1 resolution, DQ-4 result, and a
free-source acquisition roadmap (R-A..R-E).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Second audit axis (data, not code): is the information in the DB actually good?
Baseline scan of the live 29-paper corpus + four open investigation items
prepared self-contained for a cold session.
- Good: chunks 0/1507 fail the F-16 re-gate, healthy sizing, 0 dups.
- DQ-1 (HIGH): 12/29 papers have zero citations; 11 of them have an openalex_id,
so OpenAlex returned empty referenced_works (source-coverage limit, not a bug)
— propose an S2 references supplement.
- DQ-2 (MED): 3 papers no abstract; 1911.00966 fully metadata-less (OpenAlex 404).
- DQ-3 (MED): chunk-vs-source fidelity not yet verified.
- DQ-4 (MED): retrieval relevance not yet measured.
Includes connection steps, exact ids, priorities, and a reproduce appendix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The C-7 re-ingest hit M-1 live: TRUNCATE succeeded, then ingest failed on the
missing chunks.section column (schema.sql never applied to the live DB). Adding
the column then failed with 'must be owner' — the app user 'researcher' has DML
+ TRUNCATE but not DDL on the postgres-owned core tables. M-1's fix therefore
needs a privileged migration role, not just an idempotent schema. Records the
unblock and the helper's new preflight.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Part D synthesis: C-11 HIGH (lead-ID collision → silent overwrite), R-8 MED, D-4 LOW.
- Part E wiki-rest: R-9 MED (cross-ref injection corrupts LaTeX math), C-12/C-13/R-10/R-11 LOW.
- Part F embed/quality/models: C-14/R-12 LOW + C-7 corroboration. Cleanest modules.
- Part G cli/config: C-15 MED (embedding_dim vs hardcoded vector(1024)), R-13/S-4/R-14/D-5 LOW.
Executive summary: 45 findings total (8 HIGH, 11 MED, 26 LOW). Five root themes;
dominant one is un-normalized identity (C-1/C-7/C-10/M-1). Whole repo reviewed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Closes request (a) — the second pass over loop-1 §8 open items:
- Part A grounding guard: C-4 (HIGH, zero-claim pages bypass quarantine),
C-5/R-1 MED, C-6/R-2/R-3 LOW.
- Part B sources/: C-7 (HIGH, papers.id stored in OpenAlex URL form vs the
bare-ID contract), T-3 MED, R-4/R-5/C-9 LOW.
- Part C parsing/: C-10 MED (formulas/figures key on filename not paper.id),
R-6/R-7 LOW — all latent (live corpus is .txt).
Running tally: 30 findings, 7 HIGH. Common root: un-normalized identity.
Parts D-G (synthesis/wiki/embed/cli) still open.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Audit of main @5dfa6e4: 4 HIGH (S-1 secret-in-tree, M-1 missing migration
path, C-1 unpropagated ID-mismatch fix in discover.py, D-1 stale ADR-F15
validation, T-1 untested resolution SQL) plus MED/LOW findings. Findings
only; remediation planning deferred to the next loop phase.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Closes audit N-02: docs/adr/ADR-F13-synthesis-engine.md was missing,
blocking R-34. Documents spike result (GO, Opus APPROVE 2 passes),
four-stage lead taxonomy, grounding discipline, three-level
zero-fact-leak invariant, and falsification-gate in _parse_conjecture_output.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
