user2595/codex-py
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: audit remediation Wave 2 — identity normalization C-7, C-10 (+ T-3, ADR D-1) #13

Merged
user2595 merged 5 commits from fix/audit-wave-2 into main 2026-06-16 04:53:41 +00:00
Conversation 0 Commits 5 Files Changed 5 +238 -7
user2595 commented 2026-06-15 10:14:07 +00:00
Owner
Copy Link

Wave 2 of audit remediation — identity normalization

The un-normalized-identity root behind the audit (C-7) and its riders. Stacked on
the Wave 1 branch (fix/audit-wave-1); the diff is the three commits below.
Requires a one-time corpus re-ingest — see the migration note.

Fixes

  • C-7 + T-3 (HIGH/MED) fix(ingest) — ingest stored OpenAlex's URL-form
    doi/id as papers.id (live DB confirmed 35/36 rows URL-form), breaking
    bare-id lookups (graph related / discover citing / search-by-id) and
    cross-citation matching. ingest now keys papers.id on the caller-supplied
    id    ; OpenAlex's work id stays as openalex_id. T-3: the OpenAlex test fixture
    used a bare doi (unlike the real API) and never asserted paper.id, which
    masked this — fixture made representative + assertion pinned.
  • C-10 (MED) fix(ingest) — formulas/figures were keyed on the PDF filename
    stem, violating the FK to papers(id) on rich (.pdf) ingest; now keyed on
    papers.id. The rich-ingest test is strengthened into a real regression
    (stem ≠ id).
  • D-1 docs(adr) — ADR-F15 audit addendum: the "ID-format mismatch"
    limitation is resolved in code (C-1 resolver + C-7); the spike numbers predate
    it and are flagged for re-measurement after re-ingest.

Migration required (one-time, destructive)

papers.id changes from URL to bare form. Existing rows are not auto-replaced
(different PK), so the corpus must be wiped and rebuilt:
TRUNCATE papers CASCADE; then bash ingest_all.sh. A helper script accompanies
this work. After migration, C-7 is verified live (ids bare) and the ADR-F15 spike
is re-measured.

Verification

  • Full suite: 333 passing; ruff + mypy clean. C-7 / C-10 covered by
    regression tests. Live end-to-end verification follows the re-ingest.

🤖 Generated with Claude Code

## Wave 2 of audit remediation — identity normalization The un-normalized-identity root behind the audit (C-7) and its riders. Stacked on the Wave 1 branch (`fix/audit-wave-1`); the diff is the three commits below. **Requires a one-time corpus re-ingest** — see the migration note. ### Fixes - **C-7 + T-3** (HIGH/MED) `fix(ingest)` — ingest stored OpenAlex's URL-form doi/id as `papers.id` (live DB confirmed **35/36** rows URL-form), breaking bare-id lookups (`graph related` / `discover citing` / search-by-id) and cross-citation matching. ingest now keys `papers.id` on the **caller-supplied id**; OpenAlex's work id stays as `openalex_id`. T-3: the OpenAlex test fixture used a bare `doi` (unlike the real API) and never asserted `paper.id`, which masked this — fixture made representative + assertion pinned. - **C-10** (MED) `fix(ingest)` — formulas/figures were keyed on the PDF filename stem, violating the FK to `papers(id)` on rich (`.pdf`) ingest; now keyed on `papers.id`. The rich-ingest test is strengthened into a real regression (stem ≠ id). - **D-1** `docs(adr)` — ADR-F15 audit addendum: the "ID-format mismatch" limitation is resolved in code (C-1 resolver + C-7); the spike numbers predate it and are flagged for re-measurement after re-ingest. ### Migration required (one-time, destructive) `papers.id` changes from URL to bare form. Existing rows are **not** auto-replaced (different PK), so the corpus must be wiped and rebuilt: `TRUNCATE papers CASCADE;` then `bash ingest_all.sh`. A helper script accompanies this work. After migration, C-7 is verified live (ids bare) and the ADR-F15 spike is re-measured. ### Verification - Full suite: **333 passing**; `ruff` + `mypy` clean. C-7 / C-10 covered by regression tests. Live end-to-end verification follows the re-ingest. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
user2595 changed target branch from fix/audit-wave-1 to main 2026-06-16 04:53:29 +00:00
user2595 added 5 commits 2026-06-16 04:53:29 +00:00
fix(ingest): canonicalise papers.id to the caller id (audit C-7, T-3) c36e7c6ee7 
openalex.fetch_paper fills paper.id from OpenAlex's URL-form doi/id
(e.g. https://doi.org/10.48550/arxiv.math/0603097), and ingest upserted that as
papers.id. Live DB confirmed 35/36 ids were URL-form — breaking bare-id lookups
(graph related / discover citing / search by id) and leaving cited_id matching
to rely solely on openalex_id.

ingest now sets paper.id to the caller-supplied id (the arXiv id / DOI the CLI
and ingest scripts use); OpenAlex's work id is retained as openalex_id and in
paper_identifiers. Regression test: fetch_paper returns a URL-form id, papers.id
upsert uses the bare caller id.

T-3: the OpenAlex test fixture used a bare 'doi' (not the URL form the API
emits) and never asserted paper.id, which masked this. Fixture now uses the URL
form and the test pins fetch_paper's mapping.

NOTE: existing rows keep their URL-form ids until re-ingested — run ingest_all.sh
to migrate the 36-paper corpus (chosen migration path).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix(ingest): key formulas/figures on papers.id, not filename stem (audit C-10) 8e2f9e037d 
extract_formulas/extract_figures derive paper_id from Path(pdf_path).stem, which
need not equal papers.id (and won't, given canonical ids). Inserting that stem
violated the formulas/figures -> papers(id) FK on rich (.pdf) ingest. The DELETE
already used paper.id; the INSERTs now do too.

Strengthened the rich-ingest test: the fake formula/figure carry a filename-stem
paper_id distinct from papers.id, and the test asserts the inserted rows are
keyed on papers.id.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
docs(adr): audit addendum to ADR-F15 — ID mismatch resolved (audit D-1) ce75bc02e5 
The ADR's GO rested on a live-corpus spike (478 dangling, Bobenko/Springborn
rank 7, 5.7% spread) measured before the resolution + canonicalisation landed.
Add an audit addendum: mark the 'Known Limitation: ID-Format Mismatch' as
resolved (C-1 shared resolver + C-7 canonical ids), and flag the Spike Result
numbers as pending re-measurement after the C-7 re-ingest. History preserved.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
chore(infra): add canonical-id re-ingest migration helper (audit C-7) 1ff8052b69 
One-time, guarded migration for the C-7 id-format change: a plain re-ingest
would duplicate every paper (new bare-id row beside the old URL-id PK), so this
wipes (TRUNCATE papers CASCADE) and rebuilds via ingest_all.sh.

Safety: requires the SSH tunnel, prints a BEFORE snapshot, gates the TRUNCATE
behind an explicit 'MIGRATE' confirmation, then prints AFTER verification —
C-7 (url_form_ids should be 0) and C-1 via the real resolver-based
discovery_leads() (ingested papers leaked should be 0). Uses .venv psycopg
(psql is not installed); DATABASE_URL is sourced, never echoed.

Joins PR #13 (Wave 2). Read-only verification SQL validated against the live DB.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix(infra): preflight schema check before TRUNCATE in migration helper 9f172f5c68 
The helper TRUNCATEd papers CASCADE and only then ran ingest_all.sh, which
failed on the missing chunks.section column (audit M-1) — leaving the corpus
wiped. Add a preflight that verifies chunks.section exists BEFORE the
destructive step and aborts early (no TRUNCATE) with the owner-level ALTER to
run, since the app user cannot alter postgres-owned tables.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
user2595 merged commit 07929bd4fe into main 2026-06-16 04:53:41 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
user2595
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: user2595/codex-py#13
No description provided.
Delete Branch "fix/audit-wave-2"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?