codex-py/docs/audit/DATA-QUALITY-2026-06-15.md

DATA-QUALITY AUDIT — codex knowledge base (handoff for a fresh session)

Status: baseline scan done 2026-06-15. DQ-1 RESOLVED 2026-06-16 (S2 supplement applied to live DB + wired into ingest). DQ-4 MEASURED 2026-06-16 — retrieval is strong after fixing a P0 codex search paper crash. DQ-2 RESOLVED 2026-06-16 (1911.00966 fully recovered from arXiv; s00454 abstract from S2; book chapter is a documented limit; recovery wired into ingest). DQ-3 still open. New: DQ-5 (ingest_paper not idempotent for DOI papers — blocks R-A/R-C). Roadmap R-A..R-E added 2026-06-16 (free-source acquisition levers — see Roadmap section). Author: Audit-Loop (Opus). Intended reader: a cold session with no memory of the audit conversation — everything needed to continue is in this file.

---

Why this exists — the second audit axis

The code/pipeline audit (AUDIT-2026-06-15-loop-1.md, -full-repo.md) verified the loader is correct: IDs are canonical, the citation graph is consistent, the ingest pipeline does what it claims. It did not ask whether the information in the database is actually good. A correct loader can faithfully load thin, incomplete, or unrepresentative data. This document is that second axis — a data-quality assessment of the live corpus, independent of code correctness.

---

How to reach the data (self-contained)

• Live DB: Jetson PostgreSQL via an SSH tunnel (open it first):

  ssh -f -N -L 5433:localhost:5432 alfred@192.168.178.103
  

• Credentials: .env.jetson-ingest (gitignored) holds DATABASE_URL (postgresql://researcher:…@localhost:5433/papers). The researcher role is DML-only (read freely; no DDL).
• Run probes: set -a; source .env.jetson-ingest; set +a then PYTHONPATH=. .venv/bin/python with psycopg + psycopg.rows.dict_row. Reusable domain helpers: codex.quality (is_quality_chunk, _bib_score, classify_section), codex.graph, codex.discover.
• Corpus snapshot (2026-06-15, post canonical-id migration): 29 papers, 1507 chunks, 590 citations.

---

Baseline — what is already GOOD (measured, no action needed)

• Chunk content is clean. Re-running the F-16 quality gate over all stored chunks: 0 / 1507 fail is_quality_chunk (none too-short, none low-alpha / OCR-ish, none bibliography-like). The gate did its job on the .txt ingest.
• Chunk sizing healthy. chars: min 478 / median 2913 / max 4154; 0 exact duplicate chunk bodies.
• section column is low-signal (98 % body; 13 bibliography, 9 theorem, 2 intro, 1 proof). This confirms code-audit R-12 — word-window chunks rarely start at a section header. Not a data defect; the column is just weak.

---

Findings (open investigation items)

DQ-1 — Citation coverage: 12 / 29 papers (41 %) had ZERO citations · RESOLVED 2026-06-16

Resolution (what was done):

• Verified upstream, not a bug. Queried the OpenAlex API directly for all 11 zero-citation papers that have an openalex_id: every one returns referenced_works_count = 0 server-side, W-ids all match what we stored. They are type: preprint (arXiv) / article / dissertation — works OpenAlex indexes without a parsed bibliography. Confirmed source-coverage limit.
• Semantic Scholar supplement applied to the live DB (idempotent backfill, ON CONFLICT (citing_id,cited_id) DO NOTHING): +330 citations (590 → 920). Zero-out-edge papers 12 → 2; in-corpus citing coverage 17/29 → 27/29; graph 489→704 nodes, 590→920 edges; discovery-lead targets 472→677. 9 of the new edges are in-corpus (e.g. 2305.10988 → 10.1007/s00454-019-00132-8).
• Wired into ingest so future re-ingests self-heal: when OpenAlex returns an empty referenced_works, ingest.py now falls back to _s2_reference_supplement() (citing_id rewritten to canonical papers.id, DOI cited-ids lowercased). Same helper drove the one-off backfill.
• Bug fixed (would have crashed the batch): semanticscholar.fetch_references used data.get("data", []), but S2 returns HTTP 200 {"data": null} for a known paper with no parsed refs (e.g. depositonce-5415) → TypeError. Changed to data.get("data") or []. Removed a now-redundant discarded S2 probe in the OpenAlex-404 arXiv branch. New regression test added.
• Irreducible remainder (documented limit): the 2 depositonce theses (10.14279/depositonce-20357, -5415) stay at zero — neither OpenAlex nor S2 indexes references for TU-Berlin institutional DOIs. Accept as inherent.
• Files touched (uncommitted in working tree): codex/sources/semanticscholar.py, codex/ingest.py, tests/ingest/test_ingest.py. Full suite: 331 passed.

Original finding (for context):

• Measured: 12 papers contribute no out-edges; the 590-edge citation graph comes from only ~17 papers. The whole F-15 layer (PageRank / coupling / co-citation / discovery leads) therefore runs on ~59 % of the corpus.
• Sharpening — it is mostly a source-coverage limit, not an ingest bug: 11 of the 12 zero-citation papers do have an openalex_id, i.e. ingest called openalex.fetch_citations(openalex_id) and OpenAlex returned an empty referenced_works list. Only 1911.00966 has no openalex_id (OpenAlex 404 → arXiv/S2 fallback, which also yielded nothing).
• Zero-citation ids: 1005.2698, 1505.01341, 1911.00966 (no oa), 2206.13461, 2305.10988, 2310.17529, 2601.22903, math/0001176, math/0503219, math/0603097, 10.14279/depositonce-20357, 10.14279/depositonce-5415. (Pattern: arXiv preprints + the two depositonce theses — works OpenAlex indexes without parsed references.)
• To investigate next:
  1. Confirm it is OpenAlex coverage, not a fetch_citations bug: for 2–3 of the ids, hit GET https://api.openalex.org/works/<openalex_id> and check whether referenced_works is genuinely empty server-side.
  2. If genuinely empty: enrich via the Semantic Scholar references path (semanticscholar.fetch_references("arXiv:<id>")) as a supplement for OpenAlex-empty papers — S2 often has references where OpenAlex doesn't. Note S2 cited_ids are bare DOI/arXiv, so they flow through the existing RESOLVED_CITATIONS_SQL resolver (audit C-1) fine.
  3. Decide whether the F-15 graph_min_corpus_size warning should also flag low citing-paper coverage, not just paper count.
• Acceptance: either (a) citation coverage materially improves after an S2 supplement, or (b) documented as an inherent OpenAlex-coverage limit with the graph caveated accordingly.

DQ-2 — Metadata gaps: 3 no-abstract, 1 fully metadata-less · RESOLVED 2026-06-16

Resolution (what was done):

• 1911.00966 fully recovered from the arXiv Atom API (OpenAlex 404'd on it): title "A discrete version of Liouville's theorem on conformal maps", Pinkall & Springborn, 2019, 384-char abstract → bibkey PinkallSpringborn2019 auto-generated, abstract embedded (non-zero). It was the worst node in the corpus — now @cite-able, in wiki grounding, and its 10 chunks are visible to chunk search (previously filtered out by bibkey IS NOT NULL). Citations were already added in DQ-1 (27 S2 refs).
• 10.1007/s00454-019-00132-8 abstract recovered from S2 (1186 chars) and re-embedded (targeted UPDATE, not full re-ingest — see DQ-5).
• 10.1007/978-3-642-17413-1_7 (book chapter): documented inherent limit. No abstract in OpenAlex, S2, or Crossref (verified). Left as the single remaining zero-vector paper; user may add an abstract by hand.
• Corpus now: 1 paper without abstract (the book chapter), down from 3+1. Zero-vector search pollution (DQ-4 secondary finding) reduced from 2 tail fillers to 1.
• Wired into ingest so this self-heals: OpenAlex-404 on an arXiv id now recovers metadata via arxiv.fetch_metadata (new); an empty abstract is supplemented via _recover_abstract → S2 then Crossref (new codex/sources/crossref.py). New semanticscholar.fetch_abstract.
• Files touched: codex/sources/arxiv.py, codex/sources/crossref.py (new), codex/sources/semanticscholar.py, codex/ingest.py, + tests. Full suite: 342.

Original finding (for context):

• Measured:
  • No abstract (3): 10.1007/978-3-642-17413-1_7, 10.1007/s00454-019-00132-8, 1911.00966. These get a zero-vector abstract_emb, so paper-level semantic search can't place them.
  • No bibkey / year (1): 1911.00966 only — and its authors array is empty too. This paper is fully degraded (OpenAlex 404 → Paper(id, title="") fallback): no abstract, no bibkey, no year, empty authors, no citations. With no bibkey it cannot be @cited and is invisible to wiki grounding (which keys on bibkey).
• To investigate next:
  1. 1911.00966 — confirm the arXiv id is correct and whether OpenAlex/S2 has it under a different id (DOI?); if recoverable, re-ingest to populate metadata. If not, decide: keep as a degraded node or drop.
  2. The 2 no-abstract DOIs — check if OpenAlex has an abstract_inverted_index that the mapper missed, or if the abstract is genuinely absent upstream.
• Acceptance: every paper has at least a bibkey + non-zero abstract embedding, or the exceptions are documented with rationale.

DQ-3 — Content fidelity (chunks vs source .txt): NOT YET VERIFIED · MED

• Open question: does the stored chunk set faithfully reconstruct each source file, or did the chunker / filter_chunks silently drop material (e.g. an abstract, a section, math-heavy passages)? The whole corpus was ingested from .txt (PAPERS_DIR=/Users/tarikmoussa/Desktop/ConformalLabpp/papers/txt), so the .tex/.pdf paths never ran.
• To investigate next: for a sample of ~5 papers, compare len("".join(stored chunks)) against len(source.txt) (coverage %), and eyeball the first/last chunk vs the file head/tail. Flag papers where coverage is low (content lost) — F-16 dropping >X % of a paper is a signal.
• Acceptance: sampled papers retain ≳ the expected fraction of source text; no paper is silently gutted by the quality gate.

DQ-4 — Retrieval quality · MEASURED 2026-06-16 — good, after fixing a P0 crash

P0 bug found + fixed: codex search paper was crash-broken on the live DB. The CLI passed the query embedding uncast, so pgvector raised operator does not exist: vector <-> double precision[] on every paper search. The working call sites (mcp_server.py, wiki.py) cast %(emb)s::vector; the CLI did not. Fixed both <-> occurrences in codex/cli.py search_paper; added a regression guard in tests/cli/test_cli.py asserting the cast (mocked-DB unit tests can't catch the type error — this is why it survived the code audit).

Relevance verdict (post-fix): strong. Six domain queries, both surfaces (paper-level abstract search + chunk-level). rank-1 was the exactly-correct paper in 6/6 queries for chunk search and 5/6 for paper-level (Laplace–Beltrami put the exact paper at rank-2 behind the closely-related "Polygon Laplacian Made Simple" — acceptable). Examples: "combinatorial Yamabe flow" → math/0306167 (exact, all 5 top chunks that paper); "variational principle for Delaunay triangulations" → math/0603097 (exact); "circle packing rigidity" → 2601.22903 (exact). Three of the DQ-1-rescued papers (2305.10988, 1005.2698, depositonce-5415) now surface as top hits. The KB is trustworthy for lookups.

Secondary finding → reinforces DQ-2: zero-vector abstract pollution. Papers with no abstract get a zero abstract_emb, which sits at constant L2 distance ≈1.000 from every query and appears as filler in the paper-level tail whenever < 5 strongly-relevant papers exist. 1911.00966 (fully degraded, DQ-2) showed up at distance 1.000 in 4/6 queries. Harmless at rank>1 with a visible 1.000 score, but noise — and a concrete reason to fix DQ-2. Chunk-level search is immune (it filters bibkey IS NOT NULL, and 1911.00966 has no bibkey/chunks).

Note: the MCP search docstring claims "hybrid dense + FTS" but the SQL is dense-only; score = 1.0 - dist can go negative (bge-m3 L2 ranges 0–2). Ranking is fine (monotonic); only the absolute score label is misleading.

Original open question (for context):

• Open question: end-to-end, do real queries return relevant results? Clean chunks + a working index don't guarantee useful retrieval.
• To investigate next: run a handful of domain queries through the actual search path (codex search paper "<q>" and the chunk-level MCP search), e.g. "discrete conformal map", "circle packing rigidity", "combinatorial Yamabe flow", "discrete Laplace-Beltrami operator", and judge whether the top-5 hits are on-topic and point at the right papers. Cross-check a couple against --cite-boost to see if the boost helps or hurts on this corpus.
• Acceptance: a short relevance table (query → top-5 → on/off-topic) good enough to trust the KB for lookups, or a list of failure modes to fix.

DQ-5 — ingest_paper is NOT idempotent for DOI papers · HIGH (blocks R-A/R-C) · found 2026-06-16

• Symptom: re-ingesting an existing DOI paper crashes with UniqueViolation: papers_openalex_id_key. Surfaced trying to re-ingest 10.1007/s00454-019-00132-8 during the DQ-2 backfill.
• Root cause: openalex._map_paper sets Paper.id = data["doi"], which is the full URL https://doi.org/10.1007/…. The stored canonical id is the bare 10.1007/… (post the M-1 canonical-id migration). So INSERT … ON CONFLICT (id) finds no id match, attempts a fresh INSERT, and trips the separate unique constraint on openalex_id (already held by the canonical-id row).
• Confirmed: openalex.fetch_paper("10.1007/s00454-019-00132-8").id == "https://doi.org/10.1007/s00454-019-00132-8" (≠ stored bare id).
• Impact: any re-ingest of a DOI paper aborts. This blocks roadmap R-A (GROBID on PDFs) and R-C (.tex ingest) — both re-ingest existing papers. Also implies new DOI ingests store full-URL ids, inconsistent with the bare-id corpus (latent split-identity risk).
• Proposed fix (handle with care — M-1 incident territory): normalize the DOI in _map_paper to the canonical bare, lower-cased form (strip https://doi.org/), matching what the M-1 migration produced. Add a re-ingest idempotency test over a DOI paper. Did not fix inline this session — the canonical-id derivation is exactly where the M-1 migration incident occurred, so it needs its own focused change + a live re-ingest check.
• Workaround used for DQ-2: targeted UPDATE … SET abstract, abstract_emb instead of full re-ingest.

---

Roadmap — data-acquisition levers (post-audit, all free sources)

These are forward-looking acquisition improvements, distinct from the DQ-1..DQ-4 audit findings (which assessed existing data). They came out of a strategy discussion on 2026-06-16: for this Math/CS corpus the valuable data (references, abstracts, full text) is almost entirely open (arXiv, OpenAlex, Crossref, S2), so the quality lever is combining more free sources, not buying paywall access. Paywall/uni-login was explicitly considered and deferred (see R-E). Each item is self-contained so a cold session can pick it up.

R-A — Run GROBID reference extraction on arXiv PDFs · HIGH lever, LOW-MED effort

• ⚠ Blocked by DQ-5: re-ingesting existing DOI papers currently crashes (papers_openalex_id_key). Fix DQ-5 first, or the GROBID re-ingest aborts.
• What: the whole corpus was ingested from .txt (PAPERS_DIR=…/papers/txt), so the GROBID PDF path never ran. Re-ingest with a PDF source_path per paper so codex.parsing.grobid.extract_references parses each bibliography.
• Why: extracts references the APIs lack — a complement to DQ-1's S2 supplement. Crucially, the 2 depositonce theses that stayed at zero in DQ-1 (no API references anywhere) almost certainly have a references section in their open-access PDFs → GROBID could finally give them out-edges. Closes the last 2/29.
• How: no new code — ingest already merges + dedups api_citations + pdf_citations (codex/ingest.py:288). GROBID server is already configured (GROBID_URL=http://192.168.178.103:8070 in .env.jetson-ingest). Download free arXiv PDFs, re-ingest with source_path.
• Acceptance: the 2 theses gain references; published papers gain GROBID-parsed refs beyond OpenAlex/S2 coverage; citing coverage 27/29 → 29/29.

R-B — Add Crossref as a third citation + abstract source · MED lever, MED effort

• What: new codex.sources.crossref client hitting Crossref REST /works/{doi} for the reference array and abstract.
• Why: Crossref carries publisher-deposited references for many DOIs (free, polite pool via mailto) — a third leg after OpenAlex/S2. It also recovers abstracts OpenAlex is contractually barred from redistributing (likely the 2 Springer DOIs missing abstracts in DQ-2). cited_ids are DOIs → flow through the existing graph resolver unchanged.
• How: mirror codex/sources/openalex.py (tenacity retry, polite pool). Extend the ingest fallback chain to OpenAlex-empty → S2 → Crossref (same _s2_reference_supplement shape). Add unit tests with mocked HTTP.
• Acceptance: Crossref recovers refs and/or an abstract for ≥1 paper where OpenAlex+S2 are empty.

R-C — Prefer arXiv .tex source over .txt for chunk fidelity · MED lever, MED-HIGH effort

• What: re-ingest from arXiv LaTeX source through the existing codex.parsing.tex.latex_to_text path instead of flattened .txt.
• Why: highest-fidelity input — preserves math, structure, and section boundaries. Directly serves DQ-3 (content fidelity) and audit R-12 (the section column is low-signal because word-window .txt chunks rarely start at a real header; .tex headers fix this).
• How: download free arXiv source tarballs, feed .tex to ingest with source_path. Handle multi-file projects (main .tex + includes).
• Acceptance: section column gains signal (fewer body-only); DQ-3 coverage vs source improves; no regression in the F-16 quality gate.

R-D — F-15: warn on low citing-paper coverage, not just paper count · SMALL

• What: the open DQ-1 sub-item: graph_min_corpus_size only flags total paper count. Add a warning when the share of papers with ≥1 out-edge is low (e.g. < 80%), since that is what actually starves PageRank/coupling.
• Why: at 17/29 the graph silently ran on 59% of the corpus with no signal. Post-DQ-1 it's 27/29 (93%) — but the guard should catch future regressions.
• Acceptance: codex graph report surfaces citing-coverage % and warns below threshold.

R-E — Paywall / institutional access · DEFERRED — only for a paywall-only expansion

• Decision (2026-06-16): a uni login does not help the current corpus — every DQ-1/DQ-2 gap is in openly-available works (arXiv preprints, OA theses), and all metadata/citation sources are free APIs. Revisit only if the corpus expands to papers that exist solely behind a paywall with no preprint; then institutional access → full-text PDF → GROBID refs (R-A) + clean chunks (R-C).
• ToS caveat: manual download of individual papers you have legitimate access to is fine; automated bulk download through a uni proxy (EZproxy/Shibboleth) violates most publisher terms and can get the whole institution's access revoked. Do not wire a publisher login into the ingest pipeline.

---

Priority for the next session

1. DQ-1 — DONE 2026-06-16 (live DB at 920 citations, 27/29 citing coverage; remedy wired into ingest).
2. DQ-4 — DONE 2026-06-16 (P0 search paper crash fixed; relevance verified strong; surfaced the zero-vector pollution that motivates DQ-2).
3. DQ-2 — DONE 2026-06-16 (1911.00966 fully recovered; s00454 abstract from S2; book chapter documented; recovery wired into ingest).
4. DQ-5 (NEXT — ingest_paper not idempotent for DOI papers; blocks R-A/R-C; careful — M-1 canonical-id territory), then DQ-3 (chunk-vs-source fidelity).
5. Roadmap levers (see section above): R-A (GROBID on arXiv PDFs — closes the last 2 zero-citation theses; needs DQ-5 first), R-B (Crossref refs — abstract half already shipped in DQ-2), R-C (.tex ingest — serves DQ-3; needs DQ-5 first), R-D (F-15 coverage warning, quick). R-E (paywall) deferred.

DQ-1..DQ-4 are data/assessment work; DQ-5 + roadmap R-A..R-D add new code (canonical-id fix, re-ingest scripts, Crossref references, a .tex path, a graph warning) on top of the already-remediated pipeline (see the AUDIT-* docs and PRs #12–#14). The DQ-1/DQ-2 ingest wiring + Crossref/arXiv abstract recovery already landed this session.

---

Appendix — reproduce the baseline scan

import os, psycopg, statistics
from collections import Counter, defaultdict
from psycopg.rows import dict_row
from codex.config import Settings
from codex.quality import is_quality_chunk, _bib_score

s = Settings()
with psycopg.connect(os.environ["DATABASE_URL"], row_factory=dict_row) as c:
    papers = c.execute("SELECT id, bibkey, title, abstract, year, authors, openalex_id FROM papers").fetchall()
    chunks = c.execute("SELECT paper_id, content, section FROM chunks").fetchall()
    cit = c.execute("SELECT citing_id, count(*) n FROM citations GROUP BY citing_id").fetchall()

# zero-citation papers (DQ-1)
cited = {r["citing_id"] for r in cit}
zero = [p["id"] for p in papers if p["id"] not in cited]
# metadata gaps (DQ-2): p["abstract"] empty / p["bibkey"] None / not p["authors"]
# F-16 re-gate (baseline): [ch for ch in chunks if not is_quality_chunk(ch["content"], settings=s)]
# section dist (R-12): Counter(ch["section"] or "(null)" for ch in chunks)