codex-py/docs/audit/DATA-QUALITY-2026-06-15.md

# DATA-QUALITY AUDIT — codex knowledge base (handoff for a fresh session)

**Status:** baseline scan done 2026-06-15. **DQ-1 RESOLVED 2026-06-16** (S2 supplement
applied to live DB + wired into ingest). **DQ-4 MEASURED 2026-06-16** — retrieval is
strong after fixing a P0 `codex search paper` crash. **DQ-2 RESOLVED 2026-06-16**
(1911.00966 fully recovered from arXiv; s00454 abstract from S2; book chapter is a
documented limit; recovery wired into ingest). **DQ-3 still open. New: DQ-5**
(`ingest_paper` not idempotent for DOI papers — blocks R-A/R-C).
**Roadmap R-A..R-E** added 2026-06-16 (free-source acquisition levers — see Roadmap section).
**Author:** Audit-Loop (Opus). **Intended reader:** a *cold* session with no memory
of the audit conversation — everything needed to continue is in this file.

---

## Why this exists — the second audit axis

The code/pipeline audit (`AUDIT-2026-06-15-loop-1.md`, `-full-repo.md`) verified the
**loader is correct**: IDs are canonical, the citation graph is consistent, the
ingest pipeline does what it claims. It did **not** ask whether the *information
in the database is actually good*. A correct loader can faithfully load thin,
incomplete, or unrepresentative data. This document is that second axis — a
**data-quality** assessment of the live corpus, independent of code correctness.

---

## How to reach the data (self-contained)

- **Live DB:** Jetson PostgreSQL via an SSH tunnel (open it first):
  ```
  ssh -f -N -L 5433:localhost:5432 alfred@192.168.178.103
  ```
- **Credentials:** `.env.jetson-ingest` (gitignored) holds `DATABASE_URL`
  (`postgresql://researcher:…@localhost:5433/papers`). The `researcher` role is
  DML-only (read freely; no DDL).
- **Run probes:** `set -a; source .env.jetson-ingest; set +a` then
  `PYTHONPATH=. .venv/bin/python` with `psycopg` + `psycopg.rows.dict_row`.
  Reusable domain helpers: `codex.quality` (`is_quality_chunk`, `_bib_score`,
  `classify_section`), `codex.graph`, `codex.discover`.
- **Corpus snapshot (2026-06-15, post canonical-id migration):**
  **29 papers, 1507 chunks, 590 citations.**

---

## Baseline — what is already GOOD (measured, no action needed)

- **Chunk content is clean.** Re-running the F-16 quality gate over all stored
  chunks: **0 / 1507 fail** `is_quality_chunk` (none too-short, none low-alpha /
  OCR-ish, none bibliography-like). The gate did its job on the `.txt` ingest.
- **Chunk sizing healthy.** chars: min 478 / median 2913 / max 4154; **0 exact
  duplicate** chunk bodies.
- **`section` column is low-signal** (98 % `body`; 13 `bibliography`, 9 `theorem`,
  2 `intro`, 1 `proof`). This confirms code-audit **R-12** — word-window chunks
  rarely start at a section header. Not a data defect; the column is just weak.

---

## Findings (open investigation items)

### DQ-1 — Citation coverage: 12 / 29 papers (41 %) had ZERO citations · **RESOLVED 2026-06-16**

**Resolution (what was done):**
- **Verified upstream, not a bug.** Queried the OpenAlex API directly for all 11
  zero-citation papers that have an `openalex_id`: every one returns
  `referenced_works_count = 0` server-side, W-ids all match what we stored. They
  are `type: preprint` (arXiv) / `article` / `dissertation` — works OpenAlex
  indexes without a parsed bibliography. Confirmed source-coverage limit.
- **Semantic Scholar supplement applied to the live DB** (idempotent backfill,
  `ON CONFLICT (citing_id,cited_id) DO NOTHING`): **+330 citations (590 → 920)**.
  Zero-out-edge papers **12 → 2**; in-corpus citing coverage **17/29 → 27/29**;
  graph 489→704 nodes, 590→920 edges; discovery-lead targets 472→677. 9 of the
  new edges are in-corpus (e.g. `2305.10988 → 10.1007/s00454-019-00132-8`).
- **Wired into ingest** so future re-ingests self-heal: when OpenAlex returns an
  empty `referenced_works`, `ingest.py` now falls back to
  `_s2_reference_supplement()` (citing_id rewritten to canonical `papers.id`,
  DOI cited-ids lowercased). Same helper drove the one-off backfill.
- **Bug fixed (would have crashed the batch):** `semanticscholar.fetch_references`
  used `data.get("data", [])`, but S2 returns HTTP 200 `{"data": null}` for a
  known paper with no parsed refs (e.g. `depositonce-5415`) → `TypeError`.
  Changed to `data.get("data") or []`. Removed a now-redundant discarded S2 probe
  in the OpenAlex-404 arXiv branch. New regression test added.
- **Irreducible remainder (documented limit):** the **2 `depositonce` theses**
  (`10.14279/depositonce-20357`, `-5415`) stay at zero — neither OpenAlex nor S2
  indexes references for TU-Berlin institutional DOIs. Accept as inherent.
- **Files touched (uncommitted in working tree):** `codex/sources/semanticscholar.py`,
  `codex/ingest.py`, `tests/ingest/test_ingest.py`. Full suite: 331 passed.

**Original finding (for context):**
- **Measured:** 12 papers contribute **no** out-edges; the 590-edge citation graph
  comes from only ~17 papers. The whole F-15 layer (PageRank / coupling /
  co-citation / discovery leads) therefore runs on ~59 % of the corpus.
- **Sharpening — it is mostly a source-coverage limit, not an ingest bug:**
  **11 of the 12** zero-citation papers *do* have an `openalex_id`, i.e. ingest
  called `openalex.fetch_citations(openalex_id)` and OpenAlex returned an **empty
  `referenced_works`** list. Only `1911.00966` has no `openalex_id` (OpenAlex
  404 → arXiv/S2 fallback, which also yielded nothing).
- **Zero-citation ids:** `1005.2698`, `1505.01341`, `1911.00966` (no oa),
  `2206.13461`, `2305.10988`, `2310.17529`, `2601.22903`, `math/0001176`,
  `math/0503219`, `math/0603097`, `10.14279/depositonce-20357`,
  `10.14279/depositonce-5415`. (Pattern: arXiv preprints + the two `depositonce`
  theses — works OpenAlex indexes without parsed references.)
- **To investigate next:**
  1. Confirm it is OpenAlex coverage, not a `fetch_citations` bug: for 2–3 of the
     ids, hit `GET https://api.openalex.org/works/<openalex_id>` and check whether
     `referenced_works` is genuinely empty server-side.
  2. If genuinely empty: enrich via the **Semantic Scholar references** path
     (`semanticscholar.fetch_references("arXiv:<id>")`) as a *supplement* for
     OpenAlex-empty papers — S2 often has references where OpenAlex doesn't. Note
     S2 cited_ids are bare DOI/arXiv, so they flow through the existing
     `RESOLVED_CITATIONS_SQL` resolver (audit C-1) fine.
  3. Decide whether the F-15 `graph_min_corpus_size` warning should also flag *low
     citing-paper coverage*, not just paper count.
- **Acceptance:** either (a) citation coverage materially improves after an S2
  supplement, or (b) documented as an inherent OpenAlex-coverage limit with the
  graph caveated accordingly.

### DQ-2 — Metadata gaps: 3 no-abstract, 1 fully metadata-less · **RESOLVED 2026-06-16**

**Resolution (what was done):**
- **`1911.00966` fully recovered** from the arXiv Atom API (OpenAlex 404'd on it):
  title "A discrete version of Liouville's theorem on conformal maps",
  Pinkall & Springborn, 2019, 384-char abstract → bibkey `PinkallSpringborn2019`
  auto-generated, abstract embedded (non-zero). It was the worst node in the
  corpus — now `@cite`-able, in wiki grounding, and its **10 chunks are visible
  to chunk search** (previously filtered out by `bibkey IS NOT NULL`). Citations
  were already added in DQ-1 (27 S2 refs).
- **`10.1007/s00454-019-00132-8` abstract recovered from S2** (1186 chars) and
  re-embedded (targeted `UPDATE`, not full re-ingest — see DQ-5).
- **`10.1007/978-3-642-17413-1_7` (book chapter): documented inherent limit.**
  No abstract in OpenAlex, S2, **or Crossref** (verified). Left as the single
  remaining zero-vector paper; user may add an abstract by hand.
- **Corpus now: 1 paper without abstract** (the book chapter), down from 3+1.
  Zero-vector search pollution (DQ-4 secondary finding) reduced from 2 tail
  fillers to 1.
- **Wired into ingest** so this self-heals: OpenAlex-404 on an arXiv id now
  recovers metadata via `arxiv.fetch_metadata` (new); an empty abstract is
  supplemented via `_recover_abstract` → S2 then Crossref (new
  `codex/sources/crossref.py`). New `semanticscholar.fetch_abstract`.
- **Files touched:** `codex/sources/arxiv.py`, `codex/sources/crossref.py` (new),
  `codex/sources/semanticscholar.py`, `codex/ingest.py`, + tests. Full suite: 342.

**Original finding (for context):**
- **Measured:**
  - **No abstract (3):** `10.1007/978-3-642-17413-1_7`,
    `10.1007/s00454-019-00132-8`, `1911.00966`. These get a **zero-vector**
    `abstract_emb`, so paper-level semantic search can't place them.
  - **No bibkey / year (1):** `1911.00966` only — and its authors array is empty
    too. This paper is **fully degraded** (OpenAlex 404 → `Paper(id, title="")`
    fallback): no abstract, no bibkey, no year, empty authors, no citations. With
    no bibkey it cannot be `@cite`d and is invisible to wiki grounding (which keys
    on bibkey).
- **To investigate next:**
  1. `1911.00966` — confirm the arXiv id is correct and whether OpenAlex/S2 has it
     under a different id (DOI?); if recoverable, re-ingest to populate metadata.
     If not, decide: keep as a degraded node or drop.
  2. The 2 no-abstract DOIs — check if OpenAlex has an `abstract_inverted_index`
     that the mapper missed, or if the abstract is genuinely absent upstream.
- **Acceptance:** every paper has at least a bibkey + non-zero abstract embedding,
  or the exceptions are documented with rationale.

### DQ-3 — Content fidelity (chunks vs source `.txt`): NOT YET VERIFIED · **MED**
- **Open question:** does the stored chunk set faithfully reconstruct each source
  file, or did the chunker / `filter_chunks` silently drop material (e.g. an
  abstract, a section, math-heavy passages)? The whole corpus was ingested from
  `.txt` (`PAPERS_DIR=/Users/tarikmoussa/Desktop/ConformalLabpp/papers/txt`), so
  the `.tex`/`.pdf` paths never ran.
- **To investigate next:** for a sample of ~5 papers, compare
  `len("".join(stored chunks))` against `len(source.txt)` (coverage %), and
  eyeball the first/last chunk vs the file head/tail. Flag papers where coverage
  is low (content lost) — F-16 dropping >X % of a paper is a signal.
- **Acceptance:** sampled papers retain ≳ the expected fraction of source text;
  no paper is silently gutted by the quality gate.

### DQ-4 — Retrieval quality · **MEASURED 2026-06-16 — good, after fixing a P0 crash**

**P0 bug found + fixed: `codex search paper` was crash-broken on the live DB.**
The CLI passed the query embedding uncast, so pgvector raised
`operator does not exist: vector <-> double precision[]` on every paper search.
The working call sites (`mcp_server.py`, `wiki.py`) cast `%(emb)s::vector`; the
CLI did not. Fixed both `<->` occurrences in `codex/cli.py search_paper`; added a
regression guard in `tests/cli/test_cli.py` asserting the cast (mocked-DB unit
tests can't catch the type error — this is why it survived the code audit).

**Relevance verdict (post-fix): strong.** Six domain queries, both surfaces
(paper-level abstract search + chunk-level). **rank-1 was the exactly-correct
paper in 6/6 queries** for chunk search and 5/6 for paper-level (Laplace–Beltrami
put the exact paper at rank-2 behind the closely-related "Polygon Laplacian Made
Simple" — acceptable). Examples: "combinatorial Yamabe flow" → `math/0306167`
(exact, all 5 top chunks that paper); "variational principle for Delaunay
triangulations" → `math/0603097` (exact); "circle packing rigidity" →
`2601.22903` (exact). Three of the DQ-1-rescued papers (`2305.10988`,
`1005.2698`, `depositonce-5415`) now surface as top hits. The KB is trustworthy
for lookups.

**Secondary finding → reinforces DQ-2: zero-vector abstract pollution.** Papers
with no abstract get a zero `abstract_emb`, which sits at constant L2 distance
≈1.000 from every query and appears as filler in the paper-level tail whenever
< 5 strongly-relevant papers exist. `1911.00966` (fully degraded, DQ-2) showed up
at distance 1.000 in 4/6 queries. Harmless at rank>1 with a visible 1.000 score,
but noise — and a concrete reason to fix DQ-2. Chunk-level search is immune (it
filters `bibkey IS NOT NULL`, and 1911.00966 has no bibkey/chunks).

**Note:** the MCP `search` docstring claims "hybrid dense + FTS" but the SQL is
dense-only; `score = 1.0 - dist` can go negative (bge-m3 L2 ranges 0–2). Ranking
is fine (monotonic); only the absolute score label is misleading.

**Original open question (for context):**
- **Open question:** end-to-end, do real queries return *relevant* results? Clean
  chunks + a working index don't guarantee useful retrieval.
- **To investigate next:** run a handful of domain queries through the actual
  search path (`codex search paper "<q>"` and the chunk-level MCP `search`), e.g.
  "discrete conformal map", "circle packing rigidity", "combinatorial Yamabe
  flow", "discrete Laplace-Beltrami operator", and judge whether the top-5 hits
  are on-topic and point at the right papers. Cross-check a couple against
  `--cite-boost` to see if the boost helps or hurts on this corpus.
- **Acceptance:** a short relevance table (query → top-5 → on/off-topic) good
  enough to trust the KB for lookups, or a list of failure modes to fix.

### DQ-5 — `ingest_paper` is NOT idempotent for DOI papers · **HIGH (blocks R-A/R-C)** · found 2026-06-16
- **Symptom:** re-ingesting an existing DOI paper crashes with
  `UniqueViolation: papers_openalex_id_key`. Surfaced trying to re-ingest
  `10.1007/s00454-019-00132-8` during the DQ-2 backfill.
- **Root cause:** `openalex._map_paper` sets `Paper.id = data["doi"]`, which is the
  **full URL** `https://doi.org/10.1007/…`. The stored canonical id is the **bare**
  `10.1007/…` (post the M-1 canonical-id migration). So `INSERT … ON CONFLICT (id)`
  finds no id match, attempts a fresh INSERT, and trips the *separate* unique
  constraint on `openalex_id` (already held by the canonical-id row).
- **Confirmed:** `openalex.fetch_paper("10.1007/s00454-019-00132-8").id ==
  "https://doi.org/10.1007/s00454-019-00132-8"` (≠ stored bare id).
- **Impact:** any re-ingest of a DOI paper aborts. This **blocks roadmap R-A
  (GROBID on PDFs) and R-C (.tex ingest)** — both re-ingest existing papers. Also
  implies *new* DOI ingests store full-URL ids, inconsistent with the bare-id
  corpus (latent split-identity risk).
- **Proposed fix (handle with care — M-1 incident territory):** normalize the DOI
  in `_map_paper` to the canonical bare, lower-cased form (strip
  `https://doi.org/`), matching what the M-1 migration produced. Add a re-ingest
  idempotency test over a DOI paper. Did **not** fix inline this session — the
  canonical-id derivation is exactly where the M-1 migration incident occurred, so
  it needs its own focused change + a live re-ingest check.
- **Workaround used for DQ-2:** targeted `UPDATE … SET abstract, abstract_emb`
  instead of full re-ingest.

---

## Roadmap — data-acquisition levers (post-audit, all free sources)

These are *forward-looking acquisition improvements*, distinct from the DQ-1..DQ-4
audit findings (which assessed existing data). They came out of a strategy
discussion on 2026-06-16: for this Math/CS corpus the valuable data (references,
abstracts, full text) is almost entirely **open** (arXiv, OpenAlex, Crossref, S2),
so the quality lever is *combining more free sources*, not buying paywall access.
Paywall/uni-login was explicitly considered and **deferred** (see R-E). Each item
is self-contained so a cold session can pick it up.

### R-A — Run GROBID reference extraction on arXiv PDFs · **HIGH lever, LOW-MED effort**
- **⚠ Blocked by DQ-5:** re-ingesting existing DOI papers currently crashes
  (`papers_openalex_id_key`). Fix DQ-5 first, or the GROBID re-ingest aborts.
- **What:** the whole corpus was ingested from `.txt` (`PAPERS_DIR=…/papers/txt`),
  so the GROBID PDF path never ran. Re-ingest with a PDF `source_path` per paper
  so `codex.parsing.grobid.extract_references` parses each bibliography.
- **Why:** extracts references the APIs lack — a *complement* to DQ-1's S2
  supplement. Crucially, the **2 `depositonce` theses** that stayed at zero in
  DQ-1 (no API references anywhere) almost certainly have a references section in
  their open-access PDFs → GROBID could finally give them out-edges. Closes the
  last 2/29.
- **How:** no new code — ingest already merges + dedups `api_citations +
  pdf_citations` ([codex/ingest.py:288](../../codex/ingest.py)). GROBID server is
  already configured (`GROBID_URL=http://192.168.178.103:8070` in
  `.env.jetson-ingest`). Download free arXiv PDFs, re-ingest with `source_path`.
- **Acceptance:** the 2 theses gain references; published papers gain
  GROBID-parsed refs beyond OpenAlex/S2 coverage; citing coverage 27/29 → 29/29.

### R-B — Add Crossref as a third citation + abstract source · **MED lever, MED effort**
- **What:** new `codex.sources.crossref` client hitting Crossref REST
  `/works/{doi}` for the `reference` array and `abstract`.
- **Why:** Crossref carries **publisher-deposited references** for many DOIs (free,
  polite pool via `mailto`) — a third leg after OpenAlex/S2. It also recovers
  abstracts OpenAlex is contractually barred from redistributing (likely the 2
  Springer DOIs missing abstracts in DQ-2). cited_ids are DOIs → flow through the
  existing graph resolver unchanged.
- **How:** mirror `codex/sources/openalex.py` (tenacity retry, polite pool). Extend
  the ingest fallback chain to OpenAlex-empty → S2 → Crossref (same
  `_s2_reference_supplement` shape). Add unit tests with mocked HTTP.
- **Acceptance:** Crossref recovers refs and/or an abstract for ≥1 paper where
  OpenAlex+S2 are empty.

### R-C — Prefer arXiv `.tex` source over `.txt` for chunk fidelity · **MED lever, MED-HIGH effort**
- **What:** re-ingest from arXiv LaTeX source through the existing
  `codex.parsing.tex.latex_to_text` path instead of flattened `.txt`.
- **Why:** highest-fidelity input — preserves math, structure, and section
  boundaries. Directly serves **DQ-3** (content fidelity) and audit **R-12** (the
  `section` column is low-signal because word-window `.txt` chunks rarely start at
  a real header; `.tex` headers fix this).
- **How:** download free arXiv source tarballs, feed `.tex` to ingest with
  `source_path`. Handle multi-file projects (main `.tex` + includes).
- **Acceptance:** `section` column gains signal (fewer `body`-only); DQ-3 coverage
  vs source improves; no regression in the F-16 quality gate.

### R-D — F-15: warn on low *citing-paper* coverage, not just paper count · **SMALL**
- **What:** the open DQ-1 sub-item: `graph_min_corpus_size` only flags total paper
  count. Add a warning when the share of papers with ≥1 out-edge is low (e.g.
  < 80%), since that is what actually starves PageRank/coupling.
- **Why:** at 17/29 the graph silently ran on 59% of the corpus with no signal.
  Post-DQ-1 it's 27/29 (93%) — but the guard should catch future regressions.
- **Acceptance:** `codex graph report` surfaces citing-coverage % and warns below
  threshold.

### R-E — Paywall / institutional access · **DEFERRED — only for a paywall-only expansion**
- **Decision (2026-06-16):** a uni login does **not** help the current corpus —
  every DQ-1/DQ-2 gap is in openly-available works (arXiv preprints, OA theses),
  and all metadata/citation sources are free APIs. Revisit *only* if the corpus
  expands to papers that exist solely behind a paywall with no preprint; then
  institutional access → full-text PDF → GROBID refs (R-A) + clean chunks (R-C).
- **ToS caveat:** manual download of individual papers you have legitimate access
  to is fine; **automated bulk download through a uni proxy (EZproxy/Shibboleth)
  violates most publisher terms and can get the whole institution's access
  revoked.** Do not wire a publisher login into the ingest pipeline.

---

## Priority for the next session
1. ~~**DQ-1**~~ — **DONE 2026-06-16** (live DB at 920 citations, 27/29 citing
   coverage; remedy wired into ingest).
2. ~~**DQ-4**~~ — **DONE 2026-06-16** (P0 `search paper` crash fixed; relevance
   verified strong; surfaced the zero-vector pollution that motivates DQ-2).
3. ~~**DQ-2**~~ — **DONE 2026-06-16** (1911.00966 fully recovered; s00454 abstract
   from S2; book chapter documented; recovery wired into ingest).
4. **DQ-5** (NEXT — `ingest_paper` not idempotent for DOI papers; **blocks R-A/R-C**;
   careful — M-1 canonical-id territory), then **DQ-3** (chunk-vs-source fidelity).
5. **Roadmap levers** (see section above): **R-A** (GROBID on arXiv PDFs — closes
   the last 2 zero-citation theses; needs DQ-5 first), **R-B** (Crossref refs —
   abstract half already shipped in DQ-2), **R-C** (`.tex` ingest — serves DQ-3;
   needs DQ-5 first), **R-D** (F-15 coverage warning, quick). **R-E** (paywall)
   deferred.

DQ-1..DQ-4 are *data*/assessment work; DQ-5 + roadmap R-A..R-D add **new code**
(canonical-id fix, re-ingest scripts, Crossref references, a `.tex` path, a graph
warning) on top of the already-remediated pipeline (see the AUDIT-* docs and PRs
#12–#14). The DQ-1/DQ-2 ingest wiring + Crossref/arXiv abstract recovery already
landed this session.

---

## Appendix — reproduce the baseline scan

```python
import os, psycopg, statistics
from collections import Counter, defaultdict
from psycopg.rows import dict_row
from codex.config import Settings
from codex.quality import is_quality_chunk, _bib_score

s = Settings()
with psycopg.connect(os.environ["DATABASE_URL"], row_factory=dict_row) as c:
    papers = c.execute("SELECT id, bibkey, title, abstract, year, authors, openalex_id FROM papers").fetchall()
    chunks = c.execute("SELECT paper_id, content, section FROM chunks").fetchall()
    cit = c.execute("SELECT citing_id, count(*) n FROM citations GROUP BY citing_id").fetchall()

# zero-citation papers (DQ-1)
cited = {r["citing_id"] for r in cit}
zero = [p["id"] for p in papers if p["id"] not in cited]
# metadata gaps (DQ-2): p["abstract"] empty / p["bibkey"] None / not p["authors"]
# F-16 re-gate (baseline): [ch for ch in chunks if not is_quality_chunk(ch["content"], settings=s)]
# section dist (R-12): Counter(ch["section"] or "(null)" for ch in chunks)
```