codex-py/docs/audit/DATA-QUALITY-2026-06-15.md

DATA-QUALITY AUDIT — codex knowledge base (handoff for a fresh session)

Status: baseline scan done 2026-06-15; four investigation items open (DQ-1…DQ-4). Author: Audit-Loop (Opus). Intended reader: a cold session with no memory of the audit conversation — everything needed to continue is in this file.

---

Why this exists — the second audit axis

The code/pipeline audit (AUDIT-2026-06-15-loop-1.md, -full-repo.md) verified the loader is correct: IDs are canonical, the citation graph is consistent, the ingest pipeline does what it claims. It did not ask whether the information in the database is actually good. A correct loader can faithfully load thin, incomplete, or unrepresentative data. This document is that second axis — a data-quality assessment of the live corpus, independent of code correctness.

---

How to reach the data (self-contained)

• Live DB: Jetson PostgreSQL via an SSH tunnel (open it first):

  ssh -f -N -L 5433:localhost:5432 alfred@192.168.178.103
  

• Credentials: .env.jetson-ingest (gitignored) holds DATABASE_URL (postgresql://researcher:…@localhost:5433/papers). The researcher role is DML-only (read freely; no DDL).
• Run probes: set -a; source .env.jetson-ingest; set +a then PYTHONPATH=. .venv/bin/python with psycopg + psycopg.rows.dict_row. Reusable domain helpers: codex.quality (is_quality_chunk, _bib_score, classify_section), codex.graph, codex.discover.
• Corpus snapshot (2026-06-15, post canonical-id migration): 29 papers, 1507 chunks, 590 citations.

---

Baseline — what is already GOOD (measured, no action needed)

• Chunk content is clean. Re-running the F-16 quality gate over all stored chunks: 0 / 1507 fail is_quality_chunk (none too-short, none low-alpha / OCR-ish, none bibliography-like). The gate did its job on the .txt ingest.
• Chunk sizing healthy. chars: min 478 / median 2913 / max 4154; 0 exact duplicate chunk bodies.
• section column is low-signal (98 % body; 13 bibliography, 9 theorem, 2 intro, 1 proof). This confirms code-audit R-12 — word-window chunks rarely start at a section header. Not a data defect; the column is just weak.

---

Findings (open investigation items)

DQ-1 — Citation coverage: 12 / 29 papers (41 %) have ZERO citations · HIGH

• Measured: 12 papers contribute no out-edges; the 590-edge citation graph comes from only ~17 papers. The whole F-15 layer (PageRank / coupling / co-citation / discovery leads) therefore runs on ~59 % of the corpus.
• Sharpening — it is mostly a source-coverage limit, not an ingest bug: 11 of the 12 zero-citation papers do have an openalex_id, i.e. ingest called openalex.fetch_citations(openalex_id) and OpenAlex returned an empty referenced_works list. Only 1911.00966 has no openalex_id (OpenAlex 404 → arXiv/S2 fallback, which also yielded nothing).
• Zero-citation ids: 1005.2698, 1505.01341, 1911.00966 (no oa), 2206.13461, 2305.10988, 2310.17529, 2601.22903, math/0001176, math/0503219, math/0603097, 10.14279/depositonce-20357, 10.14279/depositonce-5415. (Pattern: arXiv preprints + the two depositonce theses — works OpenAlex indexes without parsed references.)
• To investigate next:
  1. Confirm it is OpenAlex coverage, not a fetch_citations bug: for 2–3 of the ids, hit GET https://api.openalex.org/works/<openalex_id> and check whether referenced_works is genuinely empty server-side.
  2. If genuinely empty: enrich via the Semantic Scholar references path (semanticscholar.fetch_references("arXiv:<id>")) as a supplement for OpenAlex-empty papers — S2 often has references where OpenAlex doesn't. Note S2 cited_ids are bare DOI/arXiv, so they flow through the existing RESOLVED_CITATIONS_SQL resolver (audit C-1) fine.
  3. Decide whether the F-15 graph_min_corpus_size warning should also flag low citing-paper coverage, not just paper count.
• Acceptance: either (a) citation coverage materially improves after an S2 supplement, or (b) documented as an inherent OpenAlex-coverage limit with the graph caveated accordingly.

DQ-2 — Metadata gaps: 3 no-abstract, 1 fully metadata-less · MED

• Measured:
  • No abstract (3): 10.1007/978-3-642-17413-1_7, 10.1007/s00454-019-00132-8, 1911.00966. These get a zero-vector abstract_emb, so paper-level semantic search can't place them.
  • No bibkey / year (1): 1911.00966 only — and its authors array is empty too. This paper is fully degraded (OpenAlex 404 → Paper(id, title="") fallback): no abstract, no bibkey, no year, empty authors, no citations. With no bibkey it cannot be @cited and is invisible to wiki grounding (which keys on bibkey).
• To investigate next:
  1. 1911.00966 — confirm the arXiv id is correct and whether OpenAlex/S2 has it under a different id (DOI?); if recoverable, re-ingest to populate metadata. If not, decide: keep as a degraded node or drop.
  2. The 2 no-abstract DOIs — check if OpenAlex has an abstract_inverted_index that the mapper missed, or if the abstract is genuinely absent upstream.
• Acceptance: every paper has at least a bibkey + non-zero abstract embedding, or the exceptions are documented with rationale.

DQ-3 — Content fidelity (chunks vs source .txt): NOT YET VERIFIED · MED

• Open question: does the stored chunk set faithfully reconstruct each source file, or did the chunker / filter_chunks silently drop material (e.g. an abstract, a section, math-heavy passages)? The whole corpus was ingested from .txt (PAPERS_DIR=/Users/tarikmoussa/Desktop/ConformalLabpp/papers/txt), so the .tex/.pdf paths never ran.
• To investigate next: for a sample of ~5 papers, compare len("".join(stored chunks)) against len(source.txt) (coverage %), and eyeball the first/last chunk vs the file head/tail. Flag papers where coverage is low (content lost) — F-16 dropping >X % of a paper is a signal.
• Acceptance: sampled papers retain ≳ the expected fraction of source text; no paper is silently gutted by the quality gate.

DQ-4 — Retrieval quality: NOT YET MEASURED · MED

• Open question: end-to-end, do real queries return relevant results? Clean chunks + a working index don't guarantee useful retrieval.
• To investigate next: run a handful of domain queries through the actual search path (codex search paper "<q>" and the chunk-level MCP search), e.g. "discrete conformal map", "circle packing rigidity", "combinatorial Yamabe flow", "discrete Laplace-Beltrami operator", and judge whether the top-5 hits are on-topic and point at the right papers. Cross-check a couple against --cite-boost to see if the boost helps or hurts on this corpus.
• Acceptance: a short relevance table (query → top-5 → on/off-topic) good enough to trust the KB for lookups, or a list of failure modes to fix.

---

Priority for the next session

1. DQ-1 (biggest lever — 41 % of the corpus is invisible to the graph; the S2 supplement is concrete and reuses existing code).
2. DQ-4 (cheap, high-information — tells you if the KB is actually usable).
3. DQ-2, then DQ-3.

All four are data work (queries + maybe a re-ingest/enrichment), not code-audit work — the code is already remediated (see the AUDIT-* docs and PRs #12–#14).

---

Appendix — reproduce the baseline scan

import os, psycopg, statistics
from collections import Counter, defaultdict
from psycopg.rows import dict_row
from codex.config import Settings
from codex.quality import is_quality_chunk, _bib_score

s = Settings()
with psycopg.connect(os.environ["DATABASE_URL"], row_factory=dict_row) as c:
    papers = c.execute("SELECT id, bibkey, title, abstract, year, authors, openalex_id FROM papers").fetchall()
    chunks = c.execute("SELECT paper_id, content, section FROM chunks").fetchall()
    cit = c.execute("SELECT citing_id, count(*) n FROM citations GROUP BY citing_id").fetchall()

# zero-citation papers (DQ-1)
cited = {r["citing_id"] for r in cit}
zero = [p["id"] for p in papers if p["id"] not in cited]
# metadata gaps (DQ-2): p["abstract"] empty / p["bibkey"] None / not p["authors"]
# F-16 re-gate (baseline): [ch for ch in chunks if not is_quality_chunk(ch["content"], settings=s)]
# section dist (R-12): Counter(ch["section"] or "(null)" for ch in chunks)