codex-py/docs/audit/DATA-QUALITY-2026-06-15.md

# DATA-QUALITY AUDIT — codex knowledge base (handoff for a fresh session)

**Status:** baseline scan done 2026-06-15; four investigation items open (DQ-1…DQ-4).
**Author:** Audit-Loop (Opus). **Intended reader:** a *cold* session with no memory
of the audit conversation — everything needed to continue is in this file.

---

## Why this exists — the second audit axis

The code/pipeline audit (`AUDIT-2026-06-15-loop-1.md`, `-full-repo.md`) verified the
**loader is correct**: IDs are canonical, the citation graph is consistent, the
ingest pipeline does what it claims. It did **not** ask whether the *information
in the database is actually good*. A correct loader can faithfully load thin,
incomplete, or unrepresentative data. This document is that second axis — a
**data-quality** assessment of the live corpus, independent of code correctness.

---

## How to reach the data (self-contained)

- **Live DB:** Jetson PostgreSQL via an SSH tunnel (open it first):
  ```
  ssh -f -N -L 5433:localhost:5432 alfred@192.168.178.103
  ```
- **Credentials:** `.env.jetson-ingest` (gitignored) holds `DATABASE_URL`
  (`postgresql://researcher:…@localhost:5433/papers`). The `researcher` role is
  DML-only (read freely; no DDL).
- **Run probes:** `set -a; source .env.jetson-ingest; set +a` then
  `PYTHONPATH=. .venv/bin/python` with `psycopg` + `psycopg.rows.dict_row`.
  Reusable domain helpers: `codex.quality` (`is_quality_chunk`, `_bib_score`,
  `classify_section`), `codex.graph`, `codex.discover`.
- **Corpus snapshot (2026-06-15, post canonical-id migration):**
  **29 papers, 1507 chunks, 590 citations.**

---

## Baseline — what is already GOOD (measured, no action needed)

- **Chunk content is clean.** Re-running the F-16 quality gate over all stored
  chunks: **0 / 1507 fail** `is_quality_chunk` (none too-short, none low-alpha /
  OCR-ish, none bibliography-like). The gate did its job on the `.txt` ingest.
- **Chunk sizing healthy.** chars: min 478 / median 2913 / max 4154; **0 exact
  duplicate** chunk bodies.
- **`section` column is low-signal** (98 % `body`; 13 `bibliography`, 9 `theorem`,
  2 `intro`, 1 `proof`). This confirms code-audit **R-12** — word-window chunks
  rarely start at a section header. Not a data defect; the column is just weak.

---

## Findings (open investigation items)

### DQ-1 — Citation coverage: 12 / 29 papers (41 %) have ZERO citations · **HIGH**
- **Measured:** 12 papers contribute **no** out-edges; the 590-edge citation graph
  comes from only ~17 papers. The whole F-15 layer (PageRank / coupling /
  co-citation / discovery leads) therefore runs on ~59 % of the corpus.
- **Sharpening — it is mostly a source-coverage limit, not an ingest bug:**
  **11 of the 12** zero-citation papers *do* have an `openalex_id`, i.e. ingest
  called `openalex.fetch_citations(openalex_id)` and OpenAlex returned an **empty
  `referenced_works`** list. Only `1911.00966` has no `openalex_id` (OpenAlex
  404 → arXiv/S2 fallback, which also yielded nothing).
- **Zero-citation ids:** `1005.2698`, `1505.01341`, `1911.00966` (no oa),
  `2206.13461`, `2305.10988`, `2310.17529`, `2601.22903`, `math/0001176`,
  `math/0503219`, `math/0603097`, `10.14279/depositonce-20357`,
  `10.14279/depositonce-5415`. (Pattern: arXiv preprints + the two `depositonce`
  theses — works OpenAlex indexes without parsed references.)
- **To investigate next:**
  1. Confirm it is OpenAlex coverage, not a `fetch_citations` bug: for 2–3 of the
     ids, hit `GET https://api.openalex.org/works/<openalex_id>` and check whether
     `referenced_works` is genuinely empty server-side.
  2. If genuinely empty: enrich via the **Semantic Scholar references** path
     (`semanticscholar.fetch_references("arXiv:<id>")`) as a *supplement* for
     OpenAlex-empty papers — S2 often has references where OpenAlex doesn't. Note
     S2 cited_ids are bare DOI/arXiv, so they flow through the existing
     `RESOLVED_CITATIONS_SQL` resolver (audit C-1) fine.
  3. Decide whether the F-15 `graph_min_corpus_size` warning should also flag *low
     citing-paper coverage*, not just paper count.
- **Acceptance:** either (a) citation coverage materially improves after an S2
  supplement, or (b) documented as an inherent OpenAlex-coverage limit with the
  graph caveated accordingly.

### DQ-2 — Metadata gaps: 3 no-abstract, 1 fully metadata-less · **MED**
- **Measured:**
  - **No abstract (3):** `10.1007/978-3-642-17413-1_7`,
    `10.1007/s00454-019-00132-8`, `1911.00966`. These get a **zero-vector**
    `abstract_emb`, so paper-level semantic search can't place them.
  - **No bibkey / year (1):** `1911.00966` only — and its authors array is empty
    too. This paper is **fully degraded** (OpenAlex 404 → `Paper(id, title="")`
    fallback): no abstract, no bibkey, no year, empty authors, no citations. With
    no bibkey it cannot be `@cite`d and is invisible to wiki grounding (which keys
    on bibkey).
- **To investigate next:**
  1. `1911.00966` — confirm the arXiv id is correct and whether OpenAlex/S2 has it
     under a different id (DOI?); if recoverable, re-ingest to populate metadata.
     If not, decide: keep as a degraded node or drop.
  2. The 2 no-abstract DOIs — check if OpenAlex has an `abstract_inverted_index`
     that the mapper missed, or if the abstract is genuinely absent upstream.
- **Acceptance:** every paper has at least a bibkey + non-zero abstract embedding,
  or the exceptions are documented with rationale.

### DQ-3 — Content fidelity (chunks vs source `.txt`): NOT YET VERIFIED · **MED**
- **Open question:** does the stored chunk set faithfully reconstruct each source
  file, or did the chunker / `filter_chunks` silently drop material (e.g. an
  abstract, a section, math-heavy passages)? The whole corpus was ingested from
  `.txt` (`PAPERS_DIR=/Users/tarikmoussa/Desktop/ConformalLabpp/papers/txt`), so
  the `.tex`/`.pdf` paths never ran.
- **To investigate next:** for a sample of ~5 papers, compare
  `len("".join(stored chunks))` against `len(source.txt)` (coverage %), and
  eyeball the first/last chunk vs the file head/tail. Flag papers where coverage
  is low (content lost) — F-16 dropping >X % of a paper is a signal.
- **Acceptance:** sampled papers retain ≳ the expected fraction of source text;
  no paper is silently gutted by the quality gate.

### DQ-4 — Retrieval quality: NOT YET MEASURED · **MED**
- **Open question:** end-to-end, do real queries return *relevant* results? Clean
  chunks + a working index don't guarantee useful retrieval.
- **To investigate next:** run a handful of domain queries through the actual
  search path (`codex search paper "<q>"` and the chunk-level MCP `search`), e.g.
  "discrete conformal map", "circle packing rigidity", "combinatorial Yamabe
  flow", "discrete Laplace-Beltrami operator", and judge whether the top-5 hits
  are on-topic and point at the right papers. Cross-check a couple against
  `--cite-boost` to see if the boost helps or hurts on this corpus.
- **Acceptance:** a short relevance table (query → top-5 → on/off-topic) good
  enough to trust the KB for lookups, or a list of failure modes to fix.

---

## Priority for the next session
1. **DQ-1** (biggest lever — 41 % of the corpus is invisible to the graph; the S2
   supplement is concrete and reuses existing code).
2. **DQ-4** (cheap, high-information — tells you if the KB is actually usable).
3. **DQ-2**, then **DQ-3**.

All four are *data* work (queries + maybe a re-ingest/enrichment), not code-audit
work — the code is already remediated (see the AUDIT-* docs and PRs #12–#14).

---

## Appendix — reproduce the baseline scan

```python
import os, psycopg, statistics
from collections import Counter, defaultdict
from psycopg.rows import dict_row
from codex.config import Settings
from codex.quality import is_quality_chunk, _bib_score

s = Settings()
with psycopg.connect(os.environ["DATABASE_URL"], row_factory=dict_row) as c:
    papers = c.execute("SELECT id, bibkey, title, abstract, year, authors, openalex_id FROM papers").fetchall()
    chunks = c.execute("SELECT paper_id, content, section FROM chunks").fetchall()
    cit = c.execute("SELECT citing_id, count(*) n FROM citations GROUP BY citing_id").fetchall()

# zero-citation papers (DQ-1)
cited = {r["citing_id"] for r in cit}
zero = [p["id"] for p in papers if p["id"] not in cited]
# metadata gaps (DQ-2): p["abstract"] empty / p["bibkey"] None / not p["authors"]
# F-16 re-gate (baseline): [ch for ch in chunks if not is_quality_chunk(ch["content"], settings=s)]
# section dist (R-12): Counter(ch["section"] or "(null)" for ch in chunks)
```