docs/audit/DATA-QUALITY-2026-06-15.md

@@ -0,0 +1,160 @@
+# DATA-QUALITY AUDIT — codex knowledge base (handoff for a fresh session)
+
+**Status:** baseline scan done 2026-06-15; four investigation items open (DQ-1…DQ-4).
+**Author:** Audit-Loop (Opus). **Intended reader:** a *cold* session with no memory
+of the audit conversation — everything needed to continue is in this file.
+
+---
+
+## Why this exists — the second audit axis
+
+The code/pipeline audit (`AUDIT-2026-06-15-loop-1.md`, `-full-repo.md`) verified the
+**loader is correct**: IDs are canonical, the citation graph is consistent, the
+ingest pipeline does what it claims. It did **not** ask whether the *information
+in the database is actually good*. A correct loader can faithfully load thin,
+incomplete, or unrepresentative data. This document is that second axis — a
+**data-quality** assessment of the live corpus, independent of code correctness.
+
+---
+
+## How to reach the data (self-contained)
+
+- **Live DB:** Jetson PostgreSQL via an SSH tunnel (open it first):
+  ```
+  ssh -f -N -L 5433:localhost:5432 alfred@192.168.178.103
+  ```
+- **Credentials:** `.env.jetson-ingest` (gitignored) holds `DATABASE_URL`
+  (`postgresql://researcher:…@localhost:5433/papers`). The `researcher` role is
+  DML-only (read freely; no DDL).
+- **Run probes:** `set -a; source .env.jetson-ingest; set +a` then
+  `PYTHONPATH=. .venv/bin/python` with `psycopg` + `psycopg.rows.dict_row`.
+  Reusable domain helpers: `codex.quality` (`is_quality_chunk`, `_bib_score`,
+  `classify_section`), `codex.graph`, `codex.discover`.
+- **Corpus snapshot (2026-06-15, post canonical-id migration):**
+  **29 papers, 1507 chunks, 590 citations.**
+
+---
+
+## Baseline — what is already GOOD (measured, no action needed)
+
+- **Chunk content is clean.** Re-running the F-16 quality gate over all stored
+  chunks: **0 / 1507 fail** `is_quality_chunk` (none too-short, none low-alpha /
+  OCR-ish, none bibliography-like). The gate did its job on the `.txt` ingest.
+- **Chunk sizing healthy.** chars: min 478 / median 2913 / max 4154; **0 exact
+  duplicate** chunk bodies.
+- **`section` column is low-signal** (98 % `body`; 13 `bibliography`, 9 `theorem`,
+  2 `intro`, 1 `proof`). This confirms code-audit **R-12** — word-window chunks
+  rarely start at a section header. Not a data defect; the column is just weak.
+
+---
+
+## Findings (open investigation items)
+
+### DQ-1 — Citation coverage: 12 / 29 papers (41 %) have ZERO citations · **HIGH**
+- **Measured:** 12 papers contribute **no** out-edges; the 590-edge citation graph
+  comes from only ~17 papers. The whole F-15 layer (PageRank / coupling /
+  co-citation / discovery leads) therefore runs on ~59 % of the corpus.
+- **Sharpening — it is mostly a source-coverage limit, not an ingest bug:**
+  **11 of the 12** zero-citation papers *do* have an `openalex_id`, i.e. ingest
+  called `openalex.fetch_citations(openalex_id)` and OpenAlex returned an **empty
+  `referenced_works`** list. Only `1911.00966` has no `openalex_id` (OpenAlex
+  404 → arXiv/S2 fallback, which also yielded nothing).
+- **Zero-citation ids:** `1005.2698`, `1505.01341`, `1911.00966` (no oa),
+  `2206.13461`, `2305.10988`, `2310.17529`, `2601.22903`, `math/0001176`,
+  `math/0503219`, `math/0603097`, `10.14279/depositonce-20357`,
+  `10.14279/depositonce-5415`. (Pattern: arXiv preprints + the two `depositonce`
+  theses — works OpenAlex indexes without parsed references.)
+- **To investigate next:**
+  1. Confirm it is OpenAlex coverage, not a `fetch_citations` bug: for 2–3 of the
+     ids, hit `GET https://api.openalex.org/works/<openalex_id>` and check whether
+     `referenced_works` is genuinely empty server-side.
+  2. If genuinely empty: enrich via the **Semantic Scholar references** path
+     (`semanticscholar.fetch_references("arXiv:<id>")`) as a *supplement* for
+     OpenAlex-empty papers — S2 often has references where OpenAlex doesn't. Note
+     S2 cited_ids are bare DOI/arXiv, so they flow through the existing
+     `RESOLVED_CITATIONS_SQL` resolver (audit C-1) fine.
+  3. Decide whether the F-15 `graph_min_corpus_size` warning should also flag *low
+     citing-paper coverage*, not just paper count.
+- **Acceptance:** either (a) citation coverage materially improves after an S2
+  supplement, or (b) documented as an inherent OpenAlex-coverage limit with the
+  graph caveated accordingly.
+
+### DQ-2 — Metadata gaps: 3 no-abstract, 1 fully metadata-less · **MED**
+- **Measured:**
+  - **No abstract (3):** `10.1007/978-3-642-17413-1_7`,
+    `10.1007/s00454-019-00132-8`, `1911.00966`. These get a **zero-vector**
+    `abstract_emb`, so paper-level semantic search can't place them.
+  - **No bibkey / year (1):** `1911.00966` only — and its authors array is empty
+    too. This paper is **fully degraded** (OpenAlex 404 → `Paper(id, title="")`
+    fallback): no abstract, no bibkey, no year, empty authors, no citations. With
+    no bibkey it cannot be `@cite`d and is invisible to wiki grounding (which keys
+    on bibkey).
+- **To investigate next:**
+  1. `1911.00966` — confirm the arXiv id is correct and whether OpenAlex/S2 has it
+     under a different id (DOI?); if recoverable, re-ingest to populate metadata.
+     If not, decide: keep as a degraded node or drop.
+  2. The 2 no-abstract DOIs — check if OpenAlex has an `abstract_inverted_index`
+     that the mapper missed, or if the abstract is genuinely absent upstream.
+- **Acceptance:** every paper has at least a bibkey + non-zero abstract embedding,
+  or the exceptions are documented with rationale.
+
+### DQ-3 — Content fidelity (chunks vs source `.txt`): NOT YET VERIFIED · **MED**
+- **Open question:** does the stored chunk set faithfully reconstruct each source
+  file, or did the chunker / `filter_chunks` silently drop material (e.g. an
+  abstract, a section, math-heavy passages)? The whole corpus was ingested from
+  `.txt` (`PAPERS_DIR=/Users/tarikmoussa/Desktop/ConformalLabpp/papers/txt`), so
+  the `.tex`/`.pdf` paths never ran.
+- **To investigate next:** for a sample of ~5 papers, compare
+  `len("".join(stored chunks))` against `len(source.txt)` (coverage %), and
+  eyeball the first/last chunk vs the file head/tail. Flag papers where coverage
+  is low (content lost) — F-16 dropping >X % of a paper is a signal.
+- **Acceptance:** sampled papers retain ≳ the expected fraction of source text;
+  no paper is silently gutted by the quality gate.
+
+### DQ-4 — Retrieval quality: NOT YET MEASURED · **MED**
+- **Open question:** end-to-end, do real queries return *relevant* results? Clean
+  chunks + a working index don't guarantee useful retrieval.
+- **To investigate next:** run a handful of domain queries through the actual
+  search path (`codex search paper "<q>"` and the chunk-level MCP `search`), e.g.
+  "discrete conformal map", "circle packing rigidity", "combinatorial Yamabe
+  flow", "discrete Laplace-Beltrami operator", and judge whether the top-5 hits
+  are on-topic and point at the right papers. Cross-check a couple against
+  `--cite-boost` to see if the boost helps or hurts on this corpus.
+- **Acceptance:** a short relevance table (query → top-5 → on/off-topic) good
+  enough to trust the KB for lookups, or a list of failure modes to fix.
+
+---
+
+## Priority for the next session
+1. **DQ-1** (biggest lever — 41 % of the corpus is invisible to the graph; the S2
+   supplement is concrete and reuses existing code).
+2. **DQ-4** (cheap, high-information — tells you if the KB is actually usable).
+3. **DQ-2**, then **DQ-3**.
+
+All four are *data* work (queries + maybe a re-ingest/enrichment), not code-audit
+work — the code is already remediated (see the AUDIT-* docs and PRs #12–#14).
+
+---
+
+## Appendix — reproduce the baseline scan
+
+```python
+import os, psycopg, statistics
+from collections import Counter, defaultdict
+from psycopg.rows import dict_row
+from codex.config import Settings
+from codex.quality import is_quality_chunk, _bib_score
+
+s = Settings()
+with psycopg.connect(os.environ["DATABASE_URL"], row_factory=dict_row) as c:
+    papers = c.execute("SELECT id, bibkey, title, abstract, year, authors, openalex_id FROM papers").fetchall()
+    chunks = c.execute("SELECT paper_id, content, section FROM chunks").fetchall()
+    cit = c.execute("SELECT citing_id, count(*) n FROM citations GROUP BY citing_id").fetchall()
+
+# zero-citation papers (DQ-1)
+cited = {r["citing_id"] for r in cit}
+zero = [p["id"] for p in papers if p["id"] not in cited]
+# metadata gaps (DQ-2): p["abstract"] empty / p["bibkey"] None / not p["authors"]
+# F-16 re-gate (baseline): [ch for ch in chunks if not is_quality_chunk(ch["content"], settings=s)]
+# section dist (R-12): Counter(ch["section"] or "(null)" for ch in chunks)
+```