diff --git a/docs/audit/DATA-QUALITY-2026-06-15.md b/docs/audit/DATA-QUALITY-2026-06-15.md new file mode 100644 index 0000000..f016a79 --- /dev/null +++ b/docs/audit/DATA-QUALITY-2026-06-15.md @@ -0,0 +1,160 @@ +# DATA-QUALITY AUDIT — codex knowledge base (handoff for a fresh session) + +**Status:** baseline scan done 2026-06-15; four investigation items open (DQ-1…DQ-4). +**Author:** Audit-Loop (Opus). **Intended reader:** a *cold* session with no memory +of the audit conversation — everything needed to continue is in this file. + +--- + +## Why this exists — the second audit axis + +The code/pipeline audit (`AUDIT-2026-06-15-loop-1.md`, `-full-repo.md`) verified the +**loader is correct**: IDs are canonical, the citation graph is consistent, the +ingest pipeline does what it claims. It did **not** ask whether the *information +in the database is actually good*. A correct loader can faithfully load thin, +incomplete, or unrepresentative data. This document is that second axis — a +**data-quality** assessment of the live corpus, independent of code correctness. + +--- + +## How to reach the data (self-contained) + +- **Live DB:** Jetson PostgreSQL via an SSH tunnel (open it first): + ``` + ssh -f -N -L 5433:localhost:5432 alfred@192.168.178.103 + ``` +- **Credentials:** `.env.jetson-ingest` (gitignored) holds `DATABASE_URL` + (`postgresql://researcher:…@localhost:5433/papers`). The `researcher` role is + DML-only (read freely; no DDL). +- **Run probes:** `set -a; source .env.jetson-ingest; set +a` then + `PYTHONPATH=. .venv/bin/python` with `psycopg` + `psycopg.rows.dict_row`. + Reusable domain helpers: `codex.quality` (`is_quality_chunk`, `_bib_score`, + `classify_section`), `codex.graph`, `codex.discover`. +- **Corpus snapshot (2026-06-15, post canonical-id migration):** + **29 papers, 1507 chunks, 590 citations.** + +--- + +## Baseline — what is already GOOD (measured, no action needed) + +- **Chunk content is clean.** Re-running the F-16 quality gate over all stored + chunks: **0 / 1507 fail** `is_quality_chunk` (none too-short, none low-alpha / + OCR-ish, none bibliography-like). The gate did its job on the `.txt` ingest. +- **Chunk sizing healthy.** chars: min 478 / median 2913 / max 4154; **0 exact + duplicate** chunk bodies. +- **`section` column is low-signal** (98 % `body`; 13 `bibliography`, 9 `theorem`, + 2 `intro`, 1 `proof`). This confirms code-audit **R-12** — word-window chunks + rarely start at a section header. Not a data defect; the column is just weak. + +--- + +## Findings (open investigation items) + +### DQ-1 — Citation coverage: 12 / 29 papers (41 %) have ZERO citations · **HIGH** +- **Measured:** 12 papers contribute **no** out-edges; the 590-edge citation graph + comes from only ~17 papers. The whole F-15 layer (PageRank / coupling / + co-citation / discovery leads) therefore runs on ~59 % of the corpus. +- **Sharpening — it is mostly a source-coverage limit, not an ingest bug:** + **11 of the 12** zero-citation papers *do* have an `openalex_id`, i.e. ingest + called `openalex.fetch_citations(openalex_id)` and OpenAlex returned an **empty + `referenced_works`** list. Only `1911.00966` has no `openalex_id` (OpenAlex + 404 → arXiv/S2 fallback, which also yielded nothing). +- **Zero-citation ids:** `1005.2698`, `1505.01341`, `1911.00966` (no oa), + `2206.13461`, `2305.10988`, `2310.17529`, `2601.22903`, `math/0001176`, + `math/0503219`, `math/0603097`, `10.14279/depositonce-20357`, + `10.14279/depositonce-5415`. (Pattern: arXiv preprints + the two `depositonce` + theses — works OpenAlex indexes without parsed references.) +- **To investigate next:** + 1. Confirm it is OpenAlex coverage, not a `fetch_citations` bug: for 2–3 of the + ids, hit `GET https://api.openalex.org/works/` and check whether + `referenced_works` is genuinely empty server-side. + 2. If genuinely empty: enrich via the **Semantic Scholar references** path + (`semanticscholar.fetch_references("arXiv:")`) as a *supplement* for + OpenAlex-empty papers — S2 often has references where OpenAlex doesn't. Note + S2 cited_ids are bare DOI/arXiv, so they flow through the existing + `RESOLVED_CITATIONS_SQL` resolver (audit C-1) fine. + 3. Decide whether the F-15 `graph_min_corpus_size` warning should also flag *low + citing-paper coverage*, not just paper count. +- **Acceptance:** either (a) citation coverage materially improves after an S2 + supplement, or (b) documented as an inherent OpenAlex-coverage limit with the + graph caveated accordingly. + +### DQ-2 — Metadata gaps: 3 no-abstract, 1 fully metadata-less · **MED** +- **Measured:** + - **No abstract (3):** `10.1007/978-3-642-17413-1_7`, + `10.1007/s00454-019-00132-8`, `1911.00966`. These get a **zero-vector** + `abstract_emb`, so paper-level semantic search can't place them. + - **No bibkey / year (1):** `1911.00966` only — and its authors array is empty + too. This paper is **fully degraded** (OpenAlex 404 → `Paper(id, title="")` + fallback): no abstract, no bibkey, no year, empty authors, no citations. With + no bibkey it cannot be `@cite`d and is invisible to wiki grounding (which keys + on bibkey). +- **To investigate next:** + 1. `1911.00966` — confirm the arXiv id is correct and whether OpenAlex/S2 has it + under a different id (DOI?); if recoverable, re-ingest to populate metadata. + If not, decide: keep as a degraded node or drop. + 2. The 2 no-abstract DOIs — check if OpenAlex has an `abstract_inverted_index` + that the mapper missed, or if the abstract is genuinely absent upstream. +- **Acceptance:** every paper has at least a bibkey + non-zero abstract embedding, + or the exceptions are documented with rationale. + +### DQ-3 — Content fidelity (chunks vs source `.txt`): NOT YET VERIFIED · **MED** +- **Open question:** does the stored chunk set faithfully reconstruct each source + file, or did the chunker / `filter_chunks` silently drop material (e.g. an + abstract, a section, math-heavy passages)? The whole corpus was ingested from + `.txt` (`PAPERS_DIR=/Users/tarikmoussa/Desktop/ConformalLabpp/papers/txt`), so + the `.tex`/`.pdf` paths never ran. +- **To investigate next:** for a sample of ~5 papers, compare + `len("".join(stored chunks))` against `len(source.txt)` (coverage %), and + eyeball the first/last chunk vs the file head/tail. Flag papers where coverage + is low (content lost) — F-16 dropping >X % of a paper is a signal. +- **Acceptance:** sampled papers retain ≳ the expected fraction of source text; + no paper is silently gutted by the quality gate. + +### DQ-4 — Retrieval quality: NOT YET MEASURED · **MED** +- **Open question:** end-to-end, do real queries return *relevant* results? Clean + chunks + a working index don't guarantee useful retrieval. +- **To investigate next:** run a handful of domain queries through the actual + search path (`codex search paper ""` and the chunk-level MCP `search`), e.g. + "discrete conformal map", "circle packing rigidity", "combinatorial Yamabe + flow", "discrete Laplace-Beltrami operator", and judge whether the top-5 hits + are on-topic and point at the right papers. Cross-check a couple against + `--cite-boost` to see if the boost helps or hurts on this corpus. +- **Acceptance:** a short relevance table (query → top-5 → on/off-topic) good + enough to trust the KB for lookups, or a list of failure modes to fix. + +--- + +## Priority for the next session +1. **DQ-1** (biggest lever — 41 % of the corpus is invisible to the graph; the S2 + supplement is concrete and reuses existing code). +2. **DQ-4** (cheap, high-information — tells you if the KB is actually usable). +3. **DQ-2**, then **DQ-3**. + +All four are *data* work (queries + maybe a re-ingest/enrichment), not code-audit +work — the code is already remediated (see the AUDIT-* docs and PRs #12–#14). + +--- + +## Appendix — reproduce the baseline scan + +```python +import os, psycopg, statistics +from collections import Counter, defaultdict +from psycopg.rows import dict_row +from codex.config import Settings +from codex.quality import is_quality_chunk, _bib_score + +s = Settings() +with psycopg.connect(os.environ["DATABASE_URL"], row_factory=dict_row) as c: + papers = c.execute("SELECT id, bibkey, title, abstract, year, authors, openalex_id FROM papers").fetchall() + chunks = c.execute("SELECT paper_id, content, section FROM chunks").fetchall() + cit = c.execute("SELECT citing_id, count(*) n FROM citations GROUP BY citing_id").fetchall() + +# zero-citation papers (DQ-1) +cited = {r["citing_id"] for r in cit} +zero = [p["id"] for p in papers if p["id"] not in cited] +# metadata gaps (DQ-2): p["abstract"] empty / p["bibkey"] None / not p["authors"] +# F-16 re-gate (baseline): [ch for ch in chunks if not is_quality_chunk(ch["content"], settings=s)] +# section dist (R-12): Counter(ch["section"] or "(null)" for ch in chunks) +```