user2595/codex-py
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

merge: reconcile data-quality roadmap with audit-remediation main (#12-15)

Browse Source 
Resolved 4 conflicts: cli.py (R-D citing-coverage warning + main's small-corpus JSON/hub-title changes both kept); ingest.py (C-7 paper.id=caller-id pin first, then DQ-2 abstract recovery); test_ingest.py + test_openalex.py (kept both branches' tests).

C-7/DQ-5 overlap (both canonicalise papers.id): kept both layers — openalex._canonical_id normalizes fetch_paper's id (DQ-5); ingest pins papers.id to the caller's bare id (C-7); they converge on the bare canonical id. Fixed the auto-merged test_fetch_paper_success that had contradictory bare/URL assertions. 402 tests pass; ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-22 15:11:26 +02:00
parent e817022097 d64edfc324
commit 9e5184385a
29 changed files with 900 additions and 193 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
codex/graph.py
View File

@@ -23,6 +23,23 @@ logger = logging.getLogger(__name__)
_MIN_PAGERANK_NODES = 5
# Single source of truth for resolving ``citations.cited_id`` to a canonical
# paper id. ``cited_id`` stores OpenAlex IDs while ``papers.id`` stores DOIs /
# arXiv ids, so an in-KB reference cited by its OpenAlex id does not match
# ``papers.id`` directly. This resolves it to the canonical ``papers.id`` (via
# ``papers.openalex_id``, then the ``paper_identifiers`` alias table); dangling
# references keep their raw id. Both :func:`build_citation_graph` and
# :mod:`codex.discover` query through this so the "what counts as ingested" rule
# cannot drift between them again (audit C-1). Trusted constant — no user input.
RESOLVED_CITATIONS_SQL = """
SELECT c.citing_id,
COALESCE(p.id, pi.paper_id, c.cited_id) AS cited_id
FROM citations c
LEFT JOIN papers p ON p.openalex_id = c.cited_id
LEFT JOIN paper_identifiers pi
ON pi.openalex_id = c.cited_id AND p.id IS NULL
"""
def build_citation_graph(conn: Any) -> nx.DiGraph:
"""Load the ``citations`` table into a directed graph.
@@ -45,15 +62,7 @@ def build_citation_graph(conn: Any) -> nx.DiGraph:
-------
nx.DiGraph with every (citing_id, cited_id) pair as an edge.
"""
rows = conn.execute(
"""
SELECT c.citing_id, COALESCE(p.id, pi.paper_id, c.cited_id) AS cited_id
FROM citations c
LEFT JOIN papers p ON p.openalex_id = c.cited_id
LEFT JOIN paper_identifiers pi
ON pi.openalex_id = c.cited_id AND p.id IS NULL
"""
).fetchall()
rows = conn.execute(RESOLVED_CITATIONS_SQL).fetchall()
g: nx.DiGraph = nx.DiGraph()
for row in rows:
g.add_edge(row["citing_id"], row["cited_id"])