Commit Graph

4 Commits

Author SHA1 Message Date
Tarik Moussa
51dc74470c feat(ingest): supplement citations from Semantic Scholar when OpenAlex is empty (DQ-1)
12/29 papers had zero citations: OpenAlex indexes arXiv preprints and theses
without a parsed reference list (verified referenced_works_count=0 server-side
for all 11 with an openalex_id). Not an ingest bug — a source-coverage limit.

- ingest.py: when OpenAlex returns no references, fall back to a Semantic
  Scholar reference supplement (_s2_reference_supplement); citing_id rewritten
  to canonical papers.id, DOI cited-ids lowercased. Removed a now-redundant
  discarded S2 probe in the OpenAlex-404 arXiv branch.
- semanticscholar.py: fix TypeError on S2's HTTP-200 {"data": null} no-refs
  responses (.get("data", []) returns None when the key is present-but-null).
- tests: regression test for the OpenAlex-empty -> S2 path.
- Live DB backfilled idempotently: +330 citations (590->920), zero-out-edge
  papers 12->2, citing coverage 17->27/29. Only the 2 TU-Berlin depositonce
  theses remain (no references in any source).
- docs: DATA-QUALITY-2026-06-15.md — DQ-1 resolution, DQ-4 result, and a
  free-source acquisition roadmap (R-A..R-E).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 23:51:48 +02:00
Tarik Moussa
9726042964 style: ruff format normalization to line-length=100
Committed code predated the line-length=100 ruff config; this brings the
five drifted files into compliance. No logic change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 08:43:53 +02:00
Tarik Moussa
52737abe3d fix(sources): correct OpenAlex ID prefixing, citations endpoint, S2 rate-limit
Review-Gate findings:
- openalex: bare DOIs/arXiv IDs need doi:/arxiv: prefix (bare IDs 404);
  add _resolve_id(); fix fetch_citations to use referenced_works field
  instead of non-existent /works/{id}/references endpoint.
- semanticscholar: wait_fixed(1) was inter-retry only; add per-request
  monotonic rate-limiter (_rate_limit()) before each httpx call.
  Add _is_retryable() filter so 404s don't burn 5 retry slots.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 23:49:35 +02:00
Tarik Moussa
b7f19b6e2c feat(sources): OpenAlex, SemanticScholar, arXiv API clients
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-04 23:31:15 +02:00