user2595/codex-py
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(ingest): normalize GROBID-derived citation DOIs to bare lower-case

Browse Source 
GROBID extracts a cited DOI verbatim from PDF reference text, so it can be mixed-case, a https://doi.org/ URL, or doi:-prefixed. The PDF citation branch inserted it unchanged, while every other path (S2 supplement, OpenAlex, papers.id itself) uses the bare lower-cased DOI. Once R-A runs GROBID reference extraction over arXiv PDFs, those un-normalized cited-ids would never equal a bare-lowercase papers.id/cited_id — dangling the cross-citations and splitting one reference into case-/URL-variant graph nodes.

Extend _norm_cited_id to strip https://doi.org//http://doi.org//doi: prefixes and lower-case (mirroring openalex._normalize_doi), making it the single canonical cited-id normalizer, and route the GROBID branch through it. arXiv ids and S2 paperIds still pass through untouched.

Add test_ingest_pdf_grobid_citations_normalize_doi covering mixed-case, URL-form, and doi:-prefixed DOIs plus an arXiv-only ref.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-17 00:39:04 +02:00
parent b371119264
commit 0021859094
2 changed files with 87 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
codex/ingest.py
View File

@@ -65,10 +65,20 @@ def _s2_id_for(pid: str) -> str | None:
def _norm_cited_id(cited_id: str) -> str:
"""Lowercase DOI cited-ids (DOIs are case-insensitive) so the graph join
does not fragment the same reference into distinct case-variant nodes.
arXiv ids and S2 paperIds are left untouched."""
return cited_id.lower() if cited_id.startswith("10.") else cited_id
"""Normalize a DOI cited-id to its canonical bare, lower-cased form.
DOIs are case-insensitive and may arrive bare (``10.…``), as a
``https://doi.org/…`` URL, or ``doi:…``-prefixed: GROBID emits whichever
form the PDF reference text used, while S2/OpenAlex hand back bare DOIs.
Stripping the prefix and lower-casing keeps the citation-graph join from
fragmenting the same reference into distinct case-/URL-variant nodes. arXiv
ids and S2 paperIds are left untouched. Mirrors ``openalex._normalize_doi``."""
s = cited_id.strip()
lowered = s.lower()
for prefix in ("https://doi.org/", "http://doi.org/", "doi:"):
if lowered.startswith(prefix):
return lowered[len(prefix) :]
return lowered if lowered.startswith("10.") else s
def _recover_abstract(paper: Paper) -> str | None:
@@ -268,7 +278,12 @@ def ingest_paper(
for ref in grobid_refs:
cited_id = ref.get("doi") or ref.get("arxiv_id")
if cited_id:
pdf_citations.append(Citation(citing_id=paper.id, cited_id=cited_id))
# Normalize the GROBID DOI (bare/URL/``doi:`` → bare lower-case)
# so PDF-derived edges join the same graph nodes as the
# S2/OpenAlex paths; arXiv ids pass through untouched.
pdf_citations.append(
Citation(citing_id=paper.id, cited_id=_norm_cited_id(cited_id))
)
else:
logger.warning("Unbekannter Dateityp: %s — kein Text-Parsing", source_path)