fix(wiki): make grounding tokenizer robust to punctuation (audit R-1)

The grounding guard tokenized with .lower().split() and split sentences on bare
.!?, so faithful claims were wrongly marked ungrounded: 'volume,' != 'volume',
and a decimal like 'V = 1.5' split into a 1-word fragment '5' that fell below
the 5-content-word floor.

- _content_words now strips edge sentence-punctuation (.,;:!?"') from tokens
  while preserving math delimiters ()=+; claim and chunk are normalised
  identically so matching stays consistent.
- _SENTENCE_SPLIT_RE splits on .!? only when followed by whitespace/end, so
  decimals no longer create spurious sentence boundaries.

Regression test: a 5-word claim with a trailing comma now grounds.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-15 16:37:30 +02:00
parent 77f9d79da0
commit e7b854db10
2 changed files with 34 additions and 3 deletions

View File

@@ -273,7 +273,13 @@ _STOPWORDS: frozenset[str] = frozenset(
}
)
_SENTENCE_SPLIT_RE = re.compile(r"[.!?]")
# Split on sentence-ending punctuation only when followed by whitespace/end, so a
# decimal like "1.5" does not create a spurious sentence boundary (audit R-1).
_SENTENCE_SPLIT_RE = re.compile(r"[.!?]+(?=\s|$)")
# Punctuation stripped from token edges before grounding comparison — sentence/
# clause marks and quotes, NOT math delimiters like ()=+ (audit R-1).
_EDGE_PUNCT = ".,;:!?\"'"
def _last_sentence(text: str) -> str:
@@ -292,8 +298,16 @@ def _last_sentence(text: str) -> str:
def _content_words(text: str) -> list[str]:
"""Return lowercased tokens with stopwords removed."""
return [w for w in text.lower().split() if w not in _STOPWORDS]
"""Return lowercased content tokens, edge-punctuation-stripped, stopwords removed.
Stripping leading/trailing sentence punctuation makes grounding robust to
surface variation ("volume," vs "volume") so a faithful paraphrase is not
marked ungrounded over a comma (audit R-1). Math delimiters (``()=+``) are
preserved. Claim and chunk tokens are normalised identically, so matching
stays consistent.
"""
tokens = (w.strip(_EDGE_PUNCT) for w in text.lower().split())
return [w for w in tokens if w and w not in _STOPWORDS]
def _run_grounding_guard(