fix(wiki): make grounding tokenizer robust to punctuation (audit R-1)

The grounding guard tokenized with .lower().split() and split sentences on bare
.!?, so faithful claims were wrongly marked ungrounded: 'volume,' != 'volume',
and a decimal like 'V = 1.5' split into a 1-word fragment '5' that fell below
the 5-content-word floor.

- _content_words now strips edge sentence-punctuation (.,;:!?"') from tokens
  while preserving math delimiters ()=+; claim and chunk are normalised
  identically so matching stays consistent.
- _SENTENCE_SPLIT_RE splits on .!? only when followed by whitespace/end, so
  decimals no longer create spurious sentence boundaries.

Regression test: a 5-word claim with a trailing comma now grounds.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-15 16:37:30 +02:00
parent 77f9d79da0
commit e7b854db10
2 changed files with 34 additions and 3 deletions

View File

@@ -213,6 +213,23 @@ def test_grounding_guard_last_sentence_grounded_earlier_hallucination_ignored()
)
def test_grounding_guard_robust_to_trailing_comma() -> None:
"""A faithful 5-word claim grounds despite a trailing comma (audit R-1).
The chunk says "... the volume function V0 is strictly concave on the angle
domain." Before edge-punctuation stripping, the claim token "concave," != the
chunk token "concave", so the only content-5-gram failed to match and the
claim was wrongly marked ungrounded.
"""
claim = Claim(
text="the volume function V0 is strictly concave,",
bibkey="Springborn2008",
locator="chunk 16",
)
result = _run_grounding_guard([claim], FIXTURE_CHUNKS)
assert result[0].grounded is True
# ---------------------------------------------------------------------------
# _inject_cross_refs
# ---------------------------------------------------------------------------