Skip to content

F-2391Detected Oct 4, 10:36 UTC

How much is a daily roaming pack in each zone, in euros?

Found in v3.4 · locale-formatting on locale-formatting · Roaming & travel

Retrieval mismatchMediumOpenClaude Sonnet 5.5

Model answer vs gold answer

84% of model-answer words appear in the gold answer

Model answer

support-assistant v3.4

A daily pack costs €3 in Zone 1, €4 in Zone 2 and Unsupported: €6 in Zone 3. Packs are valid for 24 hours from activation.

Gold answer

Human-verified · locale-formatting

Daily packs cost €3 (1 GB) in Zone 1, €4 (1 GB) in Zone 2 and €7 (500 MB) in Zone 3. Packs are valid for 24 hours from activation.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

2 of 3 retrieved chunks are relevant · gold passage at rank 2

  1. 1

    Roaming charges on pound and dollar bills

    Pack activation · kb-roaming-018 · v2026.09

    Relevant
    Daily packs are valid for 24 hours from activation.
    Similarity0.82
  2. 2

    Roaming packs

    Daily packs · kb-roaming-007 · v2026.09

    Relevant
    Daily packs: Zone 1 €3 (1 GB), Zone 2 €4 (1 GB), Zone 3 €7 (500 MB).
    Similarity0.80
  3. 3

    Roaming fair-use (2025 edition)

    Charges above allowance · kb-roaming-011-2025 · v2025.11

    Superseded versionNot relevant
    Zone 3 daily pack €6.
    Similarity0.70

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.44

rubric score · pass threshold 0.70

Zone 3 price comes from the superseded 2025 article (stale). Zones 1 and 2 are correct.

Diagnosis

Retrieval mismatch · Stale roaming articles

Root cause

Stale 2025 roaming article still indexed.

Suggested fix

Same as F-2412: remove superseded versions from the index.

Playbook: retrieval mismatch

The retriever returned the wrong article, the wrong section or a stale version.

Detection signals

  • High similarity on the wrong article title
  • Stale-version flag on top-ranked chunks
  • Exception clause split from its rule across chunk boundaries

Common fixes

  1. Filter or boost by effective date and locale metadata
  2. Re-chunk policy articles with heading-aware splitting
  3. Compare hybrid and dense retrieval on the mismatch subset
  4. Use parent-document retrieval so headings travel with chunks

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by support-assistant v3.4 run and labeled retrieval mismatch