Skip to content

F-2404Detected Sep 28, 11:22 UTC

Do I need a roaming pack for a weekend in Zurich?

Found in v3.4 · roaming-and-travel v1.4 on roaming-and-travel · Roaming & travel

Incorrect factual answersHighOpenClaude Sonnet 5.5
Open gt_1050

Model answer vs gold answer

33% of model-answer words appear in the gold answer

Model answer

support-assistant v3.4

No. Unsupported: Switzerland is in Zone 1 with the EU, so your plan's roaming allowance applies.

Gold answer

Human-verified · roaming-and-travel

Yes. Switzerland is in Zone 2 (UK, Switzerland, Balkans). Roaming is not included for Zone 2; add a daily (€4, 1 GB) or weekly (€15, 5 GB) Zone 2 pack.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

1 of 3 retrieved chunks are relevant · gold passage at rank 3

  1. 1

    Roaming zones and countries

    Zone 1: EU and EEA · kb-roaming-003 · v2026.09

    Not relevant
    Zone 1 covers the EU member states plus Iceland, Liechtenstein and Norway.
    Similarity0.84
  2. 2

    Roaming packs

    Daily packs · kb-roaming-007 · v2026.09

    Not relevant
    Daily packs are available for Zones 1 to 3.
    Similarity0.73
  3. 3

    Roaming zones and countries

    Zone 2: UK, Switzerland, Balkans · kb-roaming-003 · v2026.09

    Relevant
    Zone 2: United Kingdom, Switzerland, Serbia, Bosnia and Herzegovina, North Macedonia, Albania, Montenegro.
    Similarity0.72

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.05

rubric score · pass threshold 0.70

Switzerland is listed in Zone 2 in chunk 3. The answer asserts the opposite.

Diagnosis

Incorrect factual answers · Roaming zone lookup

Root cause

Generator relied on world knowledge (Schengen) over the retrieved zone list.

Suggested fix

Add a zone lookup table as structured context for country questions.

Playbook: incorrect factual answers

The answer contradicts the knowledge base, usually by misreading or conflating two policies.

Detection signals

  • Answer correctness below 0.7 while faithfulness stays high
  • Disagreement with gold on a key entity: plan, fee, date or zone
  • Reviewer edits the fact but keeps the structure

Common fixes

  1. Check for a stale article version before changing the prompt
  2. Add a structured extraction step for fees and plan limits
  3. Add contrastive examples for similarly worded plans and zones
  4. Route high-stakes billing intents to verified templates

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by support-assistant v3.4 run and labeled incorrect factual answers