Skip to content

F-2377Detected Sep 28, 11:38 UTC

How do I stop satellite roaming charges on a flight?

Found in v3.4 · roaming-and-travel v1.4 on roaming-and-travel · Roaming & travel

Missing contextLowResolvedClaude Sonnet 5.5

Model answer vs gold answer

57% of model-answer words appear in the gold answer

Model answer

support-assistant v3.4

Turn on airplane mode during the flight.

Gold answer

Human-verified · roaming-and-travel

Turn on Block satellite roaming in My Harbor under Roaming settings before the flight. In-flight networks are charged at €9.99 per MB and are outside every roaming zone. Airplane mode with Wi-Fi only also avoids charges.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

1 of 3 retrieved chunks are relevant · gold passage ranked first

  1. 1

    Using your phone on a cruise or flight

    In-flight networks · kb-roaming-015 · v2026.09

    Relevant
    In-flight mobile networks are charged at €9.99/MB.
    Similarity0.82
  2. 2

    Roaming packs

    Activation and expiry · kb-roaming-007 · v2026.09

    Not relevant
    Packs activate when you first use data abroad.
    Similarity0.69
  3. 3

    Roaming zones and countries

    Rest of world · kb-roaming-003 · v2026.09

    Not relevant
    Rest-of-world rates apply outside Zones 1 to 3.
    Similarity0.68

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.50

rubric score · pass threshold 0.70

Partially correct. The Block satellite roaming section was not retrieved.

Diagnosis

Missing context · Maritime and in-flight roaming

Root cause

Section was split from the in-flight pricing paragraph by 512-token chunking.

Suggested fix

Resolved by v3 semantic chunking in later nightly runs.

Playbook: missing context

A relevant article exists but was not retrieved, or was only partially retrieved.

Detection signals

  • Context recall below 0.7
  • Gold passage outside the top 10
  • Generic or hedged answer where gold is specific

Common fixes

  1. Add customer-phrased synonyms and regional terms to query expansion
  2. Increase k for long-tail intents or add a second retrieval pass
  3. Fill knowledge-base gaps, then re-index
  4. Escalate when no chunk passes the relevance threshold

Activity

3 events · 0 comments

  1. Relevant ·

    Detected by support-assistant v3.4 run and labeled missing context

  2. Daniel Okafor ·

    Triaged and assigned to self

  3. Relevant ·

    Resolved: passes on run_2026-10-03_nightly