Skip to content

F-2408Detected Oct 6, 16:40 UTC

My eSIM stopped working after I reset my phone. Do I need to pay for a new one?

Found in v3.4 · Gemini 2.5 Pro comparison on support-golden-v4 · eSIM & activation

Unsupported claimsHighOpenGemini 2.5 Pro
Open gt_1046

Model answer vs gold answer

42% of model-answer words appear in the gold answer

Model answer

v3.4 · Gemini 2.5 Pro

Yes. Unsupported: A replacement eSIM profile costs €5 and can be requested in My Harbor under Line settings > eSIM.

Gold answer

Human-verified · support-golden-v4

No. Request a new QR code free of charge in My Harbor under Line settings > eSIM > Request new QR code, or visit a store with photo ID.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

1 of 3 retrieved chunks are relevant · gold passage ranked first

  1. 1

    eSIM activation errors

    QR code already used · kb-esim-013 · v2026.09

    Relevant
    Each QR code works once. Request a new QR code in My Harbor or in store; there is no charge for a replacement profile.
    Similarity0.88
  2. 2

    Device insurance claims

    Excess fees · kb-returns-012 · v2026.09

    Not relevant
    An excess fee of €5 applies to accessory claims.
    Similarity0.61
  3. 3

    Activating an eSIM

    Scanning the QR code · kb-esim-004 · v2026.09

    Not relevant
    Open the camera and scan the QR code to add the eSIM.
    Similarity0.60

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.12

rubric score · pass threshold 0.70

The €5 figure comes from an unrelated insurance excess chunk. The relevant chunk explicitly says replacements are free.

Diagnosis

Unsupported claims · eSIM reactivation

Root cause

Generator attached a fee from a lower-ranked, irrelevant chunk.

Suggested fix

Drop chunks below a 0.65 relevance score for fee-related intents; add a numeric-claim grounding check.

Playbook: unsupported claims

The answer states something that is not present in the retrieved context.

Detection signals

  • Claim-level faithfulness below 0.85
  • Numbers, fees or durations absent from every retrieved chunk
  • Citations that do not contain the cited sentence

Common fixes

  1. Require a citation after each factual sentence and refuse when none applies
  2. Reject answers with numeric claims missing from context in a post-generation check
  3. Lower temperature for billing and policy intents
  4. Add the query to the regression set with an escalate-or-decline gold answer

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by v3.4 · Gemini 2.5 Pro run and labeled unsupported claims