Skip to content

F-2378Detected Oct 1, 02:27 UTC

Does my insurance cover a stolen phone?

Found in Nightly · v3.4 · 2026-10-01 on support-golden-v4 · Returns & warranty

Unsupported claimsHighOpenClaude Sonnet 5.5
Open gt_1060

Model answer vs gold answer

56% of model-answer words appear in the gold answer

Model answer

support-assistant v3.4

Yes, theft is covered Unsupported: with no excess fee, and you'll receive a replacement Unsupported: within 24 hours.

Gold answer

Human-verified · support-golden-v4

Yes. Theft is covered with an excess fee of €125 once you provide a police report number. Replacements are dispatched within 2 business days of claim approval.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

2 of 3 retrieved chunks are relevant · gold passage ranked first

  1. 1

    Device insurance claims

    Excess fees · kb-returns-012 · v2026.09

    Relevant
    Theft and loss claims have an excess of €125.
    Similarity0.83
  2. 2

    Device insurance claims

    Replacement timeline · kb-returns-012 · v2026.09

    Relevant
    Replacements are dispatched within 2 business days of approval.
    Similarity0.80
  3. 3

    Device insurance claims

    Claim limits · kb-returns-012 · v2026.09

    Not relevant
    Up to 2 claims per year.
    Similarity0.76

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.09

rubric score · pass threshold 0.70

Both the excess and the timeline contradict the retrieved chunks.

Diagnosis

Unsupported claims · Refund promises

Root cause

Generator ignored two explicit numeric facts in top-ranked chunks.

Suggested fix

Add numeric-claim verification against context before responding.

Playbook: unsupported claims

The answer states something that is not present in the retrieved context.

Detection signals

  • Claim-level faithfulness below 0.85
  • Numbers, fees or durations absent from every retrieved chunk
  • Citations that do not contain the cited sentence

Common fixes

  1. Require a citation after each factual sentence and refuse when none applies
  2. Reject answers with numeric claims missing from context in a post-generation check
  3. Lower temperature for billing and policy intents
  4. Add the query to the regression set with an escalate-or-decline gold answer

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by support-assistant v3.4 run and labeled unsupported claims