Skip to content

F-2382Detected Oct 4, 10:42 UTC

I bought my phone from Harbor in the UK. Is a cracked screen repaired for free under the warranty?

Found in v3.4 · locale-formatting on locale-formatting · Returns & warranty

Incorrect factual answersHighOpenClaude Sonnet 5.5
Open gt_1058

Model answer vs gold answer

8% of model-answer words appear in the gold answer

Model answer

support-assistant v3.4

Yes, Unsupported: cracked screens are repaired free of charge under the 2-year warranty.

Gold answer

Human-verified · locale-formatting

No. The warranty covers manufacturing defects only; physical damage such as a cracked screen is not covered. If you have device insurance, you can file an insurance claim instead.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

2 of 3 retrieved chunks are relevant · gold passage ranked first

  1. 1

    Device warranty

    Warranty period · kb-returns-009 · v2026.09

    Relevant
    Devices have a 2-year warranty against manufacturing defects.
    Similarity0.84
  2. 2

    Device insurance claims

    Claim limits · kb-returns-012 · v2026.09

    Not relevant
    Insurance covers up to 2 claims per year.
    Similarity0.73
  3. 3

    Device warranty

    What is not covered · kb-returns-009 · v2026.09

    Relevant
    Accidental damage, including cracked screens and liquid damage, is not covered.
    Similarity0.72

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.04

rubric score · pass threshold 0.70

Chunk 3 explicitly excludes cracked screens. The answer asserts free repair.

Diagnosis

Incorrect factual answers · Warranty exclusions

Root cause

The generator anchored on the 2-year figure in chunk 1 and dropped the exclusion clause in chunk 3; the regional framing of the question pulled attention away from coverage rules.

Suggested fix

Add a coverage-exclusion check to support-prompt v7 for warranty intents and add this question to the regression set.

Playbook: incorrect factual answers

The answer contradicts the knowledge base, usually by misreading or conflating two policies.

Detection signals

  • Answer correctness below 0.7 while faithfulness stays high
  • Disagreement with gold on a key entity: plan, fee, date or zone
  • Reviewer edits the fact but keeps the structure

Common fixes

  1. Check for a stale article version before changing the prompt
  2. Add a structured extraction step for fees and plan limits
  3. Add contrastive examples for similarly worded plans and zones
  4. Route high-stakes billing intents to verified templates

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by support-assistant v3.4 run and labeled incorrect factual answers