Skip to content

F-2398Detected Sep 21, 02:28 UTC

Can I return my phone after 20 days if I don't like it?

Found in Nightly · v3.4 · 2026-09-21 on support-golden-v4 · Returns & warranty

Incorrect factual answersMediumResolvedClaude Sonnet 5.5

Model answer vs gold answer

7% of model-answer words appear in the gold answer

Model answer

support-assistant v3.4

Yes, phones can be returned Unsupported: within 30 days if they are in as-new condition.

Gold answer

Human-verified · support-golden-v4

The return window is 14 days from delivery. After 14 days, a faulty device is handled under warranty, but change-of-mind returns are not accepted.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

2 of 3 retrieved chunks are relevant · gold passage ranked first

  1. 1

    Returning a device

    14-day return window · kb-returns-003 · v2026.09

    Relevant
    You can return a device within 14 days of delivery.
    Similarity0.87
  2. 2

    Requesting a copy of your data

    Response time · kb-privacy-002 · v2026.09

    Not relevant
    We respond to data requests within 30 days.
    Similarity0.62
  3. 3

    Device warranty

    Warranty period · kb-returns-009 · v2026.09

    Relevant
    Devices have a 2-year warranty against manufacturing defects.
    Similarity0.61

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.11

rubric score · pass threshold 0.70

The return window in chunk 1 is 14 days. The answer states 30 days, a figure that appears only in an unrelated privacy chunk.

Diagnosis

Incorrect factual answers · Return window

Root cause

Numeric bleed from an unrelated chunk.

Suggested fix

Resolved by prompt v7 citation rule: each number must cite its source chunk.

Playbook: incorrect factual answers

The answer contradicts the knowledge base, usually by misreading or conflating two policies.

Detection signals

  • Answer correctness below 0.7 while faithfulness stays high
  • Disagreement with gold on a key entity: plan, fee, date or zone
  • Reviewer edits the fact but keeps the structure

Common fixes

  1. Check for a stale article version before changing the prompt
  2. Add a structured extraction step for fees and plan limits
  3. Add contrastive examples for similarly worded plans and zones
  4. Route high-stakes billing intents to verified templates

Activity

3 events · 0 comments

  1. Relevant ·

    Detected by support-assistant v3.4 run and labeled incorrect factual answers

  2. Daniel Okafor ·

    Triaged and assigned to self

  3. Relevant ·

    Resolved: passes on run_2026-10-06_v3-4-rc