Skip to content

F-2411Detected Oct 6, 15:12 UTC

Can I get a refund for the roaming pack I didn't use?

Found in support-assistant v3.4 · release candidate on support-golden-v4 · Roaming & travel

Unsupported claimsCriticalOpenClaude Sonnet 5.5
Open gt_1043

Model answer vs gold answer

13% of model-answer words appear in the gold answer

Model answer

support-assistant v3.4

Yes. Unsupported: Unused roaming packs are refunded automatically to your next bill within 7 days of expiry.

Gold answer

Human-verified · support-golden-v4

No. Roaming packs expire 7 days after activation and unused data is not carried over or refunded.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

1 of 3 retrieved chunks are relevant · gold passage at rank 3

  1. 1

    Roaming packs

    Activation and expiry · kb-roaming-007 · v2026.09

    Not relevant
    Weekly packs are valid for 7 days from activation.
    Similarity0.79
  2. 2

    Refunds to your original payment method

    Eligibility · kb-billing-037 · v2026.09

    Not relevant
    Refunds are issued to the original payment method for eligible charges.
    Similarity0.72
  3. 3

    Roaming packs

    Unused allowance · kb-roaming-007 · v2026.09

    Relevant
    Unused data in a roaming pack does not carry over and is not refunded.
    Similarity0.70

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.04

rubric score · pass threshold 0.70

The answer promises an automatic refund. No retrieved chunk supports a refund; chunk 3 explicitly states unused data is not refunded.

Diagnosis

Unsupported claims · Refund promises

Root cause

The generator merged the generic refund article with the pack expiry rule and ignored the explicit exclusion in chunk 3.

Suggested fix

Add 'not refunded' exclusions to the refusal checklist in support-prompt v7 and add this question to the regression set.

Playbook: unsupported claims

The answer states something that is not present in the retrieved context.

Detection signals

  • Claim-level faithfulness below 0.85
  • Numbers, fees or durations absent from every retrieved chunk
  • Citations that do not contain the cited sentence

Common fixes

  1. Require a citation after each factual sentence and refuse when none applies
  2. Reject answers with numeric claims missing from context in a post-generation check
  3. Lower temperature for billing and policy intents
  4. Add the query to the regression set with an escalate-or-decline gold answer

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by support-assistant v3.4 run and labeled unsupported claims