Skip to content

F-2403Detected Oct 7, 02:24 UTC

Why was I charged twice this month?

Found in Nightly · v3.4 · 2026-10-07 on support-golden-v4 · Billing & payments

Unsupported claimsHighOpenClaude Sonnet 5.5
Open gt_1051

Model answer vs gold answer

50% of model-answer words appear in the gold answer

Model answer

support-assistant v3.4

A duplicate charge is usually a pending authorization. It will be reversed within 5 business days. If not, Unsupported: we'll refund it with a €2.50 goodwill credit.

Gold answer

Human-verified · support-golden-v4

A second charge is usually a pending card authorization. Pending authorizations drop off within 5 business days. If both are still showing after 5 business days, contact Billing support with the transaction dates so we can refund the duplicate.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

3 of 3 retrieved chunks are relevant · gold passage ranked first

  1. 1

    Duplicate or unexpected charges

    Pending authorizations · kb-billing-022 · v2026.09

    Relevant
    A pending authorization can look like a second charge.
    Similarity0.91
  2. 2

    Duplicate or unexpected charges

    Reversal timelines · kb-billing-022 · v2026.09

    Relevant
    Pending authorizations drop off within 5 business days.
    Similarity0.88
  3. 3

    Duplicate or unexpected charges

    When to contact us · kb-billing-022 · v2026.09

    Relevant
    If both charges remain after 5 business days, contact Billing support.
    Similarity0.79

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.35

rubric score · pass threshold 0.70

Retrieval is perfect. The goodwill credit is not mentioned in any chunk and is a commitment only agents can make.

Diagnosis

Unsupported claims · Refund promises

Root cause

The model adds compensation language learned from general support style.

Suggested fix

Add 'goodwill credit' and 'compensation' to the forbidden-promise list.

Playbook: unsupported claims

The answer states something that is not present in the retrieved context.

Detection signals

  • Claim-level faithfulness below 0.85
  • Numbers, fees or durations absent from every retrieved chunk
  • Citations that do not contain the cited sentence

Common fixes

  1. Require a citation after each factual sentence and refuse when none applies
  2. Reject answers with numeric claims missing from context in a post-generation check
  3. Lower temperature for billing and policy intents
  4. Add the query to the regression set with an escalate-or-decline gold answer

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by support-assistant v3.4 run and labeled unsupported claims