Skip to content

F-2376Detected Oct 2, 10:27 UTC

Is there a fee to change my billing date?

Found in v3.4 · billing-edge-cases v2.1 on billing-edge-cases · Billing & payments

Incorrect factual answersLowOpenClaude Sonnet 5.5
Open gt_1069

Model answer vs gold answer

19% of model-answer words appear in the gold answer

Model answer

support-assistant v3.4

Changing your billing date is free, and Unsupported: you can do it as often as you like.

Gold answer

Human-verified · billing-edge-cases

There is no fee, but you can change your billing date once every 6 months. The next bill is pro-rated for the days between the old and new dates.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

2 of 3 retrieved chunks are relevant · gold passage ranked first

  1. 1

    Changing your billing date

    Who can change it · kb-billing-045 · v2026.09

    Relevant
    Account holders can change the billing date once every 6 months.
    Similarity0.86
  2. 2

    Changing your billing date

    How the first bill is pro-rated · kb-billing-045 · v2026.09

    Relevant
    The next bill is pro-rated.
    Similarity0.84
  3. 3

    Understanding your monthly bill

    One-off charges · kb-billing-014 · v2026.09

    Not relevant
    One-off charges appear on your next bill.
    Similarity0.70

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.46

rubric score · pass threshold 0.70

The 6-month limit in chunk 1 is contradicted.

Diagnosis

Incorrect factual answers · Billing date changes

Root cause

Model generalized 'free' to 'unlimited'.

Suggested fix

Add contrastive example; monitor in billing-edge-cases.

Playbook: incorrect factual answers

The answer contradicts the knowledge base, usually by misreading or conflating two policies.

Detection signals

  • Answer correctness below 0.7 while faithfulness stays high
  • Disagreement with gold on a key entity: plan, fee, date or zone
  • Reviewer edits the fact but keeps the structure

Common fixes

  1. Check for a stale article version before changing the prompt
  2. Add a structured extraction step for fees and plan limits
  3. Add contrastive examples for similarly worded plans and zones
  4. Route high-stakes billing intents to verified templates

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by support-assistant v3.4 run and labeled incorrect factual answers