Skip to content

F-2372Detected Oct 2, 02:32 UTC

What does 'VAT' mean on my bill?

Found in Nightly · v3.4 · 2026-10-02 on support-golden-v4 · Billing & payments

OtherLowOpenClaude Sonnet 5.5

Model answer vs gold answer

82% of model-answer words appear in the gold answer

Model answer

support-assistant v3.4

Unsupported: VAT is value-added tax. On UK-billed accounts it is charged at

Gold answer

Human-verified · support-golden-v4

VAT is value-added tax. It is charged at 20% on UK-billed accounts and at the standard rate of your billing country on EU-billed accounts, and it is shown as a separate line on every bill.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

2 of 3 retrieved chunks are relevant · gold passage ranked first

  1. 1

    Understanding your monthly bill

    Taxes and fees · kb-billing-014 · v2026.09

    Relevant
    VAT is shown as a separate line at the standard rate of the billing country.
    Similarity0.90
  2. 2

    Currency, VAT and date formats on your bill

    VAT by billing country · kb-billing-016 · v2026.09

    Relevant
    UK-billed accounts show VAT at 20%; EU-billed accounts show the standard rate of the billing country.
    Similarity0.87
  3. 3

    Understanding your monthly bill

    Bill structure · kb-billing-014 · v2026.09

    Not relevant
    Bills are issued monthly.
    Similarity0.75

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.25

rubric score · pass threshold 0.70

Answer truncated mid-sentence; the generator stopped after the max-token stop sequence matched '%'.

Diagnosis

Other · Truncated answers

Root cause

A stop sequence containing '%' cut the answer.

Suggested fix

Remove '%' from stop sequences in the generation config.

Playbook: other

Over-refusal, wrong language without a prompt constraint, or truncated answers.

Detection signals

  • Refusal phrases on answerable items
  • Language detector mismatch
  • Output stops mid-sentence

Common fixes

  1. Relax refusal thresholds for intents with strong retrieval
  2. Raise max output tokens for multi-step procedures
  3. Tag the locale in retrieval filters

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by support-assistant v3.4 run and labeled other