Skip to content

F-2392Detected Oct 2, 10:12 UTC

What's the early termination fee if I cancel my 24-month contract after 10 months?

Found in v3.4 · billing-edge-cases v2.1 on billing-edge-cases · Plans & upgrades

Missing contextHighOpenClaude Sonnet 5.5
Open gt_1055

Model answer vs gold answer

50% of model-answer words appear in the gold answer

Model answer

support-assistant v3.4

You'll pay the remaining device balance only. Unsupported: There is no early termination fee on 24-month contracts.

Gold answer

Human-verified · billing-edge-cases

You pay the remaining device balance plus an early termination fee equal to 25% of the remaining monthly plan charges. Moving abroad and bereavement are exempt with documentation.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

2 of 3 retrieved chunks are relevant · gold passage at rank 4

  1. 1

    Early termination of contract plans

    Remaining device balance · kb-plans-031 · v2026.09

    Relevant
    The remaining device balance is due on cancellation.
    Similarity0.88
  2. 2

    Porting out to another operator

    Outstanding balances · kb-porting-006 · v2026.09

    Not relevant
    Outstanding balances are included in the final bill.
    Similarity0.74
  3. 3

    Early termination of contract plans

    Exemptions · kb-plans-031 · v2026.09

    Relevant
    Moving abroad and bereavement are exempt with documentation.
    Similarity0.73

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.14

rubric score · pass threshold 0.70

The fee section (rank 4) was not passed to the generator. The answer then asserts that no fee exists.

Diagnosis

Missing context · Contract termination

Root cause

Context assembly keeps 3 chunks for billing intents; the fee section lands at rank 4.

Suggested fix

Increase context to 5 chunks for contract intents; teach the model to say the fee was not found rather than deny it.

Playbook: missing context

A relevant article exists but was not retrieved, or was only partially retrieved.

Detection signals

  • Context recall below 0.7
  • Gold passage outside the top 10
  • Generic or hedged answer where gold is specific

Common fixes

  1. Add customer-phrased synonyms and regional terms to query expansion
  2. Increase k for long-tail intents or add a second retrieval pass
  3. Fill knowledge-base gaps, then re-index
  4. Escalate when no chunk passes the relevance threshold

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by support-assistant v3.4 run and labeled missing context