Skip to content

F-2402Detected Oct 6, 15:26 UTC

Can I keep my old number if I move from prepaid to a contract plan?

Found in support-assistant v3.4 · release candidate on support-golden-v4 · Plans & upgrades

Incorrect factual answersMediumOpenClaude Sonnet 5.5

Model answer vs gold answer

48% of model-answer words appear in the gold answer

Model answer

support-assistant v3.4

Yes, you can keep your number when moving from prepaid to a contract plan. Unsupported: The change takes effect at your next billing date.

Gold answer

Human-verified · support-golden-v4

Yes. Moving from prepaid to a contract keeps your number. The change takes effect immediately and any remaining prepaid credit is applied to your first bill.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

1 of 3 retrieved chunks are relevant · gold passage at rank 2

  1. 1

    Changing your plan

    Downgrades · kb-plans-008 · v2026.09

    Not relevant
    Downgrades take effect at the start of your next billing cycle.
    Similarity0.77
  2. 2

    Changing your plan

    Contract plans · kb-plans-008 · v2026.09

    Relevant
    Prepaid customers moving to a contract keep their number. The change is immediate; remaining credit is applied to the first bill.
    Similarity0.76
  3. 3

    Changing your plan

    Effective dates · kb-plans-008 · v2026.09

    Not relevant
    Plan changes made in My Harbor show their effective date before you confirm.
    Similarity0.70

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.48

rubric score · pass threshold 0.70

Number retention is correct, but the effective date was taken from the downgrade rule. The answer also omits the credit carry-over.

Diagnosis

Incorrect factual answers · Plan change proration

Root cause

Adjacent sections in the same article; the downgrade section ranked above the contract section.

Suggested fix

Heading-aware chunking keeps section titles in chunk text; add a section-match boost.

Playbook: incorrect factual answers

The answer contradicts the knowledge base, usually by misreading or conflating two policies.

Detection signals

  • Answer correctness below 0.7 while faithfulness stays high
  • Disagreement with gold on a key entity: plan, fee, date or zone
  • Reviewer edits the fact but keeps the structure

Common fixes

  1. Check for a stale article version before changing the prompt
  2. Add a structured extraction step for fees and plan limits
  3. Add contrastive examples for similarly worded plans and zones
  4. Route high-stakes billing intents to verified templates

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by support-assistant v3.4 run and labeled incorrect factual answers