Skip to content

F-2381Detected Oct 6, 15:48 UTC

Can I change the billing email after my contract starts?

Found in support-assistant v3.4 · release candidate on support-golden-v4 · Billing & payments

Retrieval mismatchMediumOpenClaude Sonnet 5.5
Open gt_1059

Model answer vs gold answer

33% of model-answer words appear in the gold answer

Model answer

support-assistant v3.4

Yes. Unsupported: You can edit it under Settings > Team in My Harbor.

Gold answer

Human-verified · support-golden-v4

Yes. The account holder can change the billing email in My Harbor under Billing > Billing contact. The change applies from the next bill.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

0 of 3 retrieved chunks are relevant · gold passage at rank 14

  1. 1

    Understanding your monthly bill

    Bill structure · kb-billing-014 · v2026.09

    Not relevant
    Your bill is emailed to the billing contact each month.
    Similarity0.81
  2. 2

    Recovering your account

    Account holder changes · kb-account-018 · v2026.09

    Not relevant
    Transfer of ownership requires both parties.
    Similarity0.74
  3. 3

    Early termination of contract plans

    Early termination fee · kb-plans-031 · v2026.09

    Not relevant
    Early termination fees apply to contract plans.
    Similarity0.69

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.12

rubric score · pass threshold 0.70

None of the retrieved chunks covers changing the billing contact; the gold section ranks 14th. The navigation path in the answer does not exist.

Diagnosis

Retrieval mismatch · Billing contact

Root cause

The 'Update billing contact' section is titled with product UI terms that do not match customer phrasing.

Suggested fix

Add customer-phrased aliases (billing email, invoice email) to the section metadata.

Playbook: retrieval mismatch

The retriever returned the wrong article, the wrong section or a stale version.

Detection signals

  • High similarity on the wrong article title
  • Stale-version flag on top-ranked chunks
  • Exception clause split from its rule across chunk boundaries

Common fixes

  1. Filter or boost by effective date and locale metadata
  2. Re-chunk policy articles with heading-aware splitting
  3. Compare hybrid and dense retrieval on the mismatch subset
  4. Use parent-document retrieval so headings travel with chunks

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by support-assistant v3.4 run and labeled retrieval mismatch