Skip to content

F-2396Detected Oct 6, 15:41 UTC

Does the Fall 2026 upgrade offer apply if I'm on a family plan?

Found in support-assistant v3.4 · release candidate on support-golden-v4 · Promotions

Retrieval mismatchHighOpenClaude Sonnet 5.5
Open gt_1057

Model answer vs gold answer

50% of model-answer words appear in the gold answer

Model answer

support-assistant v3.4

Yes, the Fall 2026 upgrade offer Unsupported: stacks with your family plan discount, giving you both discounts.

Gold answer

Human-verified · support-golden-v4

No. The Fall 2026 upgrade offer cannot be combined with family plan line discounts. If both apply, the larger discount is used automatically.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

1 of 3 retrieved chunks are relevant · gold passage at rank 3

  1. 1

    Refer a friend

    Limits · kb-promo-010 · v2026.09

    Not relevant
    Refer-a-friend credit can be combined with other offers.
    Similarity0.81
  2. 2

    Fall 2026 upgrade offer

    Eligibility · kb-promo-006 · v2026.09

    Not relevant
    The offer is open to customers 12 months into a contract.
    Similarity0.80
  3. 3

    Fall 2026 upgrade offer

    Combining offers · kb-promo-006 · v2026.09

    Relevant
    This offer cannot be combined with family line discounts.
    Similarity0.79

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.09

rubric score · pass threshold 0.70

The top chunk is about refer-a-friend combinations, a different promotion. The correct rule is at rank 3 and says the opposite.

Diagnosis

Retrieval mismatch · Promotion stacking

Root cause

Near-duplicate wording across promotion articles; the wrong promotion ranks first.

Suggested fix

Add promotion name as a required keyword match in hybrid retrieval; evaluate on synthetic-hard-negatives.

Playbook: retrieval mismatch

The retriever returned the wrong article, the wrong section or a stale version.

Detection signals

  • High similarity on the wrong article title
  • Stale-version flag on top-ranked chunks
  • Exception clause split from its rule across chunk boundaries

Common fixes

  1. Filter or boost by effective date and locale metadata
  2. Re-chunk policy articles with heading-aware splitting
  3. Compare hybrid and dense retrieval on the mismatch subset
  4. Use parent-document retrieval so headings travel with chunks

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by support-assistant v3.4 run and labeled retrieval mismatch