Skip to content

F-2374Detected Oct 8, 08:44 UTC

Which plan includes Disney+?

Found in v3.4 · synthetic-hard-negatives v1.3 on synthetic-hard-negatives · Plans & upgrades

Missing contextMediumOpenClaude Sonnet 5.5
Open gt_1063

Model answer vs gold answer

33% of model-answer words appear in the gold answer

Model answer

support-assistant v3.4

Unsupported: Unlimited Plus includes Disney+ Premium at no extra cost.

Gold answer

Human-verified · synthetic-hard-negatives

No current Harbor plan includes Disney+. Unlimited Plus includes a streaming credit of €12 per month that can be used with partner services listed in My Harbor.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

0 of 3 retrieved chunks are relevant · gold passage not in the top 20

  1. 1

    Changing your plan

    Upgrades · kb-plans-008 · v2026.09

    Not relevant
    Compare plans before upgrading.
    Similarity0.66
  2. 2

    Fall 2026 upgrade offer

    Discount amount · kb-promo-006 · v2026.09

    Not relevant
    Save 20% on the device upgrade.
    Similarity0.64
  3. 3

    Unlimited plan fair-use policy

    Hotspot allowance · kb-plans-019 · v2026.09

    Not relevant
    Hotspot allowance is 50 GB on Unlimited Plus.
    Similarity0.61

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.05

rubric score · pass threshold 0.70

No chunk mentions streaming services. The answer fabricates a partner bundle.

Diagnosis

Missing context · Knowledge-base gaps

Root cause

Streaming credit article not in Help Center · v2026.09 index (published 2026-10-01).

Suggested fix

Re-index the knowledge base to include articles published after 2026-09-30.

Playbook: missing context

A relevant article exists but was not retrieved, or was only partially retrieved.

Detection signals

  • Context recall below 0.7
  • Gold passage outside the top 10
  • Generic or hedged answer where gold is specific

Common fixes

  1. Add customer-phrased synonyms and regional terms to query expansion
  2. Increase k for long-tail intents or add a second retrieval pass
  3. Fill knowledge-base gaps, then re-index
  4. Escalate when no chunk passes the relevance threshold

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by support-assistant v3.4 run and labeled missing context