Skip to content

F-2385Detected Oct 6, 17:15 UTC

How many lines can I add to a family plan?

Found in v3.4 · Gemini 2.5 Pro comparison on support-golden-v4 · Plans & upgrades

Incorrect factual answersMediumOpenGemini 2.5 Pro

Model answer vs gold answer

15% of model-answer words appear in the gold answer

Model answer

v3.4 · Gemini 2.5 Pro

You can add Unsupported: up to 10 lines to a family plan, and Unsupported: each line gets its own 20 GB allowance.

Gold answer

Human-verified · support-golden-v4

A family plan supports up to 5 lines. All lines share one data pool sized by the plan.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

3 of 3 retrieved chunks are relevant · gold passage at rank 2

  1. 1

    Family plan lines and shared data

    Adding lines · kb-plans-023 · v2026.09

    Relevant
    Add lines from My Harbor > Plan > Add line.
    Similarity0.85
  2. 2

    Family plan lines and shared data

    Line limits · kb-plans-023 · v2026.09

    Relevant
    Family plans support up to 5 lines.
    Similarity0.83
  3. 3

    Family plan lines and shared data

    Shared data pool · kb-plans-023 · v2026.09

    Relevant
    All lines share one data pool.
    Similarity0.80

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.07

rubric score · pass threshold 0.70

Retrieval is complete. Both numbers contradict the context.

Diagnosis

Incorrect factual answers · Plan limits

Root cause

Generator ignored context numbers in favor of priors.

Suggested fix

Track as a model-level weakness for Gemini 2.5 Pro in the comparison report.

Playbook: incorrect factual answers

The answer contradicts the knowledge base, usually by misreading or conflating two policies.

Detection signals

  • Answer correctness below 0.7 while faithfulness stays high
  • Disagreement with gold on a key entity: plan, fee, date or zone
  • Reviewer edits the fact but keeps the structure

Common fixes

  1. Check for a stale article version before changing the prompt
  2. Add a structured extraction step for fees and plan limits
  3. Add contrastive examples for similarly worded plans and zones
  4. Route high-stakes billing intents to verified templates

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by v3.4 · Gemini 2.5 Pro run and labeled incorrect factual answers