Skip to content

F-2370Detected Oct 6, 17:38 UTC

My son is 15. Can he have his own line on my account?

Found in v3.4 · GPT-5 comparison on support-golden-v4 · Plans & upgrades

Incorrect factual answersHighOpenGPT-5
Open gt_1062

Model answer vs gold answer

33% of model-answer words appear in the gold answer

Model answer

v3.4 · GPT-5

Yes, Unsupported: customers under 18 can open their own contract in their name with parental consent.

Gold answer

Human-verified · support-golden-v4

Yes, as an additional line on your account. Contracts must be in the name of an adult account holder; the line can be registered to a user aged 12 or over with parental controls enabled by default.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

1 of 3 retrieved chunks are relevant · gold passage ranked first

  1. 1

    Family plan lines and shared data

    Adding lines · kb-plans-023 · v2026.09

    Relevant
    Add lines for family members from My Harbor.
    Similarity0.82
  2. 2

    Recovering your account

    Account holder changes · kb-account-018 · v2026.09

    Not relevant
    Account holders must be 18 or over.
    Similarity0.73
  3. 3

    Family plan lines and shared data

    Line limits · kb-plans-023 · v2026.09

    Not relevant
    Up to 5 lines per family plan.
    Similarity0.71

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.10

rubric score · pass threshold 0.70

Contradicts chunk 2 (account holders must be 18+).

Diagnosis

Incorrect factual answers · Plan limits

Root cause

Generator prior on parental-consent contracts.

Suggested fix

Add to regression set; review GPT-5 instruction adherence on policy constraints.

Playbook: incorrect factual answers

The answer contradicts the knowledge base, usually by misreading or conflating two policies.

Detection signals

  • Answer correctness below 0.7 while faithfulness stays high
  • Disagreement with gold on a key entity: plan, fee, date or zone
  • Reviewer edits the fact but keeps the structure

Common fixes

  1. Check for a stale article version before changing the prompt
  2. Add a structured extraction step for fees and plan limits
  3. Add contrastive examples for similarly worded plans and zones
  4. Route high-stakes billing intents to verified templates

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by v3.4 · GPT-5 run and labeled incorrect factual answers