Skip to content

F-2397Detected Oct 6, 17:02 UTC

My mobile data isn't working after updating to iOS 26.1.

Found in v3.4 · GPT-5 comparison on support-golden-v4 · Device troubleshooting

Unsupported claimsMediumTriagedGPT-5

Model answer vs gold answer

55% of model-answer words appear in the gold answer

Model answer

v3.4 · GPT-5

Go to Settings > Mobile Data > Mobile Data Network and make sure the APN is 'harbor.net'. If it is blank, reset network settings and install any carrier settings update. Restart the phone after both steps. If the issue continues, Unsupported: we will send a technician to your home within 48 hours.

Gold answer

Human-verified · support-golden-v4

Check the APN under Settings > Mobile Data > Mobile Data Network: it should be 'harbor.net'. If it is blank, reset network settings and install any pending carrier settings update. Restart the phone after both steps.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

3 of 3 retrieved chunks are relevant · gold passage ranked first

  1. 1

    No mobile data after a software update

    Check APN settings · kb-device-102 · v2026.09

    Relevant
    The APN should be harbor.net.
    Similarity0.90
  2. 2

    No mobile data after a software update

    Reset network settings · kb-device-102 · v2026.09

    Relevant
    Reset network settings, then install carrier settings updates.
    Similarity0.86
  3. 3

    No mobile data after a software update

    Carrier settings update · kb-device-102 · v2026.09

    Relevant
    Restart after updating carrier settings.
    Similarity0.84

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.52

rubric score · pass threshold 0.70

Steps are correct and grounded. The technician visit is not offered anywhere in context.

Diagnosis

Unsupported claims · Device troubleshooting escalations

Root cause

Over-helpful closing sentence added by the generator.

Suggested fix

Constrain closing lines to the approved escalation phrase list.

Playbook: unsupported claims

The answer states something that is not present in the retrieved context.

Detection signals

  • Claim-level faithfulness below 0.85
  • Numbers, fees or durations absent from every retrieved chunk
  • Citations that do not contain the cited sentence

Common fixes

  1. Require a citation after each factual sentence and refuse when none applies
  2. Reject answers with numeric claims missing from context in a post-generation check
  3. Lower temperature for billing and policy intents
  4. Add the query to the regression set with an escalate-or-decline gold answer

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by v3.4 · GPT-5 run and labeled unsupported claims