Skip to content

F-2406Detected Sep 25, 15:41 UTC

Someone swapped my SIM without my permission. How fast can you block it?

Found in v3.4 · account-security v1.2 on account-security · Account & security

Incorrect factual answersCriticalTriagedClaude Sonnet 5.5
Open gt_1048

Model answer vs gold answer

12% of model-answer words appear in the gold answer

Model answer

support-assistant v3.4

We can block an unauthorized SIM swap Unsupported: within 24 hours. Your line will be restored with a new SIM in Unsupported: 3 to 5 business days.

Gold answer

Human-verified · account-security

Call the fraud line or use My Harbor > Security > Report fraud; the line is blocked immediately. A replacement SIM can be issued the same day in any store with photo ID.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

1 of 3 retrieved chunks are relevant · gold passage at rank 3

  1. 1

    SIM swap protection

    In-store swaps · kb-account-005 · v2026.09

    Not relevant
    In-store SIM swaps require photo ID matching the account holder.
    Similarity0.82
  2. 2

    SIM swap protection

    How SIM swap protection works · kb-account-005 · v2026.09

    Not relevant
    You can add a 24-hour hold on swaps in My Harbor.
    Similarity0.80
  3. 3

    Reporting fraud or suspicious activity

    Blocking a line · kb-account-024 · v2026.09

    Relevant
    Reported lines are blocked immediately. Replacement SIMs are issued the same day in store.
    Similarity0.77

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.08

rubric score · pass threshold 0.70

The 24-hour figure refers to the optional swap hold, not blocking. The answer contradicts kb-account-024, which states immediate blocking and same-day replacement.

Diagnosis

Incorrect factual answers · SIM swap and fraud

Root cause

Two similar '24 hours' policies were conflated; the correct chunk ranked third.

Suggested fix

Boost kb-account-024 for fraud intents and add contrastive examples between swap hold and line blocking.

Playbook: incorrect factual answers

The answer contradicts the knowledge base, usually by misreading or conflating two policies.

Detection signals

  • Answer correctness below 0.7 while faithfulness stays high
  • Disagreement with gold on a key entity: plan, fee, date or zone
  • Reviewer edits the fact but keeps the structure

Common fixes

  1. Check for a stale article version before changing the prompt
  2. Add a structured extraction step for fees and plan limits
  3. Add contrastive examples for similarly worded plans and zones
  4. Route high-stakes billing intents to verified templates

Activity

3 events · 0 comments

  1. Relevant ·

    Detected by support-assistant v3.4 run and labeled incorrect factual answers

  2. Maya Collins ·

    Severity raised to critical: security-sensitive intent

  3. Maya Collins ·

    Triaged and assigned to self