Skip to content

F-2405Detected Oct 6, 15:20 UTC

How long does it take to port my number from another operator?

Found in support-assistant v3.4 · release candidate on support-golden-v4 · Number porting

Incorrect factual answersHighOpenClaude Sonnet 5.5
Open gt_1049

Model answer vs gold answer

21% of model-answer words appear in the gold answer

Model answer

support-assistant v3.4

Number porting takes Unsupported: 3 to 5 business days once we receive your porting code.

Gold answer

Human-verified · support-golden-v4

Once we receive a valid porting code, number transfers complete within 1 business day. Keep your old SIM active until your Harbor SIM shows signal.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

1 of 3 retrieved chunks are relevant · gold passage at rank 3

  1. 1

    Porting out to another operator

    Final bill · kb-porting-006 · v2026.09

    Not relevant
    Your final bill from Harbor is issued within 5 business days of porting out.
    Similarity0.78
  2. 2

    Bringing your number to Harbor

    Porting code · kb-porting-002 · v2026.09

    Not relevant
    Request a porting code from your current operator by text.
    Similarity0.77
  3. 3

    Bringing your number to Harbor

    Timeline · kb-porting-002 · v2026.09

    Relevant
    Transfers complete within 1 business day of receiving a valid code.
    Similarity0.75

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.15

rubric score · pass threshold 0.70

The timeline was taken from the port-out final-bill chunk. The correct port-in timeline (1 business day) is in chunk 3.

Diagnosis

Incorrect factual answers · Porting direction confusion

Root cause

Port-in and port-out articles share vocabulary; the port-out chunk ranks first.

Suggested fix

Add direction metadata (in/out) to porting articles and filter on the classified intent.

Playbook: incorrect factual answers

The answer contradicts the knowledge base, usually by misreading or conflating two policies.

Detection signals

  • Answer correctness below 0.7 while faithfulness stays high
  • Disagreement with gold on a key entity: plan, fee, date or zone
  • Reviewer edits the fact but keeps the structure

Common fixes

  1. Check for a stale article version before changing the prompt
  2. Add a structured extraction step for fees and plan limits
  3. Add contrastive examples for similarly worded plans and zones
  4. Route high-stakes billing intents to verified templates

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by support-assistant v3.4 run and labeled incorrect factual answers