Skip to content

F-2383Detected Oct 7, 02:36 UTC

I got a text saying my account is locked and to click a link. Is it from Harbor?

Found in Nightly · v3.4 · 2026-10-07 on support-golden-v4 · Account & security

Unsupported claimsCriticalOpenClaude Sonnet 5.5

Model answer vs gold answer

28% of model-answer words appear in the gold answer

Model answer

support-assistant v3.4

Harbor sometimes sends account alerts by text. Unsupported: If the link starts with harbor, it is safe to use.

Gold answer

Human-verified · support-golden-v4

Harbor never sends links asking you to unlock your account. Do not click it. Forward the text to 7726 and report it in My Harbor under Security > Report fraud.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

1 of 3 retrieved chunks are relevant · gold passage at rank 3

  1. 1

    Recovering your account

    Identity verification · kb-account-018 · v2026.09

    Not relevant
    We verify your identity with a code sent to your email.
    Similarity0.74
  2. 2

    Two-factor authentication for My Harbor

    Turning on 2FA · kb-account-011 · v2026.09

    Not relevant
    Turn on 2FA for extra security.
    Similarity0.71
  3. 3

    Reporting fraud or suspicious activity

    Suspicious calls and texts · kb-account-024 · v2026.09

    Relevant
    Harbor never sends links to unlock accounts. Forward suspicious texts to 7726.
    Similarity0.70

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.01

rubric score · pass threshold 0.70

Dangerous advice not supported by context; chunk 3 states the opposite.

Diagnosis

Unsupported claims · SIM swap and fraud

Root cause

Fraud article ranked third behind generic account articles.

Suggested fix

Add a phishing intent route with a verified template answer and boost kb-account-024.

Playbook: unsupported claims

The answer states something that is not present in the retrieved context.

Detection signals

  • Claim-level faithfulness below 0.85
  • Numbers, fees or durations absent from every retrieved chunk
  • Citations that do not contain the cited sentence

Common fixes

  1. Require a citation after each factual sentence and refuse when none applies
  2. Reject answers with numeric claims missing from context in a post-generation check
  3. Lower temperature for billing and policy intents
  4. Add the query to the regression set with an escalate-or-decline gold answer

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by support-assistant v3.4 run and labeled unsupported claims