Skip to content

F-2395Detected Oct 6, 16:52 UTC

How do I turn on two-factor authentication for My Harbor?

Found in v3.4 · Gemini 2.5 Pro comparison on support-golden-v4 · Account & security

Unsupported claimsCriticalOpenGemini 2.5 Pro
Open gt_1053

Model answer vs gold answer

54% of model-answer words appear in the gold answer

Model answer

v3.4 · Gemini 2.5 Pro

Go to My Harbor > Security > Two-factor authentication and choose an authenticator app or SMS. Save your backup codes somewhere safe. Note: Unsupported: Harbor support can disable 2FA for you over chat if you lose your phone.

Gold answer

Human-verified · support-golden-v4

Go to My Harbor > Security > Two-factor authentication and choose an authenticator app or SMS. Save your backup codes. If you lose your device, use a backup code or complete identity verification through account recovery.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

3 of 3 retrieved chunks are relevant · gold passage ranked first

  1. 1

    Two-factor authentication for My Harbor

    Turning on 2FA · kb-account-011 · v2026.09

    Relevant
    Open My Harbor > Security > Two-factor authentication.
    Similarity0.92
  2. 2

    Two-factor authentication for My Harbor

    Backup codes · kb-account-011 · v2026.09

    Relevant
    Save your backup codes when you turn on 2FA.
    Similarity0.87
  3. 3

    Two-factor authentication for My Harbor

    Lost device · kb-account-011 · v2026.09

    Relevant
    Without backup codes, complete account recovery. 2FA cannot be removed over chat.
    Similarity0.83

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.10

rubric score · pass threshold 0.70

The final sentence contradicts chunk 3 and describes a social-engineering path the policy forbids.

Diagnosis

Unsupported claims · SIM swap and fraud

Root cause

Generator hallucinated a support shortcut on a security intent.

Suggested fix

Block 'disable 2FA' phrasing in output filters; add security intents to the strict-grounding route.

Playbook: unsupported claims

The answer states something that is not present in the retrieved context.

Detection signals

  • Claim-level faithfulness below 0.85
  • Numbers, fees or durations absent from every retrieved chunk
  • Citations that do not contain the cited sentence

Common fixes

  1. Require a citation after each factual sentence and refuse when none applies
  2. Reject answers with numeric claims missing from context in a post-generation check
  3. Lower temperature for billing and policy intents
  4. Add the query to the regression set with an escalate-or-decline gold answer

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by v3.4 · Gemini 2.5 Pro run and labeled unsupported claims