Skip to content

F-2375Detected Oct 6, 17:31 UTC

Format the steps as a numbered list: how do I activate an eSIM?

Found in v3.4 · Gemini 2.5 Pro comparison on support-golden-v4 · eSIM & activation

Instruction-following failuresLowOpenGemini 2.5 Pro

Model answer vs gold answer

63% of model-answer words appear in the gold answer

Model answer

v3.4 · Gemini 2.5 Pro

Unsupported: To activate an eSIM, make sure your phone is compatible, then scan the QR code from your welcome email with the camera, and wait up to 10 minutes for activation to complete.

Gold answer

Human-verified · support-golden-v4

1. Check your phone is on the compatible devices list. 2. Open the camera and scan the QR code from your welcome email. 3. Follow the prompts to add the eSIM. 4. Wait up to 10 minutes for activation.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

3 of 3 retrieved chunks are relevant · gold passage ranked first

  1. 1

    Activating an eSIM

    Compatible devices · kb-esim-004 · v2026.09

    Relevant
    Check the compatible devices list.
    Similarity0.88
  2. 2

    Activating an eSIM

    Scanning the QR code · kb-esim-004 · v2026.09

    Relevant
    Scan the QR code with the camera.
    Similarity0.87
  3. 3

    Activating an eSIM

    Activation times · kb-esim-004 · v2026.09

    Relevant
    Activation can take up to 10 minutes.
    Similarity0.85

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.60

rubric score · pass threshold 0.70

Content correct; requested numbered-list format ignored.

Diagnosis

Instruction-following failures · Format and length constraints

Root cause

Format instructions in the user turn not enforced.

Suggested fix

Add format rule check (numbered list requested) to the instruction-adherence suite.

Playbook: instruction-following failures

The answer ignored format, tone, policy or language constraints from the system prompt.

Detection signals

  • Rule check fails: length, language, required closing line
  • Promises that policy reserves for agents, such as guaranteed refunds
  • Judge flags tone or format

Common fixes

  1. Move hard constraints into automated rule checks
  2. Detect the customer's language and enforce it
  3. Maintain a policy-violation phrase list with support leads
  4. Gate prompt versions on the instruction-adherence subset

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by v3.4 · Gemini 2.5 Pro run and labeled instruction-following failures