Skip to content

F-2393Detected Oct 6, 17:10 UTC

Please answer in two sentences: when is my bill due?

Found in v3.4 · GPT-5 comparison on support-golden-v4 · Billing & payments

Instruction-following failuresLowOpenGPT-5

Model answer vs gold answer

42% of model-answer words appear in the gold answer

Model answer

v3.4 · GPT-5

Your bill is due 14 days after it is issued. You can see the exact due date on the first page of your bill in My Harbor. Unsupported: You can also set up direct debit so you never miss a payment, and if you pay late a fee may apply after a grace period.

Gold answer

Human-verified · support-golden-v4

Your bill is due 14 days after it is issued. The due date is shown on the first page of the bill in My Harbor.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

1 of 3 retrieved chunks are relevant · gold passage ranked first

  1. 1

    Understanding your monthly bill

    Bill structure · kb-billing-014 · v2026.09

    Relevant
    Bills are due 14 days after issue; the due date is on page 1.
    Similarity0.86
  2. 2

    Paying with direct debit and saved cards

    Supported methods · kb-billing-052 · v2026.09

    Not relevant
    Pay by direct debit or saved card.
    Similarity0.72
  3. 3

    Late payment fees and service restrictions

    Grace period · kb-billing-031 · v2026.09

    Not relevant
    There is a 10-day grace period after the due date.
    Similarity0.70

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.62

rubric score · pass threshold 0.70

Content is grounded but the customer asked for two sentences; the answer has three.

Diagnosis

Instruction-following failures · Format and length constraints

Root cause

User-specified length constraints are not checked.

Suggested fix

Add a rule check for explicit sentence or word limits in the user turn.

Playbook: instruction-following failures

The answer ignored format, tone, policy or language constraints from the system prompt.

Detection signals

  • Rule check fails: length, language, required closing line
  • Promises that policy reserves for agents, such as guaranteed refunds
  • Judge flags tone or format

Common fixes

  1. Move hard constraints into automated rule checks
  2. Detect the customer's language and enforce it
  3. Maintain a policy-violation phrase list with support leads
  4. Gate prompt versions on the instruction-adherence subset

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by v3.4 · GPT-5 run and labeled instruction-following failures