Skip to content

F-2369Detected Oct 6, 17:44 UTC

Can I get a refund if I cancel within the first month?

Found in v3.4 · Gemini 2.5 Pro comparison on support-golden-v4 · Billing & payments

Instruction-following failuresCriticalOpenGemini 2.5 Pro
Open gt_1061

Model answer vs gold answer

13% of model-answer words appear in the gold answer

Model answer

v3.4 · Gemini 2.5 Pro

Yes, Unsupported: cancel within 30 days and we refund your first month in full, no questions asked.

Gold answer

Human-verified · support-golden-v4

Within 14 days of starting a contract you can cancel under the cooling-off period and pay only for usage. After 14 days, early termination terms apply.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

2 of 3 retrieved chunks are relevant · gold passage ranked first

  1. 1

    Refunds to your original payment method

    Eligibility · kb-billing-037 · v2026.09

    Relevant
    Cooling-off cancellations within 14 days are charged for usage only.
    Similarity0.84
  2. 2

    Early termination of contract plans

    Early termination fee · kb-plans-031 · v2026.09

    Relevant
    After the cooling-off period, early termination terms apply.
    Similarity0.78
  3. 3

    Requesting a copy of your data

    Response time · kb-privacy-002 · v2026.09

    Not relevant
    We respond within 30 days.
    Similarity0.60

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.02

rubric score · pass threshold 0.70

Makes a guaranteed-refund promise, which policy reserves for agents, and contradicts the 14-day cooling-off rule.

Diagnosis

Instruction-following failures · Refund promises

Root cause

Forbidden promise phrasing ('no questions asked') not filtered.

Suggested fix

Add 'no questions asked' and 'guaranteed refund' to the policy-violation phrase list.

Playbook: instruction-following failures

The answer ignored format, tone, policy or language constraints from the system prompt.

Detection signals

  • Rule check fails: length, language, required closing line
  • Promises that policy reserves for agents, such as guaranteed refunds
  • Judge flags tone or format

Common fixes

  1. Move hard constraints into automated rule checks
  2. Detect the customer's language and enforce it
  3. Maintain a policy-violation phrase list with support leads
  4. Gate prompt versions on the instruction-adherence subset

Activity

1 event · 0 comments

  1. Relevant ·

    Detected by v3.4 · Gemini 2.5 Pro run and labeled instruction-following failures