Skip to content

F-2386Detected Sep 27, 02:21 UTC

Can I pay my bill with a credit card from another person?

Found in Nightly · v3.4 · 2026-09-27 on support-golden-v4 · Billing & payments

Incorrect factual answersLowIgnoredClaude Sonnet 5.5

Model answer vs gold answer

45% of model-answer words appear in the gold answer

Model answer

support-assistant v3.4

Yes, Unsupported: any credit card can be used to pay your bill.

Gold answer

Human-verified · support-golden-v4

Yes, a third-party card can be used for one-off payments, but saved cards and direct debit must be in the account holder's name.

  • claimFlagged by the judge as unsupported or contradicted

Retrieved context

bge-m3 · hybrid (BM25 + dense, RRF) · top 3 passed to the generator

2 of 3 retrieved chunks are relevant · gold passage ranked first

  1. 1

    Paying with direct debit and saved cards

    Supported methods · kb-billing-052 · v2026.09

    Relevant
    One-off card payments can use any card. Saved cards must be in the account holder's name.
    Similarity0.84
  2. 2

    Paying with direct debit and saved cards

    Updating card details · kb-billing-052 · v2026.09

    Relevant
    Update saved card details in My Harbor.
    Similarity0.79
  3. 3

    Understanding your monthly bill

    Bill structure · kb-billing-014 · v2026.09

    Not relevant
    Your bill shows charges, credits and taxes.
    Similarity0.65

Judge verdict

LLM judge · Claude Opus 5.5 · rubric v3

0.58

rubric score · pass threshold 0.70

Partially correct: omits the saved-card restriction stated in chunk 1.

Diagnosis

Incorrect factual answers · Payment methods

Root cause

Over-simplification of a two-part rule.

Suggested fix

Gold answer accepted as borderline; tracked through the completeness rubric instead.

Playbook: incorrect factual answers

The answer contradicts the knowledge base, usually by misreading or conflating two policies.

Detection signals

  • Answer correctness below 0.7 while faithfulness stays high
  • Disagreement with gold on a key entity: plan, fee, date or zone
  • Reviewer edits the fact but keeps the structure

Common fixes

  1. Check for a stale article version before changing the prompt
  2. Add a structured extraction step for fees and plan limits
  3. Add contrastive examples for similarly worded plans and zones
  4. Route high-stakes billing intents to verified templates

Activity

2 events · 0 comments

  1. Relevant ·

    Detected by support-assistant v3.4 run and labeled incorrect factual answers

  2. Daniel Okafor ·

    Ignored: judged acceptable for one-off payments; completeness tracked separately