Evaluations

Release, comparison and nightly runs for Support Assistant, scored by the rubric v3 judge. Select two to four runs to compare them side by side.

17 release and experiment runs90 nightly runsJudge · Claude Opus 5.5 · rubric v3

Runs, last 30 days

1 running · 2 failed

Nightly pass rate

87.2%

Mean of 30 nightly runs

Best full golden-set run

86.9%

support-assistant v3.4 · release candidate

Release gate

Passed

support-assistant v3.4 · release candidate · 7 of 7 checks

107 of 107 runs

Evaluation runs

v3.4 · synthetic-hard-negatives v1.3 Experimentsynthetic-hard-negativesClaude Sonnet 5.5 340 / 540 RunningOct 8	340 / 540	0.86	0.83	0.68	0.73	4.0s	$0.0043	Oct 8, 08:05 in progress	Running
Nightly · v3.4 · 2026-10-08 Nightlysupport-golden-v4Claude Sonnet 5.5 87.2% Completedp95 4.3s · $0.0041/reqOct 8	87.2%	0.92	0.88	0.79	0.85	4.3s	$0.0041	Oct 8, 02:00 42m 35s	Completed
v3.4 · Claude Haiku 5.5 comparison Experimentsupport-golden-v4Claude Haiku 5.5 82.3% Failedp95 2.3s · $0.0013/reqOct 7	82.3%	0.89	0.83	0.79	0.84	2.3s	$0.0013	Oct 7, 07:40 1h 41m	Failed
Nightly · v3.4 · 2026-10-07 Nightlysupport-golden-v4Claude Sonnet 5.5 87.3% Completedp95 3.9s · $0.0041/reqOct 7	87.3%	0.91	0.87	0.79	0.84	3.9s	$0.0041	Oct 7, 02:00 44m 3s	Completed
v3.4 · Gemini 2.5 Pro comparison Experimentsupport-golden-v4Gemini 2.5 Pro 83.8% Completedp95 4.6s · $0.0048/reqOct 6	83.8%	0.88	0.86	0.79	0.84	4.6s	$0.0048	Oct 6, 14:06 2h 54m	Completed
v3.4 · GPT-5 comparison Experimentsupport-golden-v4GPT-5 85.3% Completedp95 5.2s · $0.0061/reqOct 6	85.3%	0.90	0.87	0.79	0.84	5.2s	$0.0061	Oct 6, 14:05 3h 8m	Completed
support-assistant v3.4 · release candidate Releasesupport-golden-v4Claude Sonnet 5.5 86.9% Completedp95 3.8s · $0.0042/reqOct 6	86.9%	0.91	0.88	0.79	0.84	3.8s	$0.0042	Oct 6, 14:02 2h 44m	Completed
Nightly · v3.4 · 2026-10-06 Nightlysupport-golden-v4Claude Sonnet 5.5 86.6% Completedp95 3.9s · $0.0042/reqOct 6	86.6%	0.91	0.87	0.79	0.85	3.9s	$0.0042	Oct 6, 02:00 43m 4s	Completed
Nightly · v3.4 · 2026-10-05 Nightlysupport-golden-v4Claude Sonnet 5.5 86.8% Completedp95 3.9s · $0.0041/reqOct 5	86.8%	0.92	0.89	0.78	0.85	3.9s	$0.0041	Oct 5, 02:00 45m 16s	Completed
v3.4 · locale-formatting Experimentlocale-formattingClaude Sonnet 5.5 80.2% Completedp95 4.1s · $0.0044/reqOct 4	80.2%	0.86	0.83	0.75	0.80	4.1s	$0.0044	Oct 4, 10:15 21m	Completed
Nightly · v3.4 · 2026-10-04 Nightlysupport-golden-v4Claude Sonnet 5.5 88.7% Completedp95 3.9s · $0.0043/reqOct 4	88.7%	0.90	0.87	0.78	0.84	3.9s	$0.0043	Oct 4, 02:00 46m 44s	Completed
Nightly · v3.4 · 2026-10-03 Nightlysupport-golden-v4Claude Sonnet 5.5 87.9% Completedp95 3.7s · $0.0042/reqOct 3	87.9%	0.90	0.88	0.80	0.83	3.7s	$0.0042	Oct 3, 02:00 44m 43s	Completed
v3.4 · billing-edge-cases v2.1 Experimentbilling-edge-casesClaude Sonnet 5.5 81.7% Completedp95 4.0s · $0.0043/reqOct 2	81.7%	0.87	0.84	0.77	0.82	4.0s	$0.0043	Oct 2, 09:30 49m	Completed
Nightly · v3.4 · 2026-10-02 Nightlysupport-golden-v4Claude Sonnet 5.5 87.3% Completedp95 3.6s · $0.0042/reqOct 2	87.3%	0.92	0.88	0.79	0.84	3.6s	$0.0042	Oct 2, 02:00 43m 7s	Completed
Nightly · v3.4 · 2026-10-01 Nightlysupport-golden-v4Claude Sonnet 5.5 87.6% Completedp95 3.8s · $0.0041/reqOct 1	87.6%	0.90	0.89	0.79	0.84	3.8s	$0.0041	Oct 1, 02:00 39m 51s	Completed