Every evaluation is only as good as its gold answers, and gold answers go stale whenever a policy changes. The teams that keep ground truth fresh treat review as a queue with a service level, not as a quarterly project.
Feed the queue from failures and policy changes
Two sources produce most review work: failures where the judge and the gold answer disagree, and knowledge-base updates that invalidate existing gold answers. Route both into one queue with the source article and section attached.
Four actions are enough
Reviewers need to approve, edit, reject or flag. Anything more slows the queue down.
- Approve: the proposed gold answer is correct and complete.
- Edit: fix the answer and record a reason code (policy change, factual error, incomplete).
- Reject: out of scope, ambiguous or duplicate.
- Flag: needs a policy owner or legal decision.
Make it fast
Keyboard shortcuts (J and K to move, A to approve, E to edit, R to reject) and the source excerpt shown next to the answer bring median review time under three minutes. Show progress against the queue size so reviewers see the backlog shrink.