Making evaluator trust visible before teams rely on its scores.
Built and evaluated a Pinterest spend-decision agent, traced failures, and proposed a connected evaluator lifecycle with early alignment, persistent status and disagreement triage.
The problem I experienced
An evaluator could run through the SDK and appear as a score column, yet be difficult to find, reuse or align in the interface. I discovered Align Evaluator late in the journey, after building and calibrating my own judges. That made a workflow problem visible: a score could look authoritative before its reliability had been checked.
My work and method
I defined 13 golden scenarios from Pinterest affiliate-commerce decision traps, inspected traces and converted failures into seven graders. Claude helped implement and run the agent and evaluation suite. I set the answer key, checked the analyses and resolved disagreements. The scenarios were authored tests, not a random sample of production traffic.
What the experiment showed
| Prompt version | Golden-label matches |
|---|---|
| Baseline | 6 of 13 |
| Rubric v2 | 8 of 13 |
| Rubric v3 | 7 of 13 |
The second version removed the dangerous spend errors in this test set, but still missed five exact decisions. A stricter third prompt regressed. An unchecked AI judge also rated the improved agent less favorably, reinforcing the need to validate the evaluator before trusting the experiment.
The interface proposal
Make SDK evaluators visible as reusable objects first. Then offer alignment when an evaluator is created, suggest existing ground truth, and show a durable aligned or unverified status wherever its scores appear. Keep disagreement triage close to the evidence.
Scope and success criteria
The proposed sequence was evaluator identity and reuse, then persistent alignment status, then richer triage and suites. Success measures would include time to first aligned evaluator and how often code-created evaluators become selectable in the UI. These are proposed measures; this take-home did not ship in Arize.