Project Overview
Agent Eval Studio is an open-source workbench for version comparison and release decisions. It connects evaluation data, scoring rules, run artifacts, and release conditions into one traceable path. Its purpose is not a polished score alone, but an explanation of why a candidate can or cannot ship.
Pain Points & Opportunities
Agent teams can over-rely on curated demos, subjective impressions, or averages. Those methods are hard to reproduce and can hide rare high-risk failures. A quality gain accompanied by worse cost or tail latency may also be unsuitable for release.
Goals & Metrics
The goal was a repeatable A/B baseline with identical data, a shared rubric, structured bad cases, and explicit gates. Reviewers should run the whole loop without an API key, while the absence of a live-model provider remains visible.
User Research
The design follows the essential evidence chain for Agent evaluation: a Golden Set stabilizes input, dimension-level rubrics resist single-score distortion, bad cases explain failure patterns, and multi-objective gates connect evaluation to release constraints.
Solution Design
The data layer stores tasks, expectations, and risk labels. Baseline and candidate receive identical input. Scoring produces five dimensions; analysis groups task types and bad cases; gates jointly check quality, cost, P95 latency, and safety; reporting emits both machine-readable JSON and human-readable HTML.
Execution Plan
I used TypeScript and Node.js to implement the CLI, deterministic provider, dataset loading, scoring aggregation, gates, and report generation, with automated tests for critical logic. OpenAI-compatible configuration remains an extension point, and mock results are explicitly not presented as live-model outcomes.
Results Review
The current version runs all 60 samples end to end, compares two deterministic configurations, and outputs dimension scores, bad cases, a release decision, JSON artifacts, and an HTML report. It validates the evaluation product and engineering path, not an improvement in any real model or prompt.
Iteration Reflection
The core product value of evaluation is not automated scoring; it is reviewable decision evidence. The next step is to add a live provider and human-calibrated examples, then compare automated and human judgments. Until then, marking the validation boundary clearly is more valuable than manufacturing a complete-looking production metric.
