title: Eval-Driven Development (EDD)
kind: sop
triggers
tools

Standard Operating Procedure: Eval-Driven Development (EDD)

Default: When changing agent prompts, MCP tool schemas, or routing behavior, use EDD - not ad-hoc playground checks. Companion: docs/edd.md, evals/edd/README.md.

Loop (red → green → refactor)

Red - Define intent: Add a JSONL case (id, prompt, optional history / tags / expect) and point a YAML suite at it with metrics (tool_selection, schema_match, argument_correctness, task_completion, criteria_judge, mcp_use, plan_adherence, step_efficiency, plugin, llm_as_judge, self_correction, terminal_fallback).
Green - Implement interface: Register the tool contract (MCP JSON under evals/edd/tools/ or suite mcp_tools) and system prompt. Run wk eval run --suite … --style local (or --style http / --style cli).
Refactor - Refine context: Iterate tool description / parameter hints / system prompt until routing and schema assertions pass without hallucinated parameters.

CLI

Command	Purpose
`wk eval run --suite <path> --model <name>`	Execute one suite
`wk eval watch --suite <path> --target <file>`	Re-run on prompt / tool schema changes
`wk eval report --format md\|json --out <dir> [--github-summary]`	Markdown or JSON cost/latency/failure report; optional Actions job summary
`wk eval ci --threshold-routing 95 --out out/reports`	Headless gate; fail if routing accuracy < threshold
`wk eval run --suite evals/edd/goldens/architecture_routing.yaml --style cli --cli cursor-agent --model …`	Live golden (not CI)
`wk eval dataset lint\|dedupe\|synthesize\|from-trace`	Dataset hygiene (schema lint, dedupe, paraphrases, prod promote)

kit and agent-kit are aliases of wk.

IDEs vs live keys

Cursor and GitHub Copilot already load Kit via .cursorrules and .github/copilot-instructions.md. Run evals with --style local; no provider key. --style http POSTs to an OpenAI-compatible /chat/completions. --style cli shells out to one assistant binary for agent and judge. Key order, nightly vs PR: docs/edd.md.

CI

Local gate: .github/workflows/ci.yml Verify runs kit check / wk eval ci with --style local. Cases tagged requires-live are skipped so paraphrases do not fail CI. kit check covers architecture routing, kit-knowledge, Cloudflare ops, safety, self-correction, and terminal-fallback suites.
Live nightly: .github/workflows/edd-live.yml runs on a schedule when KIT_EVAL_API_KEY is set and KIT_EVAL_MODEL is a real provider model. That job includes requires-live rows. Missing KIT_EVAL_API_KEY skips the job; it does not fall through to OPENAI_API_KEY.
Threshold gating: --threshold-routing 95 blocks merges when routing/schema extraction fails more than 5% of routing-tagged cases.
Artifacts: Reports upload with if: always() (out/reports/eval-report.md, edd-report.md / .json).
Job summaries: CI workflows publish an overview table plus the full Markdown report to $GITHUB_STEP_SUMMARY (via wk eval report --github-summary, or automatically when GITHUB_ACTIONS=true). Open the workflow run → Summary to read pass rate, routing/schema, and failure traces without downloading artifacts. Unit tests use pnpm test:ci, which also writes out/reports/unit-test-report.md into that Summary.

Reports

wk eval report emits overall pass rate, token/latency cost, routing + schema adherence, and failure traces (expected vs actual tool/args, diagnosis, suggested fix). Example: evals/edd/examples/eval-report.md.

Under GitHub Actions, the same Markdown is folded into the job summary so green/red checks carry meaning (what gated, which suites, metrics).

Production bridge

Live spans share eval field names (emitAgentSpan). Hard failures convert to JSONL via productionTraceToJsonl / wk eval dataset from-trace. Shadow sample + judge: wk eval shadow --infile … --sample 0.05. See SOPs/edd-production-telemetry.md.

IDE session → EDD (debug / lessons)

When a miss appears in the current Cursor/Copilot thread (wrong tool, bad args, prompt/schema drift), agent-debug must promote a case from context (no user paste required), then red/green with wk eval (hypothesis-driven-debug.md §11). Lessons that capture the same friction set EDD case + optional Promote to evals/edd/*.jsonl (templates/lesson.md). Other IDE threads remain invisible until reopened or exported.

Markdown source