Eval-Driven Development (EDD)

Connecting an LLM to MCP tools, APIs, or terminals turns a chatbot into a decision-maker. Failures rarely look like stack traces. They look like a wrong tool, a hallucinated parameter, or an infinite retry loop.

EDD treats prompts and tool schemas as version-controlled, evaluated contracts. Waykit ships the harness, CI gates, and production closed loop as one learning loop inside the feature lifecycle.

The loop (same shape as TDD)

1. Red - Define intent. JSONL cases and YAML metrics assert the tool (and arguments) you expect, and that chatty questions do not invent tool calls.
2. Green - Implement the interface. Register the MCP/tool contract and minimal system instructions. Run until asserts pass.
3. Refactor - Refine context. Iterate descriptions and constraints without breaking existing cases. Gate merges with wk eval ci --threshold-routing 95.

Why run it this way

Bare wk eval still validates which Kit skill activates. wk eval run|watch|report|ci validates how an agent calls tools. Use EDD whenever you change prompts, tool schemas, or routing.

Quick start

curl -fsSL https://raw.githubusercontent.com/mzworthington/waykit/main/install.sh | sh
wk init . --mcp default --hook

wk eval run --suite evals/edd/architecture_routing.yaml --model scripted
wk eval ci --threshold-routing 95 --out out/reports
wk eval report --format md --out out/reports
wk eval watch --suite evals/edd/architecture_routing.yaml --target evals/edd
wk eval dataset lint --dataset evals/edd/architecture_routing.jsonl

kit and agent-kit are aliases of wk.

Cursor, Copilot, and API keys

Cursor is the reference host for progressive skills and MCP compose. Copilot, Claude Code, Gemini CLI, and Windsurf get the same canonical AGENTS.md via thin stubs (wk export-rules / wk init), not equal skill or MCP discovery. Daily work and the merge gate use --style local (alias: --model scripted). You do not need an OpenAI (or any provider) API key for that.

Each run has one style for both the agent under test and the judge:

--style local never spends, even if a key is in the environment. Cases tagged requires-live are skipped on local. --style http and --style cli run them.

Cursor Agent CLI as the agent under test

cursor-agent is not OpenAI /chat/completions. Kit prompts it in --mode=ask and expects a JSON envelope { "content": "…", "tool_calls": [{ "name": "<eval tool>", "arguments": {} }] } using only registered eval tools. After a tool call, the harness fills content from the mock JSON (quality metrics grade that grounded text). Token totals come from the CLI JSON usage object when present (inputTokens / input_tokens / prompt_tokens); otherwise Kit estimates ~4 characters per token. Suite summaries include a rough USD line using $0.003 per 1k tokens (DEFAULT_TOKEN_USD_PER_1K). Override with KIT_EVAL_TOKEN_USD_PER_1K, or set it to 0 to hide USD. Local/scripted models omit USD. Judge calls use the same --cli and --model.

noglob wk eval run --suite evals/edd/architecture_routing.yaml \
  --style cli --cli cursor-agent --model cursor-grok-4.6-medium

When a key is used

The runner takes the first non-empty value:

1. KIT_EVAL_API_KEY (preferred; this is the secret nightly CI looks for)
2. OPENAI_API_KEY
3. ANTHROPIC_API_KEY

That value is sent as Authorization: Bearer … to an OpenAI-compatible {baseUrl}/chat/completions. Base URL resolution: KIT_EVAL_BASE_URL, then OPENAI_BASE_URL, then https://api.openai.com/v1.

ANTHROPIC_API_KEY is only useful if KIT_EVAL_BASE_URL points at a gateway that accepts Anthropic keys on the OpenAI request shape. Anthropic’s native Messages API is not this client.

The same key and model are reused for:

• Agent: given this prompt and these tools, which call do you make?
• Judge: second completion for llm_as_judge / criteria_judge / task_completion

# Local model server (no paid key)
wk eval run --suite evals/edd/architecture_routing.yaml \
  --style http --base-url http://localhost:11434/v1 --model llama3.1

# Cursor Agent CLI (agent and judge)
noglob wk eval run --suite evals/edd/architecture_routing.yaml \
  --style cli --cli cursor-agent --model cursor-grok-4.6-medium

PR Verify and wk check stay --style local (no key). When you run a live model, prefer http for the nightly job; cli is for local iteration (cursor-agent, claude, or agy). CLI runs hit subscription rate limits and have weaker structured-output guarantees.

Optional: KIT_EVAL_MODEL. Rough USD uses $0.003 per 1k tokens unless KIT_EVAL_TOKEN_USD_PER_1K is set (0 disables). Local OpenAI-compatible servers also work via KIT_EVAL_BASE_URL.

Metrics and suites

Beyond tool selection and schema shape, suites can assert:

Full metric table and harness layout: evals/edd/README.md.

CI

Nightly skips the whole job if KIT_EVAL_API_KEY is empty. It does not fall through to OPENAI_API_KEY.

Verify and the nightly live job (plus Pages deploy) publish a job summary: what the gate means, then an EDD overview table and collapsible full report via wk eval report --github-summary.

Local live run

export KIT_EVAL_API_KEY='…'   # or OPENAI_API_KEY
# optional: export KIT_EVAL_BASE_URL='https://api.openai.com/v1'
wk eval run --suite evals/edd/architecture_routing.yaml --style http --model gpt-4o-mini

You should then see requires-live cases execute instead of “Skipping N requires-live case(s)”.

Next