Eval-Driven Development (EDD) — alpha

Connecting an LLM to MCP tools, APIs, or terminals turns a chatbot into a decision-maker. Failures rarely look like stack traces. They look like a wrong tool, a hallucinated parameter, or an infinite retry loop.

Status: alpha. Waykit ships a routing and schema harness you can run in CI. It is not a full EDD framework. There is no prompt registry, no hosted dataset product, no native Anthropic or Gemini judge, and the default merge gate does not call a product LLM. Skill-trigger wk eval (no subcommand) only checks which Kit skill should activate. Use this loop when you change prompts, tool schemas, or routing. Do not treat a green wk eval ci as proof that the live agent is correct.

What you do get today:

• YAML suites + JSONL cases, mocked tools, dual-layer asserts
• A scripted (--style local) merge gate so PRs fail on keyword routing drift
• Optional http (OpenAI-compatible /chat/completions) and cli (cursor-agent, claude, agy) runs
• Dataset helpers and a production → JSONL / shadow path (wk eval dataset from-trace, wk eval shadow, wk eval miss-rate)

Still to build before this is a framework: live judges per provider, skill-trigger evals that actually run a model, stronger structured output from CLI hosts, and a tighter closed loop from prod traces into suites.

EDD treats prompts and tool schemas as version-controlled, evaluated contracts. The harness is one learning loop inside the feature lifecycle, not the whole product.

Interactive demo

Same demo-edge case as the teaching suite. Case, red, report, green, then the CI gate.

Write-up: before / after. Suite: evals/edd/demo.yaml.

The loop (same shape as TDD)

1. Red - Define intent. JSONL cases and YAML metrics assert the tool (and arguments) you expect, and that chatty questions do not invent tool calls.
2. Green - Implement the interface. Register the MCP/tool contract and minimal system instructions. Run until asserts pass.
3. Refactor - Refine context. Iterate descriptions and constraints without breaking existing cases. Gate merges with wk eval ci --threshold-routing 95.

Why run it this way

Bare wk eval still validates which Kit skill activates. wk eval run|watch|report|ci validates how an agent calls tools. Use EDD whenever you change prompts, tool schemas, or routing.

Quick start

curl -fsSL https://raw.githubusercontent.com/mzworthington/waykit/main/install.sh | sh
wk init . --mcp default --hook

wk eval run --suite evals/edd/architecture_routing.yaml --model scripted
wk eval ci --threshold-routing 95 --out out/reports
wk eval report --format md --out out/reports
wk eval watch --suite evals/edd/architecture_routing.yaml --target evals/edd
wk eval dataset lint --dataset evals/edd/architecture_routing.jsonl
wk eval miss-rate

Cursor, Claude, Copilot, Antigravity, and API keys

Rules and MCP install across those hosts: hosts. Daily work and the merge gate use --style local (alias: --model scripted). You do not need a provider API key for that. The IDE chat is never the eval runner.

Each run has one style for both the agent under test and the judge:

--style local never spends, even if a key is in the environment. Cases tagged requires-live are skipped on local. --style http and --style cli run them.

Cursor Agent CLI as the agent under test

cursor-agent is not OpenAI /chat/completions. Kit prompts it in --mode=ask and expects a JSON envelope { "content": "…", "tool_calls": [{ "name": "<eval tool>", "arguments": {} }] } using only registered eval tools. After a tool call, the harness fills content from the mock JSON (quality metrics grade that grounded text). Token totals come from the CLI JSON usage object when present (inputTokens / input_tokens / prompt_tokens); otherwise Kit estimates ~4 characters per token. Suite summaries include a rough USD line using $0.003 per 1k tokens (DEFAULT_TOKEN_USD_PER_1K). Override with wk _EVAL_TOKEN_USD_PER_1K, or set it to 0 to hide USD. Local/scripted models omit USD. Judge calls use the same --cli and --model.

noglob wk eval run --suite evals/edd/architecture_routing.yaml \
  --style cli --cli cursor-agent --model cursor-grok-4.6-medium

When a key is used

The runner takes the first non-empty value:

1. KIT_EVAL_API_KEY (OpenAI-compatible HTTP style)
2. OPENAI_API_KEY
3. ANTHROPIC_API_KEY

Weekly live CI does not use that HTTP chain. It reads CURSOR_API_KEY and shells out to cursor-agent.

That value is sent as Authorization: Bearer … to an OpenAI-compatible {baseUrl}/chat/completions. Base URL resolution: wk _EVAL_BASE_URL, then OPENAI_BASE_URL, then https://api.openai.com/v1.

ANTHROPIC_API_KEY is only useful if wk _EVAL_BASE_URL points at a gateway that accepts Anthropic keys on the OpenAI request shape. Anthropic’s native Messages API is not this client.

The same key and model are reused for:

• Agent: given this prompt and these tools, which call do you make?
• Judge: second completion for llm_as_judge / criteria_judge / task_completion

# Local model server (no paid key)
wk eval run --suite evals/edd/architecture_routing.yaml \
  --style http --base-url http://localhost:11434/v1 --model llama3.1

# Cursor Agent CLI (agent and judge)
noglob wk eval run --suite evals/edd/architecture_routing.yaml \
  --style cli --cli cursor-agent --model cursor-grok-4.6-medium

PR Verify and wk check stay --style local (no key). Weekly live uses cli (cursor-agent + CURSOR_API_KEY). Use http for OpenAI-compatible providers. CLI runs hit subscription rate limits and have weaker structured-output guarantees.

Optional: wk _EVAL_MODEL. Rough USD uses $0.003 per 1k tokens unless wk _EVAL_TOKEN_USD_PER_1K is set (0 disables). Local OpenAI-compatible servers also work via wk _EVAL_BASE_URL.

Metrics and suites

Beyond tool selection and schema shape, suites can assert:

Full metric table and harness layout: evals/edd/README.md.

CI

The scheduled job fails if CURSOR_API_KEY is empty so a missing secret cannot look green. workflow_dispatch may skip with allow_unkeyed_skip; the summary then says so. Set the secret with gh secret set CURSOR_API_KEY --body "$CURSOR_API_KEY". Optional KIT_EVAL_MODEL overrides the default Cursor model id. Scripted wk check in Verify stays the PR merge gate; live remains extra.

Verify and the weekly live job (plus Pages deploy) publish a job summary: what the gate means, then an EDD overview table and collapsible full report via wk eval report --github-summary.

Local live run

export CURSOR_API_KEY='…'
noglob wk eval run --suite evals/edd/architecture_routing.yaml \
  --style cli --cli cursor-agent --model cursor-grok-4.6-medium

You should then see requires-live cases execute instead of “Skipping N requires-live case(s)”.

Next