Eval-Driven Development (EDD) suites
Suite reference for Kit’s agent eval harness. Day-to-day steps in SOPs/eval-driven-development.md. Companion: SOPs/edd-production-telemetry.md.
Layout
evals/edd/
├── README.md
├── system_prompt.md
├── kit_knowledge_prompt.md
├── demo.yaml|.jsonl ← first-hour teaching suite
├── examples/before-after.md
├── examples/eval-report.md
├── examples/prod-trace.json
├── examples/otel-agent-loop.json ← kit.* OTel span fixture
├── examples/prod-turns.jsonl ← shadow-eval NDJSON corpus
├── architecture_routing.yaml|.jsonl ← CI seed (frozen unique intents)
├── goldens/ ← live golden + holdout (not kit check)
├── architecture_self_correction.yaml|.jsonl
├── architecture_terminal.yaml|.jsonl
├── kit_knowledge.yaml|.jsonl
├── cloudflare_ops.yaml|.jsonl
├── safety.yaml|.jsonl
└── tools/*.json
Styles
One style per run for both agent and judge.
| Style | When | What it proves |
|---|---|---|
local (CI default) | wk eval ci, PR CI (wk check), --model scripted | Harness, schema, keyword routing. Not a product LLM test. No API key. |
http | --style http --model <id> with key or --base-url; nightly .github/workflows/edd-live.yml | Same model for routing and quality metrics. Includes requires-live cases. |
cli | --style cli --cli cursor-agent|claude|agy --model <id> (--cli is required) | Same CLI binary for agent and judge. |
Cursor is the reference host for skills and MCP (AGENTS.md → .cursorrules / .github/copilot-instructions.md). Other IDEs get thin stubs, not equal discovery. None of them is the eval driver: wk eval never calls Cursor Chat or Copilot Chat. Env resolution, CI jobs, and examples: docs/edd.md (section Cursor, Copilot, and API keys).
Do not extend the local keyword driver to pass requires-live cases. Add JSONL rows instead. Volume for live ranking lives in goldens/ — not in CI seeds.
Live goldens
Architecture routing has a CI seed (architecture_routing.jsonl, unique intents, kit check) and a live golden (goldens/, 80 working + 20 holdout). Run goldens with --style cli or --style http. Do not add them to EDD_CI_SUITES. Do not grow them with dataset synthesize. Procedure: goldens/README.md.
Quick start
First-hour teaching suite (six cases, before/after story in examples/before-after.md):
wk eval run --suite evals/edd/demo.yaml --model scripted
wk eval ci --suite evals/edd/demo.yaml --threshold-routing 95 --model scripted --out out/reports
wk eval report --format md --out out/reports
Full regression / CI suites:
wk eval run --suite evals/edd/architecture_routing.yaml --model scripted
wk eval ci --suite evals/edd/kit_knowledge.yaml --threshold-routing 95 --model scripted --out out/reports
wk eval ci --suite evals/edd/cloudflare_ops.yaml --threshold-routing 95 --model scripted --out out/reports
wk eval ci --suite evals/edd/architecture_routing.yaml --threshold-routing 95 --out out/reports
wk eval ci --suite evals/edd/safety.yaml --threshold-routing 95 --model scripted --out out/reports
wk eval watch --suite evals/edd/architecture_routing.yaml --target evals/edd
kit and agent-kit alias wk.
Metrics
| Type | Asserts |
|---|---|
tool_selection | Correct tool, expect.no_tool, or ordered expect.tools[] |
schema_match | Valid JSON object args (type/shape; not value meaning) |
argument_correctness | expect.arguments_contains / per-call args match intent meaning |
task_completion | User goal achieved (expect.goal or expected tool plan); scripted heuristic or live judge |
criteria_judge | Written suite criteria + threshold (0-1); per-criterion reasons |
mcp_use | Only catalog MCP tools; expected MCP capability when intent requires it |
plan_adherence | Ordered expect.tools[] / expect.tool matches trajectory steps |
step_efficiency | Tool step count <= max_steps (defaults to plan length or 1) |
plugin | Consumer module (module: ./plugin.mjs) receives case + trajectory |
llm_as_judge | Semantic accuracy / hallucination / tone (skipped when expect.no_tool) |
self_correction | Param updates after injected errors |
terminal_fallback | Circuit breaker stops endless retries |
Harness layout (hexagonal)
Pure metric and judge logic stays inward. OpenAI-compatible HTTP, headless assistant CLIs (claude / cursor-agent / agy), and dynamic plugin imports live only in adapters.
Styles (agent + judge)
| Style | Flag | Use |
|---|---|---|
| local | default / --style local | Keyword agent + heuristic judge. Offline CI. |
| http | --style http --model <id> plus key or --base-url | Same OpenAI-compatible model for agent and judge |
| cli | --style cli --cli cursor-agent|claude|agy --model <id> | Same headless CLI for agent and judge. Cursor installs ~/.local/bin/cursor-agent. |
wk eval run --suite evals/edd/architecture_routing.yaml --style http --base-url http://localhost:11434/v1 --model llama3.1
noglob wk eval run --suite evals/edd/architecture_routing.yaml \
--style cli --cli cursor-agent --model cursor-grok-4.6-medium
Safety suite
Gateable injection / no-tool suite: evals/edd/safety.yaml. kit check runs it plus architecture routing, kit-knowledge, Cloudflare ops, self-correction, and terminal-fallback via EDD_CI_SUITES.
Dataset hygiene
wk eval dataset lint --dataset evals/edd/architecture_routing.jsonl
wk eval dataset dedupe --dataset path.jsonl --out path.deduped.jsonl
wk eval dataset synthesize --dataset path.jsonl --count 2 --out path.syn.jsonl
wk eval dataset from-trace --trace evals/edd/examples/prod-trace.json --out out/prod.jsonl
wk eval shadow --infile evals/edd/examples/prod-turns.jsonl --sample 1 --seed 1 --out out/shadow-fails.jsonl
Synthetic paraphrases keep expectations, add tags synthetic + requires-live. Use them to propose wording, not to fill the golden. Lint goldens with:
wk eval dataset lint --dataset evals/edd/goldens/architecture_routing.jsonl
wk eval dataset lint --dataset evals/edd/goldens/architecture_routing.holdout.jsonl
Production telemetry (closed loop)
Promote production misses into the suite with the same kit.* fields as eval cases:
wk eval shadow --infile evals/edd/examples/prod-turns.jsonl --sample 1 --seed 1 --out out/shadow-fails.jsonl
wk eval dataset from-trace --trace evals/edd/examples/prod-trace.json --out out/prod.jsonl
Fixtures: examples/otel-agent-loop.json, examples/prod-turns.jsonl, examples/prod-trace.json. Procedure: SOPs/edd-production-telemetry.md.
Tags
| Tag | Meaning |
|---|---|
routing | Counts toward routing accuracy |
seed | Frozen CI unique intent |
golden | Live ranking catalog |
holdout | Frozen live split — do not tune against it |
requires-live | Skipped when style is local |
prod-derived | Converted from a production miss via productionTraceToJsonl |
prompt-injection | Instruction-override attempts |
Reports
| Artifact | Role |
|---|---|
out/reports/eval-report.md | Stable alias for PR review |
out/reports/edd-report.md | Same Markdown body |
out/reports/edd-report.json | Machine-readable results |
| GitHub Actions job summary | Overview table + collapsible full report (wk eval report --github-summary) |
Includes pass rate, tokens/latency, routing + schema adherence, and failure traces. Example: examples/eval-report.md.
CI workflows (.github/workflows/ci.yml Verify, edd-live.yml) write a short “what this gate means” preamble plus the EDD overview into the run Summary tab.
Live models (optional): KIT_EVAL_API_KEY first, then OPENAI_API_KEY, then ANTHROPIC_API_KEY. Optional KIT_EVAL_BASE_URL / OPENAI_BASE_URL (OpenAI-compatible /chat/completions; default https://api.openai.com/v1), KIT_EVAL_MODEL. USD on live/CLI reports defaults to $0.003 per 1k tokens (KIT_EVAL_TOKEN_USD_PER_1K to override, 0 to disable). Nightly CI only reads KIT_EVAL_API_KEY.