Kit review: value and model-agnosticism
Status: refreshed 2026-09-01 (supersedes the Aug 30 draft on PR #22).
Open actions: kit-review-backlog.md
Verdict: Still valuable for teams that want hexagonal + DDD + TDD/XFN discipline and primarily run agents in Cursor (or another host that loads AGENTS.md + progressive skills). Still not model-agnostic in the strong sense: content is mostly portable markdown; discovery, MCP install, skill format, and the live EDD driver remain Cursor- and OpenAI-compatible-shaped. Public copy now names that split; skill-trigger evals still do not invoke a model.
What this kit is
Waykit is a governance + procedure pack for coding agents, plus a TypeScript CLI (kit) for bootstrap, MCP compose, security scan, and eval harnesses. Core ideas:
- Thin always-on bootstrap (
AGENTS.md) with on-demand skills/SOPs/philosophy. - Lifecycle roles (
agent-*) and stack profiles (lang-*/framework-*). - Architecture invariants (hexagonal, DDD, vertical slices, clean code) with applicability / opt-out (CODING_PHILOSOPHY.md, docs/ADRs).
- Eval-Driven Development for agent tool routing/schemas (
kit eval). - MCP profile catalog and multi-IDE pointer files via
kit export-rules.
Rough inventory (as of this refresh): ~45 skills, SOPs, dual eval layers (evals/suites + evals/edd), MCP profiles, install/init, nightly live-model workflow (when keyed).
What still holds (strengths)
- Context budget as design: thin
AGENTS.md, on-demand loads, one MCP profile,kit measure-context/kit check. - Lifecycle taxonomy: phase → skill routing, handovers, tests-as-catalog, orchestrator + specialists.
- EDD shape: YAML metrics + JSONL cases, scripted merge gate, optional live path, shadow/prod→JSONL story (docs/edd.md).
- Architecture floor: explicit defaults with documented applicability/opt-out (landed after the Aug 30 draft).
- Supply-chain hygiene: layout verify, lockfile, audit, secrets-out-of-repo MCP fragments.
- Portable content layer: philosophy/SOPs/role prose travel; runtime discovery does not.
What changed since the Aug 30 draft
| Finding (Aug 30) | Status now |
|---|---|
| Architecture invariants feel non-optional | Improved: applicability and opt-out + seed ADRs (#36) |
| Closed-loop / live EDD aspirational | Improved: deeper EDD suites, shadow path, nightly edd-live.yml (still skips without KIT_EVAL_API_KEY) |
| Default CI is scripted keyword driver | Still true: intentional merge gate; docs now say so clearly |
| Skill-trigger evals are theater | Still true: kit/src/edd/run_evals.ts does not invoke a model or assert required_patterns / required_output_sections |
| Multi-IDE peer-depth oversold | Improved: README, FAQ, and llms.txt name Cursor as the reference host; stubs vs discovery is explicit |
| Skill length budget slipping | Still true: agent-prune / agent-orchestrator / agent-debug / agent-copy over ~150 lines |
| Thin stack profiles | Still true: several framework-* / lang-* skills ~38–48 lines |
| Process weight | Still true: shortcuts exist; default narrative is multi-phase |
Model-agnosticism (focused answer)
| Layer | Agnostic? | Notes |
|---|---|---|
| Philosophy / SOPs / role prose | Mostly yes | Markdown procedures; model-neutral instructions |
| Host discovery & MCP | No | Cursor skill format + .cursor/mcp.json dominate |
| Multi-IDE export | Cosmetic | Pointer files ≠ equal capability |
Skill routing evals (run_evals) | N/A | No model in the loop |
| EDD live runner | Weakly | OpenAI-compatible /chat/completions; Anthropic key only via compatible gateway |
| Default CI proof | No | Scripted driver; live is nightly + secret |
Bottom line: Model-agnostic at the documentation layer; provider-shaped (OpenAI-compatible) + Cursor-centric at the runtime layer.
Value judgment (unchanged in substance)
Worth adopting when: Cursor (or manual skill mapping), shared hexagonal/TDD norms, MCP/tool agents that need a routing harness, thin always-on rules.
Weak fit when: Equal first-class Claude Code / Gemini / Copilot / Windsurf depth, out-of-the-box multi-model proof CI, architecture-flexible guidance, or a minimal prompt pack without lifecycle ceremony.
Net: High concept and structure value; medium–rising execution value (EDD and honesty improved); still not “any model / any host, same guarantees.”
Highest-leverage remaining work
Tracked in kit-review-backlog.md. Top three:
- Make skill-trigger evals assert
required_patterns/required_output_sections(copy already says they are registration / prompt hygiene, not a live model run). - Enforce role skill line budget; cut or deepen thin stack profiles.
- Surface a shorter day-to-day path (debug + light XFN) without dropping the full lifecycle when the job is a product feature.
Review stance: product/architecture assessment of the kit as shipped on main, not a PR diff review. Evidence from AGENTS.md, README.md, docs/edd.md, kit/src/edd/run_evals.ts, MCP/install paths, skill line counts, and post–Aug 30 merges (#23–#43).