GitHub

Agent Eval Report: Architecture Routing

Date: 2026-08-30 Model: claude-3-5-sonnet-latest Overall Pass Rate: 95.0% (38/40)

Performance Metrics

  • Total Tokens: 14,205 (approx. $0.04)
  • Average Latency: 840ms
  • Routing Accuracy: 95.0%
  • Schema Adherence: 100.0%

Failure Traces

Test ID: route-08

Tags: edge-case, routing

  • Prompt: "What is the database for the payment system?"
  • Expected Tool: read_architecture_yaml
  • Actual Tool: None (Conversational Response)
  • LLM Output: "I don't have access to your database, but typically payment systems use PostgreSQL..."
  • Diagnosis: Tool Selection Failure. The model refused to use the tool and hallucinated a generic answer.
  • Suggested Fix: Add a constraint to the system prompt instructing the agent to never guess architectural details and to always use the provided C4 tools.

Test ID: schema-03

Tags: extraction, schema

  • Prompt: "Check the architecture for the auth service and the payment api."
  • Expected Arguments: {"componentId": "auth-service"}
  • Actual Arguments: {"componentId": ["auth-service", "payment-api"]}
  • Diagnosis: Schema Violation. The tool only accepts a string, but the model attempted to pass an array to handle the multi-intent prompt.
  • Suggested Fix: Update the tool description to explicitly state that it can only be called for one component at a time, or update the tool's backend logic to accept arrays.

This file is the canonical example of kit eval report --format md --out out/reports output (out/reports/eval-report.md, mirrored as edd-report.md). Live runs regenerate the same structure from suite results.

Markdown source