What does this skill do?

The Skill Code Agent Evaluator is a lightweight command-line tool for comparing programming agents such as Claude Code, Aider, or Codex on your own repository. Define tasks in YAML, run each agent in an isolated Git worktree, and generate a comparative report with accuracy, cost, time, and consistency metrics to support adoption decisions with objective data.

Comparison Before Adopting
Compare several programming frameworks using your actual code before deciding which one to adopt.
Measurement after the update
Measures an agent's performance following a model or tool update.
Configuration Regression
Run regression tests when an agent changes its configuration or behavior.
Data-Driven Team Decision-Making
Support the selection of a provider with comparative metrics for cost, speed, and reliability.

Usage examples

🆚 Basic Comparison
Compare Claude Code and Aider on the "add-retry-logic" task, running each three times, and generate the report.
📈 After updating the model
Check whether Aider is still performing the same after the latest update by running the tasks in the `tasks/` directory 5 times.
🔄 Configuration rollback
Run the 3 representative tasks from my repo against claude-code to verify that they still pass all the judges.
📋 Team Decision
Generate a comparative report on Claude Code, Aider, and Codex based on five tasks from my actual workload to help me decide which one to adopt.

Features

Tasks Definable in YAML Specify the prompt, relevant files, base commit, and evaluation criteria in a separate file for each task.
Multi-agent execution Run multiple agents on the same tasks with a minimum of 3 executions to capture variability.
Isolation with Git worktree Each run executes in a separate worktree without the need for Docker, ensuring reproducibility.
Comparative Table Report Generate a table showing accuracy rate, cost, time, and consistency by agent and task.
Three Types of Judges It combines deterministic tests (pytest, command), pattern-based tests (grep), and LLMs for semantic evaluations.

Frequently asked questions

No. The tool creates a separate Git worktree for each execution, which ensures isolation without Docker.
Any programming tool that provides a command-line interface: Claude Code, Aider, Codex, and similar tools.
At least 3 per agent to account for variability. Start with 3–5 tasks that are representative of your actual workload.
Three types: deterministic (pytest, npm run build), pattern-based (grep on files), and LLM as a semantic judge.
Code Agent Evaluator — Comparison of AI Agents with Claude AI

¿Prefieres escuchar el contenido? Genera la narración de audio con un clic.