eval-driven iteration for AI skills
Stop guessing.
Start measuring.
Define test cases, run your agent with and without a skill, have an LLM grade the output, and benchmark the results. Skill-eval turns "does this skill help?" into a number.
an actual loop, rendered live
Run
`skill-eval run` executes every eval with and without your skill loaded, capturing outputs, timing, and tokens.
Grade
A judge agent scores each output against your assertions — or local matchers decide instantly, for free.
Benchmark
Pass rates, deltas, and token burn — an honest scoreboard for what your skill is worth.
why baselines
The golden rule of skill evaluation
A good eval is one where the baseline fails more often than the with-skill run. If the bare agent sails through everything, your skill is decoration — and you can't tell the difference.
skill-eval run --baseline-onlyverifies your evals actually fail without the skill.skill-eval loop --modelschecks every agent, not just the one you use.skill-eval loop --fixfeeds the judge's critique back to the agent until it lands.
+0.51 delta · your skill carries its weight
install
Up and running in one line
brew install matt-riley/tools/skill-evalgo install github.com/matt-riley/skill-evaluator@latestThe whole tool, six commands
All commands →guides
Start here, in order
First eval
From a blank skill directory to a working loop in minutes.
Writing evals
Assertions that show whether your skill actually helps.
Reading results
Pass rates, deltas, and what the benchmark is really telling you.
Cross-model
Does your skill pull its weight in claude, copilot, codex, and pi?