Testing a Claude Code plugin against no plugin at all

A plugin can pass every test I write for it and still add nothing. If Claude would have produced the same answer without the plugin, the green tick only measures Claude. Claude Code now ships claude plugin eval, which runs each test case twice, with the plugin and without it, and reports the difference. I built a small plugin to see what that difference looks like in practice. I used Claude Code 2.1.309 on Windows 11.

The plugin under test

commit-coach is deliberately small, with one of each part I wanted to test:

• A skill, conventional-commit, that writes commit messages to a set of house rules: type(scope): summary with the scope required, the whole subject line at most 50 characters, a body that explains why, and a Closes #N footer when an issue is mentioned.

• An agent, commit-critic, that reviews an existing message against the same rules.

• An MCP server, tracker, with one get_issue tool. The skill uses the issue's labels to choose the type and scope.

The house rules are the point. Claude already writes good commit messages, so a plugin that only asks for "good commit messages" would show no difference. The tracker server does not exist: it is declared in .mcp.json, but every eval run answers it from a mock file, and by default the real server is never started.

How a case is scored

A case is a directory under evals/ with a prompt.md and one or more graders. The prompt is something a user would type, not the skill's name:

---
tags: [smoke, skill]
max_turns: 6
allowed_tools: [Read, Glob, Grep, Skill]
---

Write me a commit message for this change: I renamed getUser to fetchUser
and updated the three call sites, because "get" made people think it was a
cheap cached lookup when it actually hits the API.

Each grader is a file with the check in its frontmatter. The format check is a regex, and it is free:

---
type: regex
target: last_message
flags: m
weight: 2
pattern: '^(?=[^\n]{1,50}$)(feat|fix|refactor|docs|test|perf|build|ci|chore|style)\([a-z0-9-]+\)!?: [a-z][^\n]*[^.\n]$'
---

There are six grader types: regex, tool_used, tool_order, file_exists, llm and baseline. Only the last two call a judge model. There is no custom-code grader.

Each case runs three times per arm in isolated sessions. The plugin's contribution is Δ, not the score.

One detail matters a lot. A check such as "the skill was invoked" cannot pass without the plugin, so counting it would drag the baseline to zero and inflate Δ. The runner reports those checks as indicators and leaves them out of the score in both arms. I used arm: both on the one case where I wanted it counted: a negative case where the skill must not fire.

Four cases, one per part

• writes-house-style-commit (skill): the regex above, a tool_used: Skill indicator, and an llm rubric that checks the body explains why.

• links-tracker-issue (MCP): "this is the fix for issue 42". Graders check that the mock received "id": 42, that the subject starts with fix(auth), and that the message ends with Closes #42.

• delegates-to-critic (agent): review "Fixed stuff and also updated deps." An indicator checks that commit-coach:commit-critic was dispatched, and an llm rubric checks the review itself.

• ignores-unrelated-request (negative): a merge-versus-rebase question must not trigger the skill.

Every case uses read-only tools only. When an eval grants Bash, each command runs in Claude Code's OS sandbox, and native Windows has no sandbox backend, so those runs are refused. A suite that needs a shell belongs in WSL2.

The first run

I started with a cheap smoke run of the two smoke cases, one run each, with no baseline:

claude plugin eval . --tag smoke --runs 1 --ablation none

Both passed in 10 seconds for $0.10. Then the full suite, which is 4 cases × 3 runs × 2 arms = 24 agent runs:

claude plugin eval . -j 4 --max-cost-usd 5

It took 55 seconds with an estimated list-price cost of $1.38. Three of four cases passed, and the mean Δ was +0.58.

The baseline arm shows what the plugin adds, and also what it doesn't.

The baseline taught me more than the plugin's score did. Without the plugin, Claude wrote Rename getUser to fetchUser with a body that explained the reason well. It passed the "why" rubric in all three runs and failed the format regex in all three. So explaining why is something Claude already does, and my plugin contributes the house format. Without the tracker, Claude cannot know that issue 42 is labelled bug and auth, so fix(auth) and the footer never appeared: Δ +1.00. The negative case scored 1.00 in both arms, and for a "does not over-trigger" check, a Δ of 0 is the correct result.

The red case was the judge

delegates-to-critic failed one of its three plugin runs, and the judge, Haiku by default, voted FAIL three times. Before I changed the agent, I read the evidence the judge had been given. The review told me to hold off on pushing, said to split the fix from the dependency update, flagged the missing type(scope): prefix, and proposed fix(auth): handle expired refresh tokens along with a fix(<module>) template. It met every point in the rubric. I suspect the judge wanted the literal word "reject", or did not accept the placeholder template as a proposed message, but that is a guess. The documentation describes the same pattern: when the plugin-fired indicator passes and the score does not, suspect the judge before the plugin.

I changed the rubric, not the plugin.

I rewrote the rubric to judge substance over wording, to say that advising against the push counts as rejecting, and to accept placeholder templates. Then I re-ran only that case, once with each judge:

claude plugin eval . --case delegates-to-critic -j 4
claude plugin eval . --case delegates-to-critic -j 4 --judge-model sonnet

Both gave WITH 1.00, W/OUT 0.00, and 9 of 9 PASS votes across the plugin runs. The judge calls cost about $0.003 with Haiku and about $0.068 with Sonnet, for the same verdicts. I also checked that the baseline failure was genuine: without the plugin, Claude still caught the vague subject and the need to split, but it suggested plain Fix crash when… subjects with no type(scope). It cannot know about a house rule it has never seen.

Three runs per arm is not proof that the false negative cannot return. It is enough to show that the problem was in the grader.

What I would keep

• Give the plugin something Claude doesn't already do. Otherwise Δ will be close to zero, and it will be right.

• Use regex wherever the check can be written as a pattern. In the full run, every regex grader gave the same verdict in all three runs of each arm. All the noise came from one llm grader.

• Read the judge's evidence before you change the plugin. Otherwise you end up tuning a working agent to satisfy a broken rubric.

• Set --threshold on purpose. The default is 1.0, so any imperfect case makes the command exit 1. In CI, also pin --model and --judge-model, pass --trust-plugin, and set --max-cost-usd.

The reference for everything above is the plugin evals documentation.

The sample code: https://github.com/zakariahere/plugin-eval-claude-example