Takes a skill's SKILL.md and generates a set of eval cases — trigger queries, expected output shape, and pass/fail criteria — ready to drop into the skill's evals/ directory. Use when adding evals to a new or existing skill, verifying a skill behaves as documented, or building a regression gate before modifying a skill.
Pack api-builderTier standardJob Build and review software, Build on the Claude APIv1.0.3
In Claude Code, with the api-builder plugin installed, invoke it as:
/api-builder:eval-harness-generator
Try it
A realistic prompt from the skill's acceptance evals:
Here's the full SKILL.md for our new skilldrop skill `vendor-risk-screen` (a gate skill that reads a vendor security questionnaire and emits PROCEED, REVISE, or BLOCKED; its quality bar says it must never pass a vendor that has no SOC 2 report or equivalent, and must never invent certifications not in the questionnaire). [SKILL.md pasted below] Generate the eval cases for its evals/ folder.
What a good result looks like
Triggers must be realistic. A trigger of "run this skill" is not an eval case — it doesn't test anything specific. Each trigger should be the kind of input a real user would paste.
expected_shape items must be checkable. "Output is good" is not checkable. "Output contains a numbered repro sequence starting from a clean state" is.
Cover the refusal cases. The most common gap in skilldrop evals is missing cases for the skill's anti-patterns — the bad outputs the quality bar explicitly forbids. At least 2 of the 8–12 cases must test that the skill refuses or avoids its stated failure mode.
Don't pad to 12. Eight strong cases beat twelve weak ones. Only add cases that test a distinct failure mode not already covered.
How it's checked
Measured against plain Claude Code (2026-10-23): it met 63% of its checks, against 51% without it, and a blind judge preferred the agent without it. How skills are checked
21 assertions across 3 acceptance evals, and 7 trigger queries. Every skill's checks
What the output must do
Produces 8 to 12 cases, each with id, trigger, expected_shape, must_not_include, verdict_class, and notes
Every trigger is a realistic input a user would paste (e.g. a specific questionnaire excerpt), not 'run this skill'
Every expected_shape item is checkable — it names a specific section, field, or behavior, not 'output is good'
At least 2 cases test the quality bar's failure modes, including a vendor with no SOC 2 report that must not get PROCEED
Because the skill is a gate, verdict_class values come from contracts/terminals.json and at least one case expects revise or blocked
At least one must_not_include entry forbids certifications or facts not present in the input questionnaire
A 3 to 5 sentence eval strategy note follows the JSON, naming the most important case and what a regression on it would mean
Ends with the placement reminder for packs/<pack>/skills/vendor-risk-screen/evals/ and running python3 validate.py
Saves evals.json with exactly 6 cases in the requested shape (skill_name plus an evals array of id, prompt and assertions), and the file parses as JSON
Flags that the description promises PASS or FAIL while the body defines COMPLIANT, NEEDS CHANGES and REJECT, and does not quietly build cases on one of them without saying so
Flags that the indirect-cost limit conflicts (10% of direct costs in the rules, 15% of the total in the quality bar)
The cases cover both outcomes: at least one budget that breaks a funder rule and must not be passed, and at least one clean budget that should be passed
Each prompt contains concrete budget lines and amounts a user would actually paste, not 'run the skill on a budget'
Each assertion is a checkable statement about one output, not 'the output is good'
Does not give made-up expected pass rates (it explains they can only come from running the cases), and does not invent funder rules beyond those in the SKILL.md
Gives exactly 3 tests with little or no explanation around them
Each test includes a concrete pattern excerpt with stated stitch counts, not a vague instruction
At least one test has a planted stitch-count mistake the skill should catch, and at least one has correct counts and should be left alone
The stitch counts in the tests, and the expected answers, are arithmetically correct under the file's rules
Does not add checks the file doesn't ask for, such as gauge, yarn or style comments
Delivers the tests without first asking questions
When it should and shouldn't activate
Should activate:
Write eval cases for this SKILL.md so I can put them in its evals folder
Generate trigger queries and pass/fail criteria for the release-notes skill
I'm about to edit a skill — build me a regression set from its SKILL.md first
Check this skill behaves as documented: draft eval cases covering its anti-patterns
Should leave to another skill:
Design an eval harness for our customer-support chatbot's RAG pipeline
Help me author a brand-new skilldrop skill from scratch, manifest and all
We need a golden set and LLM-judge rubric for our summarization feature in production