Use eval-harness-generator

Takes a skill's SKILL.md and generates a set of eval cases

Takes a skill's SKILL.md and generates a set of eval cases — trigger queries, expected output shape, and pass/fail criteria — ready to drop into the skill's evals/ directory.

Reach for it when adding evals to a new or existing skill, verifying a skill behaves as documented, or building a regression gate before modifying a skill.

Install it

npx skilldrop-cli install eval-harness-generator                 # Claude Code, your user folder
npx skilldrop-cli install eval-harness-generator --project       # Claude Code, this repo
npx skilldrop-cli install eval-harness-generator --ide cursor    # or kiro, codex, antigravity, copilot

It is part of the API builder pack: --pack api-builder installs it with its neighbours.

Ask for it

Say it the way you would to a colleague. Requests like these route to it:

• "Write eval cases for this SKILL.md so I can put them in its evals folder"
• "Generate trigger queries and pass/fail criteria for the release-notes skill"
• "I'm about to edit a skill — build me a regression set from its SKILL.md first"
• "Check this skill behaves as documented: draft eval cases covering its anti-patterns"

Requests like these belong to a different skill, so it shouldn't answer them:

• "Design an eval harness for our customer-support chatbot's RAG pipeline"
• "Help me author a brand-new skilldrop skill from scratch, manifest and all"
• "We need a golden set and LLM-judge rubric for our summarization feature in production"

If another skill answers instead, ask for this one by name: "use eval-harness-generator".

Check what you get

A good result:

• Triggers must be realistic. A trigger of "run this skill" is not an eval case — it doesn't test anything specific. Each trigger should be the kind of input a real user would paste.
• expected_shape items must be checkable. "Output is good" is not checkable. "Output contains a numbered repro sequence starting from a clean state" is.
• Cover the refusal cases. The most common gap in skilldrop evals is missing cases for the skill's anti-patterns — the bad outputs the quality bar explicitly forbids. At least 2 of the 8–12 cases must test that the skill refuses or avoids its stated failure mode.
• Don't pad to 12. Eight strong cases beat twelve weak ones. Only add cases that test a distinct failure mode not already covered.

Next

• SKILL.md, the instructions the agent reads
• Its page on the site
• Related: llm-eval-harness

Edit this page on GitHub · View source