How skills are checked

Every skill ships two eval files. Acceptance evals are realistic prompts with assertions the output must meet, taken from the skill's own quality bar. Trigger queries are phrasings that should activate the skill, and near-misses that belong to a named sibling instead.

Skills with both files: 113 of 113
Acceptance evals: 508, with 3582 assertions
Trigger queries: 950
Run: Benchmarked 2026-10-23, by hand

validate.py fails any skill without both files, on every pull request. The acceptance evals are run by hand with run_bench.py, as agent sessions with and without each skill; the latest result is below. A scheduled workflow asks a model each week which skill it would load for every trigger query, and once a month runs the acceptance evals under a budget; it is switched off until the repository has an API key. A miss is a prompt to look at the description, not a gate. Run them yourself with python3 run_evals.py and an ANTHROPIC_API_KEY.

The site itself is checked with the catalogue's own skills: its performance and accessibility reviews, and what was fixed.

Measured: with the skill and without it

Each acceptance eval is run twice on the same model, once with the skill and once without, as a Claude Code agent session with file and shell tools, sandboxed in an empty directory. Last run 2026-10-23, judged by claude-sonnet-5-5, 1 trial per eval.

Assertions met with the skill: 73%
Without it: 53%
Lift, points: +20 (+18 to +23)
Preferred by a blind judge: 62% (58% to 67%)
Second judge: 61% (57% to 65%), same side as the first in 398 of 508
Cost per run: $0.14 with, $0.09 without
Model: claude-sonnet-5, 520 evals

Evals	How many	Assertions met with the skill	Without	Preferred blind	Second judge
Written by the skill’s author	212 evals, 117 skills	72%	35%	69% (62% to 75%)	68% (62% to 74%)
Written without seeing the skill	308 evals, 117 skills	74%	66%	58% (52% to 63%)	56% (50% to 61%)

Assertions met is graded by a judge that doesn't know which run it is reading. The skill's author wrote the assertions, so the run without the skill is graded on a rubric it never saw. A non-answer meets 13% of them.
Preferred by a blind judge is the share of pairs where a judge that saw only the request and both results, in shuffled order, picked the skill's. 50% means no preference.
Second judge. A different model, claude-opus-5-5, judged the same pairs in the same order. The two picked the same side in 398 of 508 pairs where both chose one, so a single pair's verdict is weak evidence and the percentage across all pairs is the result. Both judges are models from one family and can share blind spots; no human has checked them.
Ranges are 95% intervals over evals. A lift range that includes 0, or a preference range that includes 50%, is not a result.
Most skills have one eval, so a single row below is an anecdote. The judge is a model. Cost is list price for the tokens used.
Skills that lost were revised after reading the judges' reasons and run again on the same evals, so part of the preference on evals their author wrote is fitted to them. Evals written by someone who never read the skill are counted separately above. Some skills were later revised, or admitted, on the verdicts from those evals too, so that row is the better guide but not a clean one; what the benchmark showed separates the evals nothing was decided on.

Reproduce it with python3 run_bench.py (RFC-0040).

Week by week

No scheduled run yet. The weekly trigger check and the monthly assertion run start once the repository has an ANTHROPIC_API_KEY secret; each run then adds a point here.

Every skill

Skill	Pack	Evals	Assertions	Met with skill	Without	Blind judge preferred	Should / shouldn't trigger	Source
`agent-adoption-stage`	ai-engineering	3	22	86%	43%	skill	4 / 4	files
`agent-budget`	ai-engineering	4	24	89%	51%	tie	4 / 4	files
`agent-config-review`	ai-engineering	5	38	54%	26%	skill	5 / 5	files
`agent-incident-response`	ai-engineering	7	56	52%	41%	skill	6 / 6	files
`agent-loop-design`	ai-engineering	4	25	77%	29%	no skill	4 / 4	files
`agent-threat-model`	ai-engineering	5	31	90%	54%	no skill	6 / 6	files
`ai-adoption-rollout`	ai-engineering	3	22	73%	48%	skill	4 / 4	files
`ai-readiness-assessment`	ai-engineering	3	20	75%	38%	no skill	4 / 4	files
`ai-usage-policy`	ai-engineering	7	54	77%	58%	skill	4 / 4	files
`ai-usage-report`	ai-engineering	3	21	57%	53%	no skill	4 / 3	files
`ai-use-case-triage`	ai-engineering	3	20	94%	52%	skill	4 / 4	files
`judge-audit`	ai-engineering	11	86	51%	36%	skill	5 / 4	files
`llm-eval-harness`	ai-engineering	3	22	59%	48%	no skill	4 / 4	files
`mcp-pairing`	ai-engineering	5	39	92%	39%	skill	5 / 5	files
`subagent-design`	ai-engineering	4	25	78%	51%	no skill	4 / 4	files
`eval-harness-generator`	api-builder	3	21	63%	51%	no skill	4 / 3	files
`prompt-caching-advisor`	api-builder	3	19	78%	66%	no skill	4 / 3	files
`token-budget-estimator`	api-builder	3	20	76%	52%	skill	4 / 3	files
`tool-use-schema-writer`	api-builder	5	38	87%	79%	no skill	4 / 3	files
`application-form-answers`	career	7	54	95%	63%	skill	4 / 4	files
`cover-letter`	career	7	51	98%	89%	skill	4 / 4	files
`job-fit-analysis`	career	7	53	90%	79%	skill	4 / 4	files
`resume-tailor`	career	9	72	96%	86%	skill	4 / 4	files
`role-shortlist`	career	7	54	76%	67%	skill	4 / 4	files
`file-to-markdown`	converters	6	39	66%	39%	no skill	5 / 4	files
`md-to-confluence`	converters	5	40	88%	32%	skill	5 / 5	files
`md-to-docx`	converters	4	26	66%	53%	no skill	4 / 4	files
`md-to-pptx`	converters	7	55	60%	55%	skill	5 / 5	files
`md-to-xlsx`	converters	4	23	91%	57%	skill	4 / 3	files
`brief-intake`	core	3	18	69%	58%	no skill	4 / 4	files
`context-graph`	core	17	134	63%	55%	skill	7 / 5	files
`council-review`	core	3	21	67%	48%	no skill	5 / 4	files
`doc-critique`	core	3	19	84%	63%	no skill	5 / 4	files
`output-hygiene`	core	4	26	55%	38%	no skill	5 / 4	files
`dashboard-spec`	data-analytics	3	21	77%	55%	no skill	4 / 4	files
`sql-review`	data-analytics	3	21	75%	65%	no skill	4 / 4	files
`brand-kit`	design	3	19	95%	40%	skill	4 / 3	files
`slide-outliner`	design	3	17	64%	75%	no skill	4 / 4	files
`accessibility-audit`	dev-team	3	21	72%	41%	skill	4 / 3	files
`agents-md-generator`	dev-team	6	34	68%	39%	tie	6 / 6	files
`behaviour-equivalence`	dev-team	5	39	69%	51%	skill	5 / 4	files
`bug-triage`	dev-team	3	19	58%	41%	no skill	4 / 4	files
`codebase-tour`	dev-team	5	40	85%	45%	skill	5 / 5	files
`dependency-upgrade-review`	dev-team	7	59	65%	41%	skill	5 / 4	files
`devils-advocate`	dev-team	7	51	69%	63%	no skill	5 / 4	files
`feature-implement-loop`	dev-team	3	21	68%	44%	skill	4 / 3	files
`launch-readiness`	dev-team	3	21	79%	51%	skill	4 / 4	files
`migration-plan`	dev-team	3	18	68%	53%	no skill	4 / 4	files
`pr-description-writer`	dev-team	5	33	82%	44%	skill	4 / 3	files
`pre-merge-review`	dev-team	4	23	44%	43%	no skill	5 / 4	files
`refactor-plan`	dev-team	7	55	57%	38%	skill	5 / 6	files
`release-notes`	dev-team	7	52	86%	67%	skill	4 / 3	files
`rollout-plan`	dev-team	7	58	73%	27%	skill	5 / 5	files
`sonar-onboard`	dev-team	3	18	87%	67%	skill	4 / 4	files
`sonar-review`	dev-team	3	18	52%	52%	skill	4 / 4	files
`tech-debt-register`	dev-team	5	36	86%	70%	skill	4 / 4	files
`test-plan-generator`	dev-team	3	18	66%	35%	skill	4 / 4	files
`user-story-splitter`	dev-team	3	18	68%	36%	skill	4 / 4	files
`content-design`	experience-design	3	20	100%	75%	skill	4 / 4	files
`design-system-spec`	experience-design	3	21	79%	64%	no skill	4 / 4	files
`information-architecture`	experience-design	3	21	83%	42%	skill	4 / 4	files
`service-blueprint`	experience-design	3	21	81%	61%	skill	4 / 3	files
`ux-writing`	experience-design	3	21	81%	60%	no skill	4 / 4	files
`bundle-size-review`	frontend	9	73	53%	40%	skill	6 / 4	files
`rendering-strategy`	frontend	5	40	78%	45%	skill	5 / 4	files
`web-performance-review`	frontend	5	39	59%	46%	skill	5 / 4	files
`dpia`	grc	3	21	90%	82%	skill	4 / 4	files
`risk-register`	grc	3	21	76%	56%	skill	4 / 4	files
`security-questionnaire-response`	grc	5	33	78%	60%	skill	4 / 4	files
`soc2-evidence-map`	grc	3	21	57%	47%	skill	4 / 4	files
`terraform-module`	infra-as-code	4	27	92%	77%	skill	4 / 4	files
`terraform-plan-review`	infra-as-code	4	26	74%	71%	skill	4 / 4	files
`business-case`	product-manager	3	21	81%	55%	skill	4 / 3	files
`feedback-synthesis`	product-manager	5	39	61%	38%	skill	5 / 6	files
`lean-canvas`	product-manager	5	39	74%	28%	skill	5 / 5	files
`okr-cascade`	product-manager	4	24	71%	67%	tie	5 / 4	files
`prd-draft`	product-manager	3	19	58%	62%	no skill	4 / 4	files
`prfaq`	product-manager	4	27	79%	62%	no skill	5 / 5	files
`requirements-interview`	product-manager	3	18	82%	53%	skill	4 / 4	files
`strategy-analysis`	product-manager	5	26	66%	35%	no skill	6 / 4	files
`success-metrics`	product-manager	4	24	75%	61%	skill	4 / 4	files
`user-journey-map`	product-manager	4	25	80%	58%	no skill	5 / 4	files
`hypothesis-comparison`	research	3	21	69%	45%	skill	4 / 4	files
`research-plan`	research	4	25	63%	45%	skill	4 / 4	files
`source-synthesis`	research	7	54	74%	67%	skill	4 / 4	files
`skill-author`	skill-engineering	3	21	72%	52%	no skill	4 / 4	files
`skill-review`	skill-engineering	3	21	90%	74%	skill	5 / 4	files
`adr-generator`	solution-architect	3	19	74%	62%	no skill	4 / 4	files
`api-contract-draft`	solution-architect	3	20	85%	40%	skill	4 / 4	files
`architecture-diagrams`	solution-architect	3	18	89%	58%	skill	4 / 4	files
`data-contract`	solution-architect	3	18	67%	40%	skill	4 / 4	files
`design-doc`	solution-architect	7	50	85%	65%	no skill	4 / 4	files
`event-contract-draft`	solution-architect	7	55	71%	39%	skill	5 / 4	files
`figma-diagrams`	solution-architect	3	16	69%	61%	no skill	4 / 4	files
`interactive-diagram`	solution-architect	13	101	79%	59%	skill	6 / 4	files
`nfr-spec`	solution-architect	3	18	66%	45%	skill	4 / 4	files
`reverse-architecture`	solution-architect	3	19	59%	48%	no skill	4 / 4	files
`threat-model`	solution-architect	3	21	86%	57%	skill	4 / 4	files
`capacity-cost-model`	sre-oncall	3	22	87%	57%	skill	4 / 4	files
`cloud-cost-review`	sre-oncall	4	26	48%	30%	skill	4 / 4	files
`delivery-metrics-report`	sre-oncall	4	24	74%	37%	no skill	4 / 4	files
`incident-comms`	sre-oncall	3	18	77%	65%	tie	4 / 4	files
`observability-plan`	sre-oncall	3	18	62%	45%	skill	4 / 4	files
`postmortem-generator`	sre-oncall	7	51	72%	56%	skill	4 / 4	files
`runbook-generator`	sre-oncall	3	19	94%	83%	no skill	4 / 4	files
`slo-spec`	sre-oncall	5	38	61%	42%	skill	5 / 4	files
`audience-profile`	stakeholder-comms	3	16	79%	44%	no skill	4 / 4	files
`decision-log`	stakeholder-comms	3	18	57%	62%	skill	4 / 4	files
`exec-summary`	stakeholder-comms	3	18	47%	57%	skill	4 / 4	files
`guide-builder`	stakeholder-comms	7	51	81%	63%	skill	4 / 4	files
`backlog-triage`	trackers	4	26	75%	59%	skill	4 / 4	files
`team-status-report`	trackers	4	27	83%	42%	skill	4 / 3	files
`tracker-brief-sync`	trackers	4	28	84%	55%	skill	4 / 4	files

Retired skills

A skill that adds only a format, and no rule the agent breaks without it, is removed rather than revised again (RFC-0046). An installed copy keeps working; skilldrop outdated says it was retired and why. The published run above was made before they left, so its totals include them.

Skill	Retired in	Why
`contribution-wizard`	0.16.13	It prescribed a skill layout this catalogue no longer uses and imposed it on other repos. To add a skill to skilldrop, follow AGENTS.md and run validate.py.
`db-schema-design`	0.16.13	On evals it was never revised against, the agent without it met as many checks (73% against 71%) and got types, time zones and constraints right unprompted.
`deck-builder`	1.2.0	On its held-out evals the agent without it built the better deck in each of two trials; rebuilt rule-first, with its script able to run, it still lost on fresh evals (judges 2 and 1 of 4; 69% of checks against 81%).
`marketing-flyer`	1.2.0	On its held-out evals, in each of two trials and with both blind judges, the agent without it built the whole flyer with placeholders where the skill stopped or used a generic template; it adds no rule the agent breaks.
`md-to-html`	1.2.0	On its held-out evals, in each of two trials and with both blind judges, the agent without it made the better page: the skill's script showed raw HTML as text and its security policy blocked embeds, and it adds no rule the agent breaks.
`mermaid-render`	1.2.0	On its held-out evals, in each of two trials and with both blind judges, the agent without it fixed the same diagrams, ran the Mermaid CLI itself and said plainly what it had not rendered; the skill adds no rule it breaks.
`metric-definition`	0.16.13	On evals it was never revised against, the agent without it did everything the skill teaches unprompted and won 13 of 16 blind verdicts.
`tech-comparison-matrix`	0.16.13	On evals it was never revised against, the agent without it did as well, and its weighted totals were wrong in 4 of 8 runs.