Design an evaluation harness for an LLM-powered feature — a versioned golden set (representative + adversarial + regression cases), the cheapest adequate grading method per case, a metric with a pre-set pass bar and regression gate, and a failure taxonomy that targets iteration. Use when the user is building or tuning an LLM feature (prompt, RAG, agent, classifier) and needs evals, a way to test prompt/model changes, or to stop shipping quality regressions on vibes.
Pack ai-engineeringTier standardJob Build and review softwarev0.1.2
In Claude Code, with the ai-engineering plugin installed, invoke it as:
/ai-engineering:llm-eval-harness
Try it
A realistic prompt from the skill's acceptance evals:
We have a RAG assistant that answers HR policy questions for Acme's 3,000 employees from a 400-page handbook. We're about to swap the model and rewrite the system prompt. We have about 1,200 real logged questions from the last quarter, and HR has hand-labeled 80 of them with correct answers. The answer that hurts most is confidently wrong parental-leave or termination guidance. Design the eval harness so we don't ship a regression.
What a good result looks like
Golden cases are real and labeled, sourced honestly. Count per bucket stated; synthetic-input + model-graded sets are flagged as the weak evidence they are.
Each case names its grading method, and the cheapest adequate one was chosen. LLM-judge is the exception that justifies itself, not the default.
The metric and both gates (no-regression + critical-subset-100%) are set before the first run. Post-hoc bars don't count.
LLM-judges are validated against human labels and bias-controlled. An unvalidated judge score is presented as such.
A held-out split exists and the no-tuning-on-test rule is stated. Reporting the number you optimized against is the cardinal eval sin.
Cost and latency are tracked beside quality, so quality gains that blow the budget are visible.
How it's checked
Measured against plain Claude Code (2026-10-23): it met 59% of its checks, against 48% without it, and a blind judge preferred the agent without it. How skills are checked
22 assertions across 3 acceptance evals, and 8 trigger queries. Every skill's checks
What the output must do
The golden set is split into representative, adversarial/edge, and regression buckets, with a case count and sourcing stated for each
Representative cases are drawn from the 1,200 logged questions and 80 HR labels the user supplied; any synthetic or model-generated cases are flagged as weak evidence
Each bucket or case names its grading method, and programmatic or structured-assertion checks are preferred over LLM-as-judge where the output is checkable
If an LLM judge is used (e.g. for faithfulness), it has its own rubric, is validated against the HR human labels, and controls for position, verbosity, and self-preference bias
The metric, the no-regression gate against baseline, and a critical subset (parental-leave and termination answers) held at 100% are all set before any run
A held-out split is defined and the rule against tuning on the test set is stated explicitly
Cost per run and p95 latency are tracked alongside the quality metric
A failure taxonomy (e.g. hallucination, refusal, missed-edge-case) is defined so iteration targets the biggest bucket
It points out that 14 of 15 passing is not a 100% pass rate, and that 15 developer-written questions about adult use do not test the rules or the incidents
It turns each of the three logged incidents (the child's Calmexa dose, the discontinued Nasaclear, the invented Dormilene and Ferrovit interaction) into test cases that must be in the test set
It does not rely on an average judge score alone: a failure on the under-12 dosing rule (and similar safety rules) blocks release on its own, whatever the average
It does not rely on an unchecked LLM judge: it checks some rules directly (such as the 120-word limit or whether the pharmacist referral is offered), or checks the judge against pharmacists' grading, or both
It does not use the 9,400 chat-log questions until redaction is approved, and says what to use in the meantime
It does not invent incidents, products or figures that are not in the files
It is no longer than two pages
It uses her 40 marked essays as the answer key: run the GPT on them and compare its marks with hers
It says to decide in advance how close counts as good enough (for example within a stated number of marks on most essays) rather than eyeballing the results
It checks for specific kinds of unfairness, such as marks rising with essay length or doing worse on particular types of essay or student, not only the overall average
It says to retest on the same essays whenever she changes the GPT's instructions
It keeps the teacher in charge of the final report grades, for example by checking the GPT's marks before they count
It does not invent results or statistics about her essays
It is a few bullets in plain language
When it should and shouldn't activate
Should activate:
How do we test this prompt change without shipping a quality regression?
Build an eval set and pass bar for our ticket-classification LLM
We want an LLM judge to grade helpfulness on our chatbot — how should we set that up?
Set up a regression gate before we keep iterating on our RAG prompts
Should leave to another skill:
Generate eval cases for this skilldrop SKILL.md so I can drop them in its evals folder
Summarize how much each team used AI tools last month from this telemetry
Compare Claude, GPT, and Gemini for our document-extraction use case and recommend one
What adoption and handle-time metrics should we track for the new AI feature?