Use llm-eval-harness

Design an evaluation harness for an LLM-powered feature

Design an evaluation harness for an LLM-powered feature — a versioned golden set (representative + adversarial + regression cases), the cheapest adequate grading method per case, a metric with a pre-set pass bar and regression gate, and a failure taxonomy that targets iteration.

Reach for it when the user is building or tuning an LLM feature (prompt, RAG, agent, classifier) and needs evals, a way to test prompt/model changes, or to stop shipping quality regressions on vibes.

Install it

npx skilldrop-cli install llm-eval-harness                 # Claude Code, your user folder
npx skilldrop-cli install llm-eval-harness --project       # Claude Code, this repo
npx skilldrop-cli install llm-eval-harness --ide cursor    # or kiro, codex, antigravity, copilot

It is part of the AI engineering pack: --pack ai-engineering installs it with its neighbours.

Ask for it

Say it the way you would to a colleague. Requests like these route to it:

• "How do we test this prompt change without shipping a quality regression?"
• "Build an eval set and pass bar for our ticket-classification LLM"
• "We want an LLM judge to grade helpfulness on our chatbot — how should we set that up?"
• "Set up a regression gate before we keep iterating on our RAG prompts"

Requests like these belong to a different skill, so it shouldn't answer them:

• "Generate eval cases for this skilldrop SKILL.md so I can drop them in its evals folder"
• "Summarize how much each team used AI tools last month from this telemetry"
• "Compare Claude, GPT, and Gemini for our document-extraction use case and recommend one"

If another skill answers instead, ask for this one by name: "use llm-eval-harness".

Check what you get

A good result:

• Golden cases are real and labeled, sourced honestly. Count per bucket stated; synthetic-input + model-graded sets are flagged as the weak evidence they are.
• Each case names its grading method, and the cheapest adequate one was chosen. LLM-judge is the exception that justifies itself, not the default.
• The metric and both gates (no-regression + critical-subset-100%) are set before the first run. Post-hoc bars don't count.
• LLM-judges are validated against human labels and bias-controlled. An unvalidated judge score is presented as such.
• A held-out split exists and the no-tuning-on-test rule is stated. Reporting the number you optimized against is the cardinal eval sin.
• Cost and latency are tracked beside quality, so quality gains that blow the budget are visible.

Use something else when

• Tabulating AI usage/adoption telemetry — that's ai-usage-report
• Product-outcome metrics for a feature (adoption, handle time) — that's success-metrics
• Picking which model/vendor to buy on price, terms or support — this measures quality on your task, which is one input to that decision
• The provider-specific API mechanics (token counting, streaming, tool-call format) — consult the provider's own reference; this skill is the eval methodology

Next

• SKILL.md, the instructions the agent reads
• Its page on the site
• Related: ai-usage-report, success-metrics

Edit this page on GitHub · View source