Use judge-audit

Audit whether an LLM-as-judge, or a panel of judges, can be trusted for the decision its scores drive, before anyone acts on them

Audit whether an LLM-as-judge, or a panel of judges, can be trusted for the decision its scores drive, before anyone acts on them — above all whether the judge errs differently on the variants being compared (prompt v1 vs v2, model A vs B, short vs long inputs), plus chance-corrected and weighted agreement with expert labels, precision and recall on a rare class, each run reported separately with the items that flipped, position and length bias, and confidence intervals, checked against the user's own gate or criterion and ending in a direct answer (ship or not, which judge to use).

Reach for it when the user asks "can we trust our LLM judge", "is our grader reliable enough to gate releases", "the grader says v2 is better, can we ship it", "which is the cheapest judge we can switch to", "our two judges agree 90%, is that good", "can we turn on auto-remove", "check our judge for position bias", or has a file of judge scores and wants to know whether to act on them.

Install it

npx skilldrop-cli install judge-audit                 # Claude Code, your user folder
npx skilldrop-cli install judge-audit --project       # Claude Code, this repo
npx skilldrop-cli install judge-audit --ide cursor    # or kiro, codex, antigravity, copilot

It is part of the AI engineering pack: --pack ai-engineering installs it with its neighbours.

It ships scripts, so it needs Python 3 (standard library only, nothing to install). What they may touch is declared in its manifest.

Ask for it

Say it the way you would to a colleague. Requests like these route to it:

• "Our LLM grader and a second model agree 88% on pass/fail. Is that enough to use it as a release gate?"
• "Here's a CSV of judge scores with three trials per item and some human labels. Can we trust the judge?"
• "Check our pairwise LLM judge for position bias before we pick the winning prompt"
• "Compute Cohen's kappa between our model judge and the annotators and tell me whether the judge is usable"
• "We want to swap our expensive judge model for a cheaper one. Here are both models' grades on the same 100 answers plus expert labels."

Requests like these belong to a different skill, so it shouldn't answer them:

• "We want an LLM judge to grade helpfulness on our chatbot. How should we set that up?"
• "Design an eval set and pass bar for our RAG assistant before we change the prompt"
• "Generate eval cases for this skilldrop SKILL.md so I can drop them in its evals folder"

If another skill answers instead, ask for this one by name: "use judge-audit".

Check what you get

A good result:

• The first line answers the question asked: ship or not, which judge, which arm it favours.
• The criterion is the user's; with none, no invented pass mark decides anything.
• Statistics are stratified by the compared variable before any pooled figure is quoted.
• Ordinal scores get weighted kappa, named as quadratic or linear.
• A rare class gets its count, confusion, recall and precision with intervals, not accuracy.
• Each run is reported on its own, with the flips, joined on item id.
• Every number matches the output for the subset and run it is labelled with, n included.
• What the data cannot measure is named, never estimated.

See a worked example

• summary panel

Use something else when

• Designing the eval set, grading methods and pass bar for an LLM feature — llm-eval-harness
• Generating eval cases for a skill's evals/ folder — eval-harness-generator
• Measuring how much teams use AI tools — ai-usage-report
• Inter-rater reliability for humans only, with no model in the loop — the statistics transfer, but the bias checks here assume a model

Next

• SKILL.md, the instructions the agent reads
• Its page on the site
• Related: ai-usage-report, eval-harness-generator, llm-eval-harness

Edit this page on GitHub · View source