judge-audit

• The first lines say judge-b should not be relied on as the release gate as things stand, and name that gate as the decision
• It does not accept the 91% agreement as evidence: it gives the chance-corrected figure (Cohen's kappa about 0.54, with about 81% agreement expected by chance) instead of raw agreement alone
• It compares the judges with the support lead's 40 labels and reports that judge-b passes 10 of the 12 replies she failed (about 83%), with an interval or a note that 12 failures is a small base
• It translates the gate: on her 40 replies the judges pass about 88% while she passes 70%, so an 85% judge pass rate does not mean 85% of replies are sendable
• It reports judge-b's flip rate between trials (12 of 80, about 15%) against judge-a's (3 of 80) and says this matters for choosing the cheaper judge
• It points out that the two judges share errors (both passed the same long replies R29–R37 that the support lead failed), so their agreement with each other does not validate them
• It notes that passes rise with reply length while the support lead's do not, and recommends judging policy correctness as a separate criterion to gate on instead of one 'good reply' score
• It does not invent figures, names, items or labels that are not in the files
• The first lines say the grader result is not enough to ship v2 on
• It finds the grader picks whichever email is shown first about 70% of the time (67 of 96 decided comparisons)
• It finds that v2 was shown first in 60 of the 96 decided comparisons despite the randomisation, so the raw 55% win rate is inflated by position
• It gives an order-balanced figure of about 50% (v2 wins about 70% when shown first and about 31% when shown second), so the data shows no evidence v2 is better
• It reports that the grader's pick changed between the two runs on about half the profiles (26 of 50)
• It recommends judging each pair in both orders and counting a win only when it holds in both, or an equivalent fix, and says that an interval on the win rate (computed over the 50 profiles, not the 100 judgements) cannot remove the order effect
• It notes there are no human labels, so even a consistent grader has not been shown to pick the better email
• It does not invent numbers, profiles or human judgements that are not in the files
• The first line or two say no, not for bonus decisions
• It corrects the 90% for chance: with each judge saying good on 17 of 20, about 75% agreement is expected by chance, giving a kappa of about 0.6
• It says 20 replies is too few to conclude, for example that the uncertainty on the agreement figure runs from near zero to near perfect
• It says the two judges agreeing does not show either is right, and that their grades need comparing with a sample graded by a person such as a team lead or QA reviewer
• It ties the answer to the decision: grades that affect individual people's bonuses need a much higher bar than tracking an overall trend
• It stays short: a few lines or bullets, no long report or tables
• It does not invent results, reply counts or statistics beyond the counts given
• Answers the actual question up front with a clear verdict: the grader should NOT be relied on to gate the v2 rollout as things stand (do-not-rely, or use-with-conditions where the conditions rule out using it alone for this v2-vs-v1 call). It does not endorse shipping on the grader's say-so.
• Shows that the pooled quadratic-weighted kappa (about 0.85 on the 32 labelled rows) hides a split by version: the grader matches Priya exactly on 14 of 16 v1 rows but only 7 of 16 v2 rows, and on v2 it scores higher than her on 9 of 16 rows and lower on none (mean grader-minus-Priya +0.625 on v2 vs 0.00 on v1).
• Spots the length confound: v2 replies are much longer (mean about 201 words vs about 80 for v1), and the grader's over-scoring grows with reply length (e.g. grader-minus-human error correlates positively with reply_words, r about 0.6 on the labelled rows), so the +0.77 average gain may be the grader rewarding length rather than quality.
• Points out that on the human-labelled rows Priya's own scores show no v2 improvement (mean about 3.25 for v2 vs about 3.44 for v1), and notes that with only 16 labels per version this human difference is itself very uncertain (a CI that spans zero), so it neither proves v2 better nor proves it worse.
• Mentions what this data cannot measure: there is a single grader run (no test-retest / flip rate) and a single human labeller (no human-human baseline), and suggests checking these rather than claiming them.
• Gives concrete next steps that fit the Thursday deadline, e.g. hand-label a fresh random sample of v2 replies (especially long ones), re-score with a rubric or prompt that controls for length, or keep a human check on the gate, rather than a generic 'collect more data'.
• Is sized to a quick 'am I good?' sanity check: leads with the verdict and the two or three numbers that drive it, without a long tutorial on kappa theory.
• Invents nothing: every number, name and file it cites comes from grades.csv or the message (no made-up grader model name, extra labellers, columns, or statistics that the data doesn't support).
• Answers both questions directly: the judges, as this experiment was run, are not trustworthy enough to declare B the winner, and B's win is not established (verdict do-not-rely, or use-with-conditions where the condition is a position-balanced rerun). It does not tell the review that B is a real winner.
• Catches the order imbalance in the first_shown field: B was shown first on 22 of the 30 questions (88 of the 120 judgements), and each question kept the same order in both trials, so B's win rate is confounded with position.
• Quantifies the position bias from the data: the first-shown answer won 90 of 120 judgements (75%; orca-judge-large 50/60, wren-judge-small 40/60), and B won only 6 of 32 judgements when A was shown first versus 67 of 88 when B was shown first.
• Reports chance-corrected agreement between the two judges rather than the raw ~65% (39 of 60 paired verdicts): Cohen's kappa about 0.33 pooled (about 0.38 in trial 1, 0.29 in trial 2), or Fleiss' kappa about 0.32 across all four ratings, and calls that fair or weak agreement, not strong.
• Reports test-retest instability from the two same-order trials: orca-judge-large changed its verdict on 8 of 30 questions and wren-judge-small on 11 of 30.
• Treats the 30 questions, not the 120 judgements, as the unit of analysis (judgements on the same question are not independent), so any confidence interval on B's win rate is clustered or bootstrapped by question, and it explains that even an interval excluding 50% would not fix the position confound.
• Recommends a specific fix: rerun every question in both orders (A-first and B-first), count a win only when it holds in both orders (flips count as ties), and add some expert or human labels, noting that the file has no ground-truth labels so agreement with experts cannot be checked yet.
• Invents nothing: every count, rate and name it cites comes from pairwise_judgements.jsonl or the message; it does not make up expert labels, extra judges, or results of a rerun that hasn't happened.
• Answers the actual question directly: recommends atlas-m (not atlas-nano) as the cheapest judge to switch to, with conditions, and says plainly that Dmitri's '80% agreement' argument does not justify nano.
• Reports chance-corrected agreement with Priya's 60 expert labels for each judge, matching the data to within about 0.02: Cohen's kappa ~0.80 for atlas-xl, ~0.72 for atlas-m, ~0.37-0.38 for atlas-nano (raw accuracy 55/60, 53/60, 47/60).
• Explains why raw agreement misleads here: 42 of the 60 expert-labelled pairs (70%) are relevant, so always answering 'relevant' already scores 70%; nano's ~78% accuracy and its ~80% raw agreement with xl (kappa ~0.42 over all 300 pairs) sit close to that baseline.
• Identifies nano's failure mode: it over-labels 'relevant' (12 of the 18 expert-irrelevant pairs judged relevant), and the false positives are concentrated on long passages (10 of 11 long irrelevant passages vs 2 of 7 short ones, using the ~400-token split in passage_tokens).
• Connects that length bias to the decision: the planned Q1 re-chunk to ~900-token chunks would be judged by a model that favours long passages, so nano would inflate the new retriever's precision@5 and could wave it through the gate.
• States the uncertainty from a 60-label (only 18 irrelevant) sample - e.g. wide confidence intervals on kappa or accuracy - and says the xl-vs-m difference is not clearly distinguishable on this data; recommends a concrete next step such as more expert labels (especially irrelevant and long passages) before or alongside the switch.
• Notes what the data cannot show rather than asserting it: each judge ran once (no test-retest or flip rate), and the expert labels come from a single annotator; does not report a position-bias result for these pointwise judgments.
• Invents nothing not in the input: no fabricated judge names, prices, label counts, extra annotators or files; any cost figures follow from the README (e.g. roughly $96/night for xl vs $22 for m and $3.60 for nano at 20k judgments).
• Answers the actual question directly: no, do not turn on auto-remove on this evidence (a 'do not rely on it' verdict for auto-removal), optionally offering a narrower use such as ranking or routing posts to human review.
• Points out that only 5 of the 100 posts are violations, so a judge that flags nothing scores 95% - the 96-97% accuracy (97% run 1, 96% run 2) is barely above that baseline.
• Reports the per-run confusion against the gold labels: each run catches 3 of 5 violations (recall 60%); run 1 has 1 false positive (precision 3/4), run 2 has 2 (precision 3/5); chance-corrected agreement with gold is moderate (Cohen's kappa ~0.65 run 1, ~0.58 run 2).
• Joins run_1.jsonl and run_2.jsonl on the post ID (item_id vs id, flagged true/false vs verdict VIOLATION/OK) rather than by line order - run 2 is in a different row order - and gets test-retest agreement of 95/100 with Cohen's kappa ~0.42; does not report the near-zero kappa (~ -0.05) that a line-by-line comparison would produce.
• Shows the instability where it matters: 5 posts flip between runs, including 2 real violations (fp-35782 and fp-31250), and of the 7 posts flagged in either run only 2 (fp-30923, fp-38748) were flagged both times.
• Flags that the self-harm violation fp-34265 was missed in both runs, and that false positives (fp-36620, fp-31299, fp-31504) mean auto-remove would delete legitimate posts.
• Quantifies how little 5 positives can show - e.g. a 95% interval on 3/5 recall spanning roughly 23% to 88% - and recommends a concrete next step such as labelling a larger, violation-enriched sample before reconsidering auto-remove.
• Invents nothing not in the input: no fabricated post IDs, counts, policy categories, thresholds or files, and the reply stays proportionate to a quick Slack-style question (a clear verdict plus the key numbers, not a long report).
• Opens with a direct answer to 'can I tell him yes': no, not as proposed (a 'do not rely on it' or 'use only with conditions' verdict), and does not endorse auto-merging every patch scored above 8 across all repos.
• Reads 'above 8' as scores 9-10 (or says which reading it used, or shows both >8 and >=8) and computes the decision metric from patches.csv: of the labelled patches scored 9-10, 20 of 24 passed tests (about 83%), so 'basically all pass' is wrong; it names or counts the 4 high-scored failures (PR-1101, PR-1085, PR-1139, PR-1004) that would have been auto-merged.
• Puts that 20/24 in context with uncertainty and a chance baseline: gives a confidence interval (e.g. Wilson about 64-93%, or Clopper-Pearson about 63-95%) and compares it with the base pass rate of 107/153 labelled patches (about 70%), making the point that the judge adds only modest lift over merging blind.
• Catches that infra-scripts has no test results at all yet supplies 15 of the 40 patches scored above 8, so the judge's precision there is unmeasured and the 20/24 figure says nothing about that repo.
• Detects size/length bias: judge scores climb with lines_changed whether or not tests pass (e.g. rank correlation around 0.56 overall, or failing patches scoring high when large); all 4 high-scored failures are 230+ lines, while all 14 labelled patches under 200 lines that scored above 8 passed.
• Uses the judge_score_rerun column for test-retest: of the 60 patches scored twice, only 17 got the same score and 9 land on opposite sides of the 'above 8' line; 14 were above 8 in at least one run but only 5 in both. It concludes the gate itself is unstable.
• Handles the 4 rows whose judge_score is ERR (PR-1222, PR-1161, PR-1127, PR-1133) by excluding them and saying so, not by treating them as 0 or dropping them silently. Any conditions it proposes follow from the findings (e.g. only small patches in repos with tests, CI must still be green, two consistent judge runs, and a labelled sample from infra-scripts before including it).
• Invents nothing: every number, patch ID, repo and test result it cites can be derived from patches.csv. It does not name or guess which LLM the bot uses, and does not claim to have run the bot or CI. The reply is sized as a sanity check (a verdict plus a handful of key numbers and conditions), not a long methodology essay.
• Answers the committee's question in the first lines with one verdict. It is not an unqualified yes: either no, or yes-with-conditions where the conditions at least exclude en-ja. The whole reply fits roughly one page, as asked.
• Takes apart the vendor's metric: it confirms LLM-linguist within-one-point agreement of about 92% (92.4% pooled) but shows the number is near-meaningless on a 4-point scale, because chance alone gives about 72-73% from these rating distributions and the linguists agree within one point about 98% of the time with each other.
• Uses chance-corrected agreement that suits an ordinal scale (weighted kappa, or Fleiss/Krippendorff with an ordinal treatment) and benchmarks the LLM against the linguists rather than an absolute cutoff. Pooled, mean quadratic-weighted kappa is about 0.85 linguist-to-linguist versus about 0.64 LLM-to-linguist (or equivalent figures from a comparable method), so the LLM is a noticeably weaker rater than any linguist.
• Breaks results out by language pair and flags en-ja. There, LLM-to-linguist quadratic-weighted kappa is about 0.47 versus about 0.88 between linguists (or three-linguist Fleiss kappa about 0.69 dropping to about 0.32-0.36 when the LLM replaces any one slot), while within-one agreement stays about 92% and hides the problem. It notes en-ja has only 40 segments (via an interval or an explicit sample-size caveat).
• Notices the LLM compresses the scale and almost never says 'meaning lost': it gave a 1 only 3 times in 150 (never in en-ja), against 25-29 times per linguist slot. On the 28 segments where at least two linguists gave a 1, the LLM gave a 1 only twice.
• Connects the agreement findings to the retranslation rule (average below 2.5): e.g. three linguists send back 44 segments, and replacing one slot with the LLM changes the send-back decision on 4-5 of the 150 segments. It also says that small number is cushioned by the two remaining linguists and does not show the LLM is as good as a human. A reply that reports agreement statistics only fails this.
• Does not invent test-retest or position-bias numbers. It notes the pilot ran the LLM once per segment, so its stability is unknown, and recommends a repeat run (and, if relevant, a check by segment type or length) before any rollout.
• Invents nothing: every figure, segment ID and language breakdown comes from adequacy_ratings.csv. It attributes no capabilities, model names or other claims to Quillmark beyond the one quoted in the request.
• Answers the actual question directly and up front: do not ship v2 on this evidence. The grader's 21-to-26 lift clears the 5-point rule on paper, but the grader can't be trusted for this comparison.
• Splits grader-vs-Priya agreement by variant instead of accepting the pooled 87% (52/60): about 93% on v1 (28/30) but 80% on v2 (24/30), and says chance-corrected agreement (e.g. Cohen's kappa) falls sharply from v1 (about 0.84) to v2 (about 0.47).
• Shows the grader's v2 errors all go one way: 6 false passes (q02, q11, q14, q19, q26, q28) and zero false fails, compared with one of each on v1 (q22 false pass, q05 false fail). So the grader errs differently on the two variants and the error inflates v2.
• Finds the length bias: all six v2 false passes are v2's longest answers (252 to 273 words; every v2 answer of 240+ words was failed by Priya but passed by the grader), and v2 answers are much longer on average than v1's (about 201 vs 118 words).
• Redoes the comparison on Priya's labels: v2 passes 20/30 vs v1's 21/30, slightly worse, not 17 points better. Or, paired by question, v2 wins on 6 questions and loses on 7. On the trusted labels the release rule is not met.
• Flags the uncertainty from only 30 questions per variant (e.g. an interval or significance test on the lift, or a note that a gap of a few points can't be resolved) and recommends a concrete next step: fix the grader's length bias (rubric or length control) and re-check it against human labels on v2-style answers, or gate on human-labelled results.
• Invents nothing: every count, question ID, word count and name it cites (Harbor, Priya, judge_eval.csv) comes from the prompt or the CSV, with no made-up extra data, grader model names or tools.
• Answers the actual question directly: no, don't turn on auto-remove with judge-b. 93% judge-to-judge agreement is not evidence that it is safe.
• Explains why 93% is misleading: the mods only labelled 12 of 200 comments (6%) as remove, so a judge that keeps everything would match the mods 94% of the time. Reports chance-corrected agreement between judge-a and judge-b (Cohen's kappa about 0.38 on run 1), and points out that two judges agreeing with each other is not the same as either agreeing with the mods.
• Scores judge-b against the mod labels on the remove class: on run 1 only 6 of its 11 removals are correct (about 55% precision) and it catches 6 of the 12 mod-labelled removes (50% recall). That is nowhere near T&S's 'no more than 1 in 20 wrong' (at least 95% precision).
• Checks judge-a against the same bar too and finds it fails as well: 10 of its 13 removals are correct (about 77%; wrong removals c007, c068, c076) and it catches 10 of 12. So keeping the expensive judge does not make auto-remove safe either.
• Reports judge-b's two runs separately rather than assuming they match: names the 5 comments that flipped between runs (c004, c029, c087, c115, c121), notes that 3 of them (c029, c087, c121) are mod-labelled removes, and says temp 0 did not make the runs deterministic, with the instability concentrated in the rare class.
• Notes that with only 12 mod-labelled removes the precision and recall estimates have very wide uncertainty (e.g. confidence intervals), so this sample can't show 95% precision even for a judge that looks good. Recommends labelling many more likely-remove comments before any auto-remove decision.
• Points out that 'catch most of the really bad stuff' has no number and asks for one or proposes one, and offers a safer interim option, such as using a judge to flag comments into a human review queue instead of removing them automatically.
• Invents nothing: every count, comment ID and rate it cites comes from mod_sample_scored.csv or the prompt, with no made-up model names, prices, comment text or extra data.

Should activate:

• Our LLM grader and a second model agree 88% on pass/fail. Is that enough to use it as a release gate?
• Here's a CSV of judge scores with three trials per item and some human labels. Can we trust the judge?
• Check our pairwise LLM judge for position bias before we pick the winning prompt
• Compute Cohen's kappa between our model judge and the annotators and tell me whether the judge is usable
• We want to swap our expensive judge model for a cheaper one. Here are both models' grades on the same 100 answers plus expert labels.

Should leave to another skill:

• We want an LLM judge to grade helpfulness on our chatbot. How should we set that up?
• Design an eval set and pass bar for our RAG assistant before we change the prompt
• Generate eval cases for this skilldrop SKILL.md so I can drop them in its evals folder
• Summarise how much each team used AI coding assistants last quarter from this telemetry export