Skip to content

Skills · dev-team

behaviour-equivalence

Prove that a rewritten, migrated or refactored component behaves the same as the old one with a golden-master comparison — record inputs and outputs at a boundary (HTTP, function calls, batch files, DB writes, events) from production-shaped traffic and edge cases, run the old system twice to find its nondeterminism, normalise narrowly field by field, replay against the new one, triage every difference as defect, intended change (with a named approver) or missing normalisation, and give a verdict of EQUIVALENT, EQUIVALENT EXCEPT LISTED CHANGES, NOT EQUIVALENT or NOT PROVEN. Use when the user asks "does the new one behave the same as the old one", "can we sign off the rewrite", "are these diffs just noise", wants a golden-master, approval or shadow-traffic comparison, a parity check between old and new outputs, or says "prove behaviour equivalence".

Pack dev-teamTier standardJob Build and review softwarev0.1.2

npx skilldrop-cli install behaviour-equivalence
Other ways to install

With the sibling skills it hands off to:

npx skilldrop-cli install behaviour-equivalence --with-related

The whole pack it belongs to:

npx skilldrop-cli install --pack dev-team

By hand, from a clone of the repo:

cp -R packs/dev-team/skills/behaviour-equivalence ~/.claude/skills/

In Claude Code, with the dev-team plugin installed, invoke it as:

/dev-team:behaviour-equivalence

Try it

A realistic prompt from the skill's acceptance evals:

We rewrote our quote service and ran a parity check. The old service is pricing_v1.py and the rewrite is pricing_v2.py; cases.jsonl has the 12 replayed requests; old.jsonl and old_rerun.jsonl are two runs of the old service; new.jsonl is the new one; rules.json is our normalisation rules, and with them the diff is clean. Background is in notes.md. Can we sign off the cutover?

What a good result looks like

Example output

Worked example: delivery-estimate endpoint rewrite · the skill's own worked example, source

Worked example

Input given to the skill

We've rewritten GET /v1/delivery-estimate (Ruby → Kotlin). I replayed 8 requests against both and recorded the responses: old_a.jsonl and old_b.jsonl are two runs of the old service on the same requests, new.jsonl is the new one. Rule we care about: orders placed before 22:00 dispatch the same day, weekdays only. The diff is full of IDs and timestamps. Is it the same?

Each record is {"case_id", "status", "body"}; the body has estimate_id, generated_at, dispatch_date, delivery_date and a carriers list. Requests (placed_at in UTC):

Caseplaced_atPostcode
e012026-07-14 08:05SW1A 1AA
e022026-07-14 21:30SW1A 1AA
e032026-01-13 21:30SW1A 1AA
e042026-07-18 10:00 (Saturday)EH1 1AA
e052026-07-14 09:00ZZ9 9ZZ (unknown)
e062026-01-13 22:15EH1 1AA
e072026-03-29 21:20 (Sunday, BST starts)SW1A 1AA
e082026-07-15 21:59EH1 1AA

What the run looked like

Old against old, no rules: estimate_id, generated_at and the order of carriers differ in all seven 200 responses; nothing else does. Those three fields get one rule each:

{"rules": [
  {"path": "body.generated_at", "rule": "ignore", "reason": "wall-clock time of the call"},
  {"path": "body.estimate_id", "rule": "regex", "pattern": "^est_[0-9a-f]{10}$", "replace": "est_<id>", "reason": "random per call"},
  {"path": "body.carriers", "rule": "sort", "reason": "old service shuffles; the checkout page sorts carriers itself"}
]}

Old against old with the rules: EQUIVALENT, 8 of 8. Old against new with the same rules:

Verdict after rules: NOT EQUIVALENT

Cases: 8 old, 8 new, 8 compared: 6 matched, 2 mismatched, 0 accepted-only

Differences by field:
  path                                      diffs  cases  case ids
  body.dispatch_date                            2      2  e02, e08
  body.delivery_date                            2      2  e02, e08

First 4 of 4 differences:
  e02  body.dispatch_date  [changed]  old=2026-07-15  new=2026-07-14
  e02  body.delivery_date  [changed]  old=2026-07-16  new=2026-07-15
  e08  body.dispatch_date  [changed]  old=2026-07-16  new=2026-07-15
  e08  body.delivery_date  [changed]  old=2026-07-20  new=2026-07-17

Reply

NOT EQUIVALENT. The new service applies the 22:00 cut-off in UTC; the old one applies it in London time. In summer, orders placed between 21:00 and 22:00 UTC (22:00–23:00 in London) now get same-day dispatch and a delivery date one to three days earlier than the old service gave.

Differences

  • 🟥 Defect: dispatch_date and delivery_date, e02 and e08. e02 (21:30 UTC, 22:30 BST): dispatch 15 Jul → 14 Jul. e08 (21:59 UTC, Wednesday): dispatch 16 Jul → 15 Jul, delivery Mon 20 Jul → Fri 17 Jul. Both are after 22:00 London time and before 22:00 UTC; e03, the same clock time in January, matches. Check where the new code converts placed_at before comparing with the cut-off.

Noise, normalised: generated_at (clock), estimate_id (random, format still checked) and carriers order (shuffled by the old service, sorted by the page). All three varied between the two old runs; nothing else did.

Covered: morning and late-evening weekday orders in winter and summer, a Saturday, an unknown postcode (422, same body), a post-cut-off winter order. Not covered: the DST change itself, because e07 falls on a Sunday and both versions roll it to Monday; the Friday after-cut-off case; bank holidays, if the old service knows any.

To confirm

  • Add a weekday order between 21:00 and 22:00 UTC on the Monday after each clock change, and a Friday 22:30 London order, then rerun.
  • Fix the time-zone conversion and rerun all 8 before anyone treats the earlier dates as an improvement.

How it's checked

Measured against plain Claude Code (2026-10-23): it met 69% of its checks, against 51% without it, and a blind judge preferred it. How skills are checked

39 assertions across 5 acceptance evals, and 9 trigger queries. Every skill's checks

What the output must do
  • The first line gives a verdict of NOT EQUIVALENT (or plainly says the cutover should not be signed off), not 'equivalent' or 'mostly the same'
  • Identifies the real difference: VAT on reduced-rate lines is 1p lower in cases c04 and c07 (113 -> 112 and 1555 -> 1554, totals 2363 -> 2362 and 33185 -> 33184)
  • Names the cause from the code: pricing_v2 uses Python round() on a Decimal (half-to-even) where pricing_v1 quantizes with ROUND_HALF_UP
  • Says the 1p tolerance rules on vat_minor, lines[].vat_minor and total_minor are what make the diff look clean, and that money must be compared exactly rather than with a tolerance
  • Uses the two old runs to separate noise from regressions: quoted_at, quote_id and line order vary between old runs, the VAT values do not, so the VAT differences are not noise
  • Keeps the three justified rules (ignore quoted_at, regex on quote_id, sort lines by sku) and does not tell the team to ignore whole objects such as body or lines
  • States what the verdict does not cover, including promo codes, which notes.md says were not in the replay set
  • Does not invent cases, amounts, approvers, production volumes or dates that are not in the files
  • Answers in a short numbered plan, not a long document, and does not claim a verdict exists yet
  • Says to record inputs from real (scrubbed) production days, plus edge cases such as an empty day, a max-size day, rejected transactions, unusual characters, and dates around midnight, month end or DST
  • Says to run the COBOL job twice on the same input first and treat only fields that differ between those runs as noise
  • Normalises the run timestamp narrowly, in the header field only, not by ignoring the whole header or whole lines
  • Sorts records only after confirming the consumer of the settlement file does not depend on record order, and names that as a check
  • Compares the rejects file, trailer or control totals, and exit codes, not only the main settlement records
  • Says amounts are compared exactly (in minor units), with no tolerance
  • Does not invent volumes, file formats, field names or bank names not in the prompt
  • The first line says no, or gives a verdict of NOT EQUIVALENT, before any breakdown
  • Catches that the 3 status changes (404 -> 200 with empty results) are not covered by the error-copy sign-off and are a contract change clients may depend on, so they are unresolved until classified and approved
  • Accepts the 16 error.message changes as an intended change only because an approver (Mei, 3 Oct) is named, and notes clients that parse the message text should be checked
  • Says the 41 tie-order differences cannot be called noise without an old-vs-old run, because the old service may break ties deterministically and users see result order
  • Points out that the old-vs-old rerun was not done, so the normalised fields (took_ms, request_id, served_by) are unverified as noise, and asks for it
  • Does not treat 340 of 400 matching as a pass rate or verdict
  • Does not invent causes, cases, counts or names that are not in the summary
  • Answers the actual question directly near the top: no, it should not be signed off as equivalent or the old box switched off on that evidence, with an explicit verdict of NOT EQUIVALENT (or NOT PROVEN at best), not EQUIVALENT.
  • Says the three differing rows (A1001 163->162, A1009 361->360, A1011 113->112) are not float noise: each raw value is an exact half (162.5, 360.5, 112.5) and the change comes from Python 3 round() rounding half to even where the Python 2 code rounded half away from zero. A systematic rounding change, classed as a defect, or as an intended change only if finance names an approver.
  • Uses A1003 (167.5 -> 168) and/or A1007 (67.5 -> 68), which are also exact halves but match because they already round to even, as evidence the pattern is deterministic and not random noise; OR otherwise shows that every half-cent input is affected by the same rule.
  • Catches that the diff in diff.txt (cut -d, -f1,2) drops the currency and run_ts columns. That hides a real difference: new_output.csv has a blank currency for A1005, caused by the ccy = row["currency"] if fee > 0 else "" line in fee_calc_py3.py. Treats it as a defect or as an intended change that needs a named approver, not as noise.
  • Says which differences can legitimately be normalised away, and only narrowly: the run_ts value, and row order (the Py2 version writes rows in arbitrary dict order, the Py3 version in input order), with a check that nothing downstream depends on row order. It does not drop whole columns.
  • Proposes a concrete fix for the rounding, such as decimal.Decimal with ROUND_HALF_UP, or an integer half-up formula, and says to re-run the comparison afterwards.
  • Flags that the evidence is too thin to switch off the old box: one day, 12 rows, all GBP, no negative amounts/refunds or other edge cases, and the old job was run only once, so its own nondeterminism is unknown. Recommends replaying more production-shaped input first. Keeps the answer proportionate to a quick Slack-style question rather than a long formal report.
  • Invents nothing that is not in the files: no extra account IDs, fee values, rates, downstream systems, or facts about Priya beyond her being the person to tell.
  • Gives a clear verdict up front that can be pasted into the change record: NOT EQUIVALENT, not the team's 'equivalent'. It also says plainly whether the Tuesday 27 Oct cutover should go ahead, with the blocking items listed.
  • Rejects dismissing seq 4 (GET /orders/ord_bad!!, old 400 INVALID_ORDER_ID vs new 500 INTERNAL_ERROR) as noise. A malformed id that now produces a server error is a behaviour change, most likely a validation defect, whatever the team thinks about how often clients send it.
  • Finds that ignoring response.body.error wholesale hides a contract change at seq 6: error.code went from QUANTITY_INVALID to VALIDATION_ERROR. Ties this to TEAM_NOTES saying the Android app switches on error.code, so this is a user-visible break, not reworded text.
  • Finds that numeric_strings_to_numbers hides a type change: total went from a string ("19.90", "5.00", "39.80") to a number (19.9, 5.0, 39.8) in seq 1, 2, 5 and 7. Ties this to TEAM_NOTES saying the web checkout passes total to the payment widget as a string, and treats it as a defect or as an intended change that needs a named approver.
  • Uses old_run1 vs old_run2 to separate the old service's real nondeterminism (request_id, date header, the id and created_at of the order created at seq 5, and the order of tags on ord_1001) from everything else. Recommends narrowing the rules to match: tags-only sorting instead of sort_all_arrays, id/created_at ignored only on the POST-created resource instead of globally, and error.message ignored instead of the whole error object.
  • Puts each remaining difference into one category (defect, intended change, or missing/over-broad normalisation), including the reworded 404 message at seq 3 and the content-type charset change. Notes that the files name no approver for any intended change, so none of them counts as approved yet.
  • Notes that 8 hand-picked requests on a seeded staging database are not production-shaped traffic. Calls out obvious coverage gaps, such as auth failures, pagination, and concurrent or large orders, and recommends replaying a production-shaped sample, or shadow traffic, with the corrected normalisation before cutover.
  • Invents nothing that is not in the files: no extra endpoints, recorded results, error codes, consumers, approvers or dates beyond those in recordings/, normalisation.yaml and TEAM_NOTES.md (Dana Okafor is the only person named).
When it should and shouldn't activate

Should activate:

  • We ported our tax calculation from Java to Go. I have recorded outputs from both on 2,000 real invoices. How do I prove they behave the same?
  • Our shadow-traffic diff between the old and new checkout API shows a bunch of differences. Are these just noise or real regressions?
  • Set up a golden master comparison for the legacy report generator before we replace it
  • The new billing service matches the old one apart from some timestamps and a few 1p differences. Can we sign off parity?
  • Review these normalisation rules for our approval tests and tell me if they hide anything

Should leave to another skill:

  • Plan how we break up the 3,000-line OrderService into smaller pieces without stopping feature work
  • Write a phased plan to move our orders table from MySQL to Postgres with a backfill and zero downtime
  • Write a test plan for the new loyalty-points feature before release
  • I'm new to this repo, where does the legacy pricing code live and how do I run its tests?

Hands off to

Related skills

codebase-tour, refactor-plan, test-plan-generator

Source

SKILL.md · the whole folder on GitHub.