Skip to content

Skills · ai-engineering

ai-usage-report

Generate a per-user or team-level report on how AI tools are being used — volume, breadth, and effectiveness signals (e.g. whether AI outputs landed in shipped artifacts vs were generated and discarded). Consumes usage-event logs exported from an MCP server or other telemetry source. Use when a manager or team lead needs to understand AI adoption beyond raw counts — including spotting "AI theater" where people invoke AI to claim usage but don't act on the outputs.

Pack ai-engineeringTier standardJob Govern AI usev0.1.2

npx skilldrop-cli install ai-usage-report
Other ways to install

With the sibling skills it hands off to:

npx skilldrop-cli install ai-usage-report --with-related

The whole pack it belongs to:

npx skilldrop-cli install --pack ai-engineering

By hand, from a clone of the repo:

cp -R packs/ai-engineering/skills/ai-usage-report ~/.claude/skills/

In Claude Code, with the ai-engineering plugin installed, invoke it as:

/ai-engineering:ai-usage-report

Try it

A realistic prompt from the skill's acceptance evals:

Here's our AI usage export for September 1-30: northwind-ai-events.csv, 4,212 rows, 14 engineers on the platform team. Columns are timestamp, user_email, tool_name, session_id, output_consumed. There's no discard field. I want the team-rollup view for our VP's monthly staff meeting — she wants to know whether people are actually using what the AI gives them or just clicking the button.

What a good result looks like

Example output

Example: Weekly engineering-team rollup · the skill's own worked example, source

Input
timestamp,user,tool,session_id,output_consumed,consumed_at,outcome
2026-05-19T09:14:23Z,alice@acme.example,claude-code,sess-001,1,2026-05-19T11:02:00Z,success
2026-05-19T09:14:55Z,alice@acme.example,claude-code,sess-001,1,2026-05-19T11:02:00Z,success
2026-05-19T09:18:02Z,alice@acme.example,slide-outliner,sess-001,1,2026-05-19T14:30:00Z,success
2026-05-19T10:01:11Z,bob@acme.example,cursor,sess-014,0,,success
2026-05-19T10:01:30Z,bob@acme.example,cursor,sess-015,0,,success
2026-05-19T10:02:11Z,bob@acme.example,cursor,sess-016,0,,success
2026-05-19T10:03:11Z,bob@acme.example,cursor,sess-017,0,,success
2026-05-19T10:04:11Z,bob@acme.example,cursor,sess-018,0,,success
2026-05-20T14:22:08Z,carol@acme.example,claude-code,sess-029,1,2026-05-22T08:11:00Z,success
2026-05-20T15:22:08Z,carol@acme.example,adr-generator,sess-029,1,2026-05-22T08:11:00Z,success
2026-05-20T16:22:08Z,carol@acme.example,design-doc,sess-030,1,2026-05-22T08:11:00Z,success
2026-05-19T11:14:23Z,dan@acme.example,claude-code,sess-101,1,2026-05-19T13:00:00Z,success
2026-05-19T11:15:23Z,dan@acme.example,claude-code,sess-101,0,,success
2026-05-20T08:14:23Z,eve@acme.example,claude-code,sess-201,1,2026-05-20T09:00:00Z,success
2026-05-20T08:15:23Z,eve@acme.example,claude-code,sess-201,1,2026-05-20T09:00:00Z,success
2026-05-20T08:16:23Z,eve@acme.example,cursor,sess-202,1,2026-05-20T10:00:00Z,success

16 events, 5 users, 5 tools, period 2026-05-19 → 2026-05-20.

Generate the three views

# Per-user (manager view, names individuals)
python3 scripts/build_report.py events.csv \
  --view per-user --from 2026-05-18 --to 2026-05-21 \
  --team-name "Acme Eng" -o out/per-user.md

# Team-rollup (aggregate, no names, safe for skip-level/exec)
python3 scripts/build_report.py events.csv \
  --view team-rollup --from 2026-05-18 --to 2026-05-21 \
  --team-name "Acme Eng" -o out/team-rollup.md

# Effectiveness (per-user template, sorted by descending single-shot rate)
python3 scripts/build_report.py events.csv \
  --view effectiveness --from 2026-05-18 --to 2026-05-21 \
  --team-name "Acme Eng" -o out/effectiveness.md
Output
# AI Usage Report — Acme Eng

**Period:** 2026-05-18 → 2026-05-21 (3 days)
**Users in scope:** 5
**Total invocations:** 16

| User | Invocations | Sessions | Avg / session | Distinct tools | Effectiveness rate | Single-shot rate |
|---|---:|---:|---:|---:|---:|---:|
| alice@acme.example | 3 | 1 | 3.0 | 2 | 100% | 0% |
| bob@acme.example | 5 | 5 | 1.0 | 1 | 0% | 100% |
| carol@acme.example | 3 | 2 | 1.5 | 3 | 100% | 50% |
| dan@acme.example | 2 | 1 | 2.0 | 1 | 50% | 0% |
| eve@acme.example | 3 | 2 | 1.5 | 2 | 100% | 50% |

Reading the data:

  • Alice — 3 invocations in 1 session, all consumed, 2 tools. Deep, engaged use.
  • Bob — 5 invocations across 5 different sessions, 0% consumed, 1 tool only. This is the AI-theater pattern: high single-shot rate + low effectiveness + low breadth. Worth a conversation — but the conversation, not the verdict.
  • Carol — varied tools, mixed effectiveness. Healthy exploratory pattern.
  • Dan — small sample. Don't over-read.
  • Eve — engaged use, good breadth.

Output — team-rollup (excerpt)

# AI Usage Rollup — Acme Eng

| Metric | Value |
|---|---:|
| Invocations / user / day (median) | 1.0 |
| Distinct tools per user (median) | 2 |
| Effectiveness rate (team avg) | 62% |
| Single-shot session rate (team avg) | 64% |
| Active users (≥1 invocation in period) | 5 of 5 |

| Engagement tier | Users | What it means |
|---|---:|---|
| **Heavy** (>20 invocations/week) | 0 | … |
| **Moderate** (5–20 / week) | 4 | … |
| **Light** (1–4 / week) | 1 | … |
| **Inactive** (0 / week) | 0 | … |

No names appear. The "Bob is doing AI theater" insight is invisible from this view by design — if a manager needs that signal, they switch to per-user and take ownership of the privacy boundary.

Demonstrating graceful degradation

If the input is missing optional fields, columns drop silently to —:

timestamp,user,tool
2026-05-19T09:14:23Z,alice@acme.example,claude-code
2026-05-19T09:14:55Z,alice@acme.example,claude-code
| User | Invocations | Sessions | Avg / session | Distinct tools | Effectiveness rate | Single-shot rate |
| alice@acme.example | 2 | — | — | 1 | — | — |

The "Where signals are missing" section then explains what couldn't be computed and why.

Demonstrating the privacy floor

With fewer than 5 users in scope, team-rollup refuses to generate:

$ python3 scripts/build_report.py small_team.csv --view team-rollup ...
error: team-rollup requires ≥5 users in scope; got 3. Aggregate views aren't
anonymous at that size. Use --view per-user and treat the output as identifying.

This is the contract — the tool won't be the one to anonymize-by-accident a team that's too small to be anonymous.

Hand-off to the user (what the agent would say)

Generated out/per-user.md — 5 users, 16 invocations, period 2026-05-19 to 2026-05-20.

Two things to know before reading:

  1. All optional signals were present, so the report includes effectiveness and single-shot columns. The numbers are real, not estimated.
  2. The most actionable pattern is in the single-shot + effectiveness columns. One user shows 100% single-shot and 0% effectiveness — a starting place to ask about engagement, not a verdict on intent. The report flags the pattern; interpretation is yours.

Want me to also produce the team-rollup view? It hides individual names and is safer for broader audiences.

How it's checked

Measured against plain Claude Code (2026-10-23): it met 57% of its checks, against 53% without it, and a blind judge preferred the agent without it. How skills are checked

21 assertions across 3 acceptance evals, and 7 trigger queries. Every skill's checks

What the output must do
  • The report header states the exact period covered (2026-09-01 to 2026-09-30)
  • The team-rollup names no individual anywhere — no emails, names, or per-person rows, even in passing
  • The handoff is three sentences above the report: volume, what the data does and doesn't show, and the most actionable pattern
  • The discard-rate signal is omitted with a note that the input has no discard field, rather than estimated or shown with '~' or 'est.' placeholders
  • The output-consumed rate and other numbers are computed from the input rows and reconcile with the 4,212-row count; no metric is invented
  • The 'AI theater' framing appears only in section headers and pattern-level observations, never as a judgment about a person
  • Recommendations target team patterns (e.g. a workshop), not specific people
  • No prompt text is embedded in the report; it works from metadata only
  • It removes the duplicated rows (the same event_id appearing twice, all Farah's events from 22-28 September) and reports Farah's figures deduplicated, for example 14 drafts generated rather than 19
  • It treats the usernames corin and c.lund as the same person, Corin, giving 17 drafts generated in total
  • It accounts for the gaps: 15-21 September is missing from the data, Delphine was on leave 1-14 September, and Ezra's tool is not in this export; it does not judge Delphine or Ezra as low adopters
  • It flags that Amara generated the most drafts (64) but only 2 ended up in published work, as a pattern worth discussing, without claiming to know her intent
  • It does not name a 'bottom two' for performance conversations from this data, and says why the data cannot support it
  • It does not invent numbers that cannot be derived from the files
  • It is a summary of about one page plus a per-person table
  • It delivers the write-up in this reply rather than asking questions first
  • It says chat counts show how much the tool is used, not whether it is used well, and does not call Priya the best or Tom the worst user on the basis of the counts
  • It does not conclude that Sam is not using AI or is doing badly; it notes that the data does not say why (for example leave or another tool)
  • It says what extra information would show whether the tool is being used well, such as whether outputs end up in client work
  • It does not invent figures beyond the five counts given (totals or shares calculated from them are fine)
  • It is a few lines suitable for the partners
When it should and shouldn't activate

Should activate:

  • Build me a monthly AI adoption report from this usage-event CSV
  • Which teams are generating AI output but never shipping it? Here's the telemetry export
  • Turn this JSONL of MCP usage events into a team rollup I can show execs
  • I suspect some people are invoking Copilot just to hit a usage target — can the logs show that?

Should leave to another skill:

  • Design a golden set and regression gate for our support-bot prompt
  • Define the product success metrics for our new AI summarization feature
  • Write the policy on which AI tools staff may use with customer data

Source

SKILL.md · the whole folder on GitHub.