
Measurement
Part of Measure productivity software with a metric contract and a baseline you can revisit
Reading a productivity software report: six measurement mistakes to catch
Six measurement mistakes to catch when you read a business productivity software report, with the question to put to the analyst about each.
Six measurement mistakes can make a neat productivity software report support the wrong decision. A reader who knows what to ask can catch each one before the report drives a licence, staffing or supplier decision, without rebuilding the analysis.
Each mistake below is one that current UK public or regulator sources address. No organisation, supplier or employee was studied, and no prevalence figure is claimed.
What to take away
- A headline figure needs a defined event, population and period before it supports a decision.
- Activity counts show that events occurred, not that work improved or cost fell.
- A trend that crosses a changed definition is not comparable with itself.
- A before-and-after change is an association unless the design includes a counterfactual.
- Person-level league tables and imported benchmarks each need a check before anyone relies on them.
The one-page review sheet
Put six questions to whoever produced the report and expect a written answer to each. The sheet pairs every mistake with the question that exposes it and the answer that should satisfy you. A vague answer is a finding in itself.
| Mistake in the report | Question to ask | Sound answer |
|---|---|---|
| Activity shown as an outcome | Which outcome did this event lead to? | A defined mechanism linking usage to a result |
| Undefined denominator | A percentage of whom, doing what, over which period? | An exact fraction with population, event and period |
| Definition change inside a trend | Did any definition change during the period? | A versioned metric and an annotated break |
| Before-and-after called causal | What would have happened without the software? | A comparison group, or a stated association only |
| Staff ranking from workflow data | Who approved person-level reporting, and why? | An assessment by privacy and employment specialists |
| Imported benchmark | Does it use our definition and population? | A matched source, sample and period, or context only |
Start with the headline figure
A headline figure means something only when its event, population and period are named. Logins, comments and tasks created show that events occurred. They do not show that users completed work more accurately, that customers benefited or that cost fell.
GDS guidance on measuring service success combines operational metrics with user research and other data sources. Its requirements are for government services, but the distinction is portable. Ask which outcome sits behind each usage event. A short set of measures runs from qualifying demand to an outcome indicator.
"Eighty per cent adopted" is uninterpretable without the eligible population, activity event and period. Invited users, licensed users and people expected to perform a task are different denominators, so ask for the exact fraction before accepting the percentage.
The Government Data Quality Framework describes six data quality dimensions, including completeness and timeliness. Use them to test the population list and event data. Illustrative case: population 240 licensed users; event submitting a month-end timesheet; period four weeks to 30 June 2026; 180 completed, so report 180/240 or 75 per cent.
Look for a break in the trend
A trend line is comparable only while its definition stays fixed. Renaming a workflow state, altering a completion rule or switching from calendar to staffed hours can create an apparent improvement. The chart may be arithmetically correct but historically incomparable. Ask whether any definition changed inside the period.
ONS defines coherence and comparability as part of statistical fitness for purpose, including comparison over time and geography. The sound answer is a versioned metric with the break annotated, not a spliced series. The metric contract and baseline show how a definition is written down so it can be revisited.
Test any claim of cause
Two observations cannot isolate a cause. Demand, staffing, work mix and season can move at the same time as a software launch, so a before-and-after change is an association, not an effect. Ask what would have happened without the software.
The Magenta Book explains that impact evaluation relies on an appropriate counterfactual and sets out theory-based, experimental and quasi-experimental approaches. Unless a qualified analyst approves a causal design, the report should say observed association. Four attribution methods show how change can be tested.
Illustrative causal test: before launch team A closed 20 tickets a week; after launch 25. A comparable team B without the tool went from 20 to 24. The adjusted change is 1 ticket, not 5, because team B improved anyway.
Check what the data says about people
Workflow data is a poor basis for ranking staff. Assignee counts and completion times omit difficulty, shared work, part-time patterns, support and activity outside the tool, and publishing a league table can change the purpose and risk of the data use.
ICO guidance on monitoring workers covers productivity tools that log work and requires lawful, fair processing. The ICO marks the page as under review following legislative change. Pause person-level reporting until privacy and employment specialists assess necessity, fairness, transparency and safeguards. Data protection by purpose is the place to start.
Check any benchmark quoted
An external benchmark is context, not a target. A vendor survey median may use another geography, product tier, company size or definition, and national economic productivity is not a substitute for workflow performance. Ask for its source, sample, period, exclusions and sponsor.
The ONS productivity collection uses measures such as output per hour, output per job and output per worker. Those units do not become ticket-cycle or adoption standards. If the benchmark does not match the internal metric, keep it as context or omit it. Benchmark research treats a benchmark as a reference class.
What to do with the answers
Select the highest-consequence claim and trace it backwards: decision, wording, method, definition, calculation and source event. Record what the evidence can support and what remains unknown. One honest limitation is more useful than six precision-looking figures built on mismatched rules.
Write the repaired definition into a metric dictionary so the next report starts from it.
- Ask for the exact fraction, population, event and period behind every percentage.
- Ask whether any definition changed during the period.
- Treat before-and-after gains as association unless a causal design is approved.
- Hold person-level reporting until specialists have assessed it.
- Keep imported benchmarks as context unless source, sample and period match.
Common questions
Who should answer the review questions?
The accountable owner for the process should answer, with the analyst who built the report. If the metric covers people, seek privacy and employment advice before publication. Record the answers and the date.
How long should I keep an annotated break in a trend?
Keep the note for as long as the old definition affects a comparison. When the series no longer includes the pre-change period, archive the note with the metric version history.



