When the Metric Can Only Say Yes

If the mechanism that produces a number also guarantees the number, it has stopped being a measure.

That sentence is the whole piece. What follows is how we learned it on our own system, and why it matters more in claims and legal operations than almost anywhere else.

Twenty-five overdue became zero, and nothing got done

A build report on one of our internal project tracking systems carried a clean before-and-after table. Overdue tasks: twenty-five before, none now.

The tasks were not completed. A load balancer built the same day had spread them across the following week at a workable seven a day. Every one of them was still outstanding. They had simply been given later due dates, so none of them met the definition of overdue any more.

That is genuinely useful work. Twenty-five things landing at once is a bad way to run a week, and seven a day is a better one. The report described it honestly everywhere except in the one number anyone actually reads.

Nobody did anything wrong. The load balancer did exactly what it was built to do, the report printed exactly what the database returned, and the number was correct. It was also incapable of saying anything else. Once a tool exists whose job is to move overdue work into the future, the overdue count will always trend to zero, whatever happens to the work.

Why the honest number was already in the report

This is the part worth sitting with. The useful metric had already been built. It was one section further up.

It counted how many times a task had been moved.

That number cannot be gamed by the rescheduler, because the rescheduler is the thing it counts. A task pushed four times is visible as a problem in a way that a task sitting comfortably inside its fourth revised due date is not. The honest measure existed, was being calculated, and was being ignored in favour of the one that looked better.

We think that is the common shape of this failure. It is rarely that nobody measured the right thing. It is that the flattering number and the honest number sit on the same page, and attention goes to the flattering one.

The second version: anything defined twice will disagree

The same week, a different fault, same family.

A chat assistant in one of our systems reported that nothing had gone quiet. A panel beside it, on the same screen, said fifty days. The panel was right.

"Gone quiet" had been written three separate times, in three separate places, with three different rules and two different thresholds. Nobody had done anything wrong here either. Each definition was written by somebody solving the problem in front of them, and each was correct on the day it was written.

There is one shared definition now, used by the board view, the briefing and the chat, so they cannot disagree.

The general version: a rule that exists in more than one place is not duplicated, it is forked. You will find out from a user rather than from the code, and you will find out at the moment somebody is relying on it.

Why claims operations are unusually prone to this

Because so much of what a claims desk measures is a date arithmetic problem, and date arithmetic is easy to move.

Ageing reports, SLA compliance, touch counts, days-to-settle, time-in-status. Every one of them is a calculation over dates and statuses that other parts of the same system are actively changing. A file re-opened resets a clock. A status renamed splits a cohort. A task rescheduled clears an ageing bucket. None of that is dishonest, and all of it makes the resulting number less about performance and more about mechanics.

Add automation on top and the effect gets stronger, because automation is very good at doing the mechanical thing consistently and at volume. A rule that quietly improved a number by hand twice a week will improve it four hundred times a week once it is automated.

Two suppliers circling the same problem

This is not just us.

Five Sigma made the point on 16 August that claims leaders can tell you what a large loss costs to handle, but far fewer can tell you what it costs to classify a document or chase a missing photo. That is why the return on automation stays so hard to prove. It reached us through the I Love Claims round-up of 19 August, and it is vendor commentary rather than research, so take the framing accordingly.

But the observation is the honest one, and it is the same disease at a different scale. If nobody knows what the manual version of a task costs, then any number you produce after automating it is unanchored. It is not that the measurement is wrong. It is that there was never a baseline for it to be wrong against.

NTT DATA reached us through the same round-up, arguing in a July piece that control over agentic AI has to sit at runtime, inside the workflow, rather than in a policy document. That is a perspective piece rather than a product announcement, though NTT DATA launched an agentic insurance platform of its own a few weeks later, so read it as vendor thinking rather than neutral analysis. It is the same argument about controls that we are making here about measures. A control that only exists in a document cannot stop anything. A measure that can only return one answer cannot tell you anything. Both look like governance and neither one is.

How to tell a real measure from a self-fulfilling one

Four questions. They take about ten minutes per metric.

What would have to happen for this to look bad? If you cannot describe a realistic scenario where the number goes the wrong way, it is not measuring anything. This is the fastest test and it catches most of them.

What else touches the inputs? List every process, rule or automation that can change the dates or statuses this number is calculated from. If one of them exists specifically to improve this number, you have found the problem.

Is there a harder number next to it that nobody looks at? There usually is. Ours was one section up the same report.

Who would notice first if it broke? If the answer is a customer or a client rather than your own team, the number is decorative.

What to do before you buy anything that promises to move a number

Agree three things in writing, before the build rather than after.

What a good outcome actually looks like, described in plain words with no metric attached. "Fewer files sitting untouched past thirty days" is a good outcome. "Ageing report green" is a metric that might or might not represent it.

Which number will show it, and what would make that number go the wrong way. If the supplier cannot answer the second half, that is worth knowing before you sign.

Who owns the definition, and where it lives. One place. If the same rule needs to exist in a report, a dashboard and an assistant, it should be calculated once and read three times, not written three times.

We would add a fourth, which is less comfortable. Keep the honest metric visible next to the flattering one. The temptation to quietly retire a number that makes things look harder is real, and the moment you retire it is the moment the dashboard stops being useful.

The part worth being straight about

We build claims and legal automation. We are, in other words, exactly the kind of supplier whose product can make a metric go green without making the work go away.

That is why this is a published piece rather than an internal note. The failure above happened on our own system, in our own build report, and the person who caught it caught it by reading the section nobody was supposed to need. If we are going to sell automation into claims operations, the least we can do is be specific about the way it goes wrong.

If you want to talk through what your desk is actually measuring before you automate any of it, get in touch. It is a better conversation than the one about features.