Benford's law: when it finds invented numbers, and when it tells you nothing

Paste the column into DIGITS. Before it tests anything it decides whether the column is a candidate at all — how many values, how many orders of magnitude they span, how many are distinct, whether they look like assigned reference numbers — and where it does not qualify it says so and withholds the grade rather than handing you a result that means nothing. The tests that do not depend on Benford are run regardless. Nothing is uploaded, and it works with the network off.

What the law actually says

In many collections of naturally occurring numbers the leading digit is not evenly spread. A one turns up about thirty per cent of the time and a nine about four and a half:

The reason is scale invariance. If a quantity grows by percentages rather than by fixed amounts, it spends longer passing through the 100s on its way to 200 than it does crossing from 900 to 1000, so it is observed with a leading 1 more often. Anything that behaves that way — populations, river lengths, invoice totals, share prices, expenses — tends to follow it.

Asked to invent numbers, people do not. They spread the leading digits roughly evenly, over-use five and six, reach for round hundreds, and avoid repeating themselves in ways that real data does not avoid. That gap is what the test detects.

The part that is usually left out

Benford's law is probably the most misapplied result in auditing, and the misapplication is always the same: running it on a column it was never going to hold for, getting a failure, and treating that failure as a finding.

It does not apply to:

• Numbers with a floor or a ceiling. Prices between £9.99 and £19.99, ages, percentages, exam marks, anything capped by a rule. The range does the work, not the process.
• Assigned numbers. Invoice numbers, employee numbers, account numbers, postcodes, phone numbers. They are labels that look like quantities.
• Columns spanning less than about two orders of magnitude. If every value is between 1,000 and 6,000 there is no room for the distribution to form.
• Small samples. With fifty rows the expected count for a leading 9 is two. You cannot conclude anything from two.
• Sums of similar things. Monthly salaries, standard fees, anything clustered around a typical value.

Which is why the candidacy check comes first here, and why a column that fails it gets a red notice naming the reason and no grade. The distribution is still drawn, because looking is fine; concluding is not.

The tests that do not need Benford at all

These rest only on the assumption that the smallest digits of a real figure are not chosen deliberately, which is far weaker and far more often true:

• Repeated exact amounts. Forty claims for exactly 47.63 is a template, not a coincidence.
• An excess of round figures. Invented numbers land on hundreds and fifties far more often than real ones.
• The spread of the last two digits. In real data the final two digits are close to uniform. In invented data they are not, because people avoid patterns they think look suspicious and in doing so make a different one.

The test that actually catches fraud

Give the tool the figure above which something else has to happen — a second signature, a tender, a director's approval — and it counts the amounts sitting just below it against those just above.

A process indifferent to the limit puts roughly the same number either side. Splitting a purchase to stay under an approval threshold does not: it piles values into the few per cent below the line and leaves a gap above it. This is the single most useful thing in the tool, it needs no distributional assumption whatsoever, and it is the one that most software leaves out because it requires asking the user a question.

Reading a p-value without fooling yourself

The chi-squared statistics are turned into probabilities using the incomplete gamma function computed in the page, so they are exact for any degrees of freedom — eight for the first digit, eighty-nine for the first two, ninety-nine for the last two — and there is no table that could have been copied down wrongly.

Two things to hold on to. A very small p-value on ten thousand rows means almost nothing on its own: with a large enough sample, any real-world column departs from any ideal distribution significantly, because no real process is exactly Benford. And a comfortable p-value does not clear anybody, because a fraudster who adds a handful of large invented entries to fifty thousand real ones will not move the distribution at all.

The honest use is ranking, not proof. It tells you which of ten thousand rows to look at first. That is worth a great deal and it is not the same thing as evidence.

The short version

Why it matters that it stays on your machine

The columns people test are ledgers, expense claims, grant returns and payment runs — other people's money, usually under an engagement letter or a duty of confidence, and often at a point where somebody is already suspected of something. Pasting that into a website is the disclosure, whatever the result turns out to be.

It is arithmetic over a column of numbers. Load the page, turn the network off, and work.