Sorting files by hash, and why a broken algorithm is still the right one here
Most of the files on any machine are the operating system, the applications and the fonts, byte-for-byte identical to millions of others. Taking them off the table is the whole job. Here is why that needs three hash algorithms at once, and why an algorithm everyone calls broken is exactly the right one to use.
Give SIEVE a folder and whatever hash lists you hold. It computes MD5, SHA-1 and SHA-256 for every file in one pass, matches against your lists, and leads with the figure that matters: not how many files there are, but what share of them the lists took off the table. It also finds duplicate content under different names, and the same name holding different content, with no list at all. Nothing is uploaded.
The first hour goes on deciding what not to look at
Most of the files on any machine are not evidence of anything. They are the operating system, the applications, the fonts, the icons, the help files — byte-for-byte identical to the same files on millions of other machines. Hashing them against a list of things already known removes them in seconds, and what is left is where the time belongs.
That is the whole job: not finding the interesting files, but eliminating the uninteresting ones so the interesting ones are visible.
Why all three algorithms, every time
Published hash sets are in different algorithms depending on when they were made. Older reference sets are MD5, a great many are SHA-1, and newer work is SHA-256. A tool that computes only one cannot read half the lists that exist — and discovering that after hashing a whole machine, because the list you were handed turns out to be MD5, is an afternoon nobody has.
So all three are computed in a single pass over each file. The cost of the extra two is small next to reading the bytes, and it means any list works.
MD5 is broken. It is also still the right tool here.
This confuses people reasonably, because both halves are true and they sound contradictory.
MD5 and SHA-1 are broken for proving a file has not been tampered with. Collisions can be constructed: somebody can build two different files with the same MD5 deliberately. If your security depends on "these hashes match, so the files are the same", that assumption no longer holds.
They are not broken for looking something up in a list somebody else made earlier. To fool that, an attacker would have to make their file collide with a specific hash already published in a reference set — a preimage attack, which nobody can do against MD5. And the reference sets exist in MD5, so refusing to use it means refusing to use them.
The practical answer is to do both: use MD5 and SHA-1 for lookups, because that is what the lists are in, and record SHA-256 for every file as well, so the evidential hash exists and nobody has to re-do the work later when somebody asks a harder question.
Reading the lists you were given
Hash lists arrive in whatever shape the person who made them felt like. One hash a line. The output of sha256sum, which is hash, two spaces, filename. A CSV with the hash in the third column. An NSRL-style export with a dozen fields. Requiring a particular format means most real lists have to be reshaped by hand first.
Instead, every token on every line that is 32, 40 or 64 hexadecimal characters is taken as a hash, and whatever is left of the line becomes the label. That is the difference between "this file is known" and "this file is known to be a Windows system font", which is most of the value of a list.
The two findings that need no list
- The same content under two names. Sometimes a backup. Sometimes a file renamed to look like something else. The hash does not care which, and it is frequently the most interesting thing in the set.
- The same name holding different content. A file that should be a known system file and is not. Worth looking at immediately.
The number to lead with
The useful headline is not the file count. It is the share the lists eliminated. Ninety per cent means your lists fit this machine and you have a manageable remainder. Twelve per cent means the lists you are carrying are for the wrong operating system, the wrong locale or the wrong decade, and the next hour should go on getting better lists rather than on reading files.
That figure tells you whether the triage is working, which no file count does.
The short version
| The job | Here | Notes |
|---|---|---|
| MD5, SHA-1 and SHA-256 in one pass | Yes | So any published list works |
| Read a list in whatever shape it arrives | Yes | Any 32, 40 or 64 hex token on the line |
| Keep the label from the list | Yes | Known, versus known to be a Windows font |
| Report the share eliminated | Yes | The figure that says whether triage is working |
| Duplicate content under different names | Yes | No list needed |
| Same name, different content | Yes | Also no list needed |
| Work with no connection | Yes | The files are the evidence |
| Supply a hash set | No | You bring the lists; they are large and they are yours |
| Tell you a file is safe | No | Known means somebody listed it, not that it is harmless |
| Find something a list never had | No | Unknown is a queue, not a verdict |
Why it matters that it stays on your machine
The files being triaged are the evidence. They belong to a case, an incident or a person, and uploading them to a service to find out which ones are boring is the one thing that cannot be undone afterwards.
Hashing is arithmetic over bytes and a browser can do it at disk speed. Load the page, turn the network off, and work.
Questions people ask about sorting files by hash