Which of these files has anybody seen before SIEVE
Most files on any machine are the operating system and are identical everywhere. Take those off the table and what is left is where the hours should go.
The files
A folder, or a selection of files. They are hashed in this tab with MD5, SHA-1 and SHA-256 at once, because a list you are handed later may be in any of the three and rehashing the whole machine is an afternoon nobody has.
The lists
Any text or CSV with hashes in it: one per line, the output of sha256sum, an NSRL-style export, or a spreadsheet column. Every token on every line that looks like a hash is taken, and whatever is left of the line becomes the label.
What is left to look at
File by file
Two findings that need no list
The record
How it works
SIEVE is for known-file triage: deciding what not to look at. Most files on any computer are the operating system and are identical everywhere, so it hashes a folder and checks each file against hash lists you supply, one of files to ignore and one of files to flag. The files are hashed in your browser and never uploaded.
The guide: Hash triage, and the MD5 question
Questions people ask about SIEVE
What is known-file triage for?
Deciding what not to look at. Most files on a machine are the operating system, the installed applications, their fonts, icons and libraries, and they are byte-for-byte identical on millions of other machines. Hashing them against a list of files already known takes them out of consideration in seconds, and the remainder, the unknown, is where examination time belongs. It is the cheapest hour saved in the whole process.
What format does a hash list need to be in?
Almost anything. One hash per line, the output of md5sum or sha256sum, a CSV with the hash in some column, an export from a reference set, a spreadsheet column pasted into a text file. Every token on every line that is thirty-two, forty or sixty-four hexadecimal characters is taken as a hash, and whatever remains on the line becomes the label, which is what turns “this is known” into “this is known to be a Windows font file”.
What is the difference between a list to ignore and a list to flag?
Only how a match is treated. A list to ignore is a reference set of files that are known and uninteresting; matching one takes the file off the table. A list to flag is the opposite: hashes of things you are looking for, and matching one marks the file. You can load several of each, and a file matching both is flagged, because a flag is the finding and being on a known-good list does not cancel it.
Why does it compute MD5 at all, if MD5 is broken?
Because a great many existing hash sets are MD5 and always will be, and being unable to read them would make the tool useless for exactly the work it is for. MD5 is broken in the sense that two different files can be made to share a hash by somebody who sets out to do it. That matters when the hash is your evidence that a file is unchanged; it does not matter when the hash is a key for looking something up in a list. The page and the record both say so, and SHA-256 is recorded for every file so that the evidential hash exists too.
How large a folder can it handle?
Thousands of files comfortably. Hashing is the slow part and the page reports progress as it goes. Individual files above about 256 megabytes are listed as not hashed rather than quietly counted as unknown, because the honest answer is that they were skipped and a silent miscount in a triage tool is worse than an obvious gap.
Is any of it sent anywhere?
No. Files are read by the browser, hashed in the page, and never leave it. The lists are read the same way. Closing the tab leaves nothing, and once the page has loaded it needs no network at all, which is how a forensic workstation is usually configured.
Doing this often? OBSCURA Case groups exact and near duplicates across a whole collection, and reviews what is left. Everything is US$5 a month or US$48 a year.