Windows 7: Choosing a Duplicate File Finder

I had long used DoubleKiller to find and delete duplicate files.  The switch to Windows 7 made this seem like a good time to review the market and see if there were better alternatives.

This question acquired urgency because I came across a need to reconcile two hard drives, and discovered that I could not.  The reason was that my customized Start Menu -- containing not only links to programs but also the entire program folders for my portable apps -- was rife with duplicate files.  These were not like duplicates among my data files.  There, I would generally want to delete duplicates.  Here, deleting duplicates would mean that programs would not run.

DoubleKiller permitted me to compare entire drives and to mark tons of duplicates for deletion, all in one move.  That sort of thing could obviously be misused.  But I had gotten reasonably good at focusing it so as to delete only what I wanted to delete.  What I wanted now -- what DoubleKiller did not offer -- was a way to exclude some subfolders within those drives and top-level folders.

This was a delicate matter.  After all, in some operations (e.g., reconciling two hard drives), I would be trusting this program to accurately identify and delete thousands of duplicate files with one click.  Judging from the number of freeware duplicate detection utilities, many of which drew approval from multiple reviewers, this was a kind of task that could be done well by a good programmer.  I just didn't want any unexpected surprises.  Fortunately, at this point I was using Beyond Compare to check my drives against backups for changes on a file-by-file basis, and I had also begun using GoodSync to analyze differences between two computers (more specifically, to stop before proceeding if more than 10% of the files differed), so I had some protections.  Nonetheless, I wanted a reliable program.

So I went looking for a replacement for DoubleKiller -- something that would render it duplicative, if you will.  Ranked by frequency of download, SnapFiles said the top five general-purpose (i.e., not image-specific) duplicate detectors were (Auslogics) Duplicate File Finder, Fast Duplicate File Finder, AllDup, Duplicate Cleaner, and LookDisk.  (DoubleKiller was sixth.)  eHow gave instructions for using Duplicate Cleaner, Auslogics Duplicate File Finder, and AntiTwin.  CNET was a muddled source for this purpose, offering general-purpose utilities (e.g., Glary) with some duplicate detection capabilities; I decided to stick with a utility dedicated to this specific task.  In two different searches of CNET, Duplicate Cleaner and Auslogics Duplicate File Finder seemed to be leaders.  A review at Gizmo named Duplicate Cleaner, Auslogics, Anti-Twin, and Fast Duplicate File Finder as the top four.  SnapFiles had not said how many downloads they had; I decided their selection was iffy.

I went to the homepages for Duplicate Cleaner, Auslogics, AntiTwin, and Fast Duplicate File Finder, looking particularly for flexibility in excluding the files within a subfolder from deletion.  From the webpage for Duplicate Cleaner, it sounded like they had this capabilty; from those for Auslogics and Fast Duplicate File Finder, I could not tell.  Anti-Twin sounded rather bare-bones -- saying, for instance, that "the software ignores file names."  For some projects, I wanted the option of comparing file names, to identify nominal duplicates that I would then manually choose among.

Conveniently, then, Duplicate Cleaner seemed to be not only the leading program named by several of the sources cited above, but also the only one among these contenders that said, right on the wrapper, that it had what I wanted.  I liked that they had a manual on their website, and that they also had the beginnings of a support forum.  I downloaded and installed it.  Its interface was more user-friendly, which meant that I immediately disliked it.  Seriously, I was turned off by the options to compare for "Same Artist," "Same Album," etc.  But I figured I would learn to ignore that sort of thing.  Its user friendliness did make it much easier to select drives or folders for comparison; and when I selected a drive and then tried to select one of that drive's folders, the program asked if I was naming the folder for exclusion.  Exactly what I wanted.  In the "More Options" area, the default comparison was MD5, but it had an option for byte-to-byte and two SHA formats.

I ran a check for duplicates, using both Duplicate Cleaner (version 2.0) and also DoubleKiller (version 1.6) on the same drives.  I had Duplicate Cleaner set to search for Same Content, any date, no file filters.  I wasn't sure how to set the file size criterion so I set it to Any Size.  It gave an option of setting a minimum file size of 1KB, but I didn't want it to skip files that were, say, 500 bytes; I had written many short text files, containing a note on this or that, that would be that size or smaller.  I went with the default MD5 content comparison type.  In DoubleKiller, I searched for files with identical sizes and CRC32 checksums, and I told it to ignore system files (because I didn't want a hundred copies of Thumbs.db or desktop.ini) as well as files whose size was equal to 0 KB.  I also told DoubleKiller to exclude .dll, .sys, vxd, and .inf files, as well as those of zero bytes and system files.  In this area, I felt that DoubleKiller's options were better, though not ideal.

I started both searches on the same machine at the same time.  During the searches, DoubleKiller remained visible, and slowly started adding visible duplicates to its list.  In a previous search, Duplicate Cleaner had seemed to stall.  Its title bar said "Not responding" and it got that sort of half-screwed-up look that graphics would get sometimes, when the computer was running out of memory or about to crash.  This time around, that didn't happen; still, it was hard to rouse it from the taskbar.  Duplicate Cleaner did not show any interesting specifics about duplicates while it was running.  Both used a screwy way of calculating the percentage of completion:  wait forever for the job to get to 1% completed, then soon it's at 5% and by the time it gets to 40% it's moving right along.  Perhaps because CRC32 checksums were easier to calculate, DoubleKiller was done first, scanning 198,645 files in 46 minutes.  Duplicate Cleaner carried on for a total of 57 minutes.  Both programs surely would have done the job much faster if they had not been competing against each other for the same computer resources.  Duplicate Cleaner found "56 Groups of duplicates" involving a total of 160 files.  DoubleKiller did not present a number, but by my count, it found 111 duplicative files.

I found the screen font displaying outputs to be much more readable in Duplicate Cleaner.  Duplicate Cleaner also allowed me to change the background colors behind alternating duplicate pairs, so as to make it easier to figure out where the alleged duplication was appearing.  Unfortunately, Duplicate Cleaner had many columns of information, but right-clicking on the column row did nothing.  In other words, it did not appear possible to suppress those columns so as to focus on the ones that mattered (e.g., MD5 hash) without scrolling right and left for every duplicative pair.

I was not satisfied with Duplicate Cleaner, and I was out of time to compare the others.  I decided, for now, to continue with DoubleKiller, and to review one of the others sometime in the future.

Long-Term File Backup and Integrity

At some point, I began saving a lot of data on my computer. These data included copies of letters, photographs, and all sorts of other things. Over time, I transferred these data from one hard drive to another, or from hard drive to CD/DVD or vice versa. On rare occasions, I would find that a file that I thought I had saved properly had somehow become corrupt. I also wondered whether I would even notice if some old files somehow disappeared from my system. These thoughts prompted me to think about adopting or developing a reliable way of making sure that my files remained in good condition over time. The Contours of the Problem It seemed to me that there were two basic concerns. One had to do with having a reliable backup system. Somehow, my files needed to move from my immediate workspace (where I had written a letter, scanned a document, imported a photograph, etc.) to longer-term storage, and I needed some reliable way of making sure that they survived each of those transitions. The other basic concern was that, in that process, the computer file that purported to be my saved document or photograph was, in fact, readable or otherwise usable. This second concern called for a way to verify the integrity of each of the saved files. (There is also an authenticity concern, not examined here, that raises the question of whether third parties have interfered with file contents in ways that a verification check might not reveal.) The backup component, in turn, seemed to have two subparts. First, I needed to have devices capable of capturing and preserving my files. Second, I needed a logging system of some sort, so that I could be confident that all of the files that had been on the original device were now present on the receiving device. Of course, a hard drive could capture my data, but would not necessarily preserve it. Hard disks were known to crash, to get stolen, and to suffer data loss when the computer system would shut down abruptly because of failure in some other component. I had long been in the habit of making backups. For some years, I had used tape for backups. More recently, I had switched to a combination of DVDs and offline hard drives. But it seemed that it could be complicated to know when a file or directory from an offline disk should be reintroduced -- to know, that is, that some file or directory on the computer had ceased to be useful and should be replaced by its backup. To avoid the need to undertake constant revalidations of large numbers of files, it made sense to work toward keeping as much data as possible offline, so that it would be unlikely to be affected by day-to-day issues in the functioning of the computer. For the files that did remain on the system, it seemed that there were two basic approaches, which one could characterize as "trust" or "verify." In the trust approach, a person would use backup software that would indicate that it had made a full, incremental, or other kind of backup; and collectively, those various backups would be assumed to provide an up-to-date counterpart of the data maintained on the system. In the verify approach, one would actually compare (by manual and/or automated means) the dates, sizes, names, and perhaps other details of the original files and the backups; and through this verification process, one could directly certify that the backup contained exactly one copy, and an accurate copy at that, of the online file. A simple example of the verify approach would be to note that, when copying one hard drive to another, there were exactly 2,324 files on both drives at the end of the copying process. Either approach would ideally produce a log of some sort, so that a person could reliably track down where the backup copy of a particular file was located. While either approach could fail, it seemed that the trust approach would be more vulnerable to, say, the unpleasant discovery that the backup did not contain what one assumed it contained. Verification of the quality of the copied files was more than a matter of noting that a certain file had been copied, or even that the copy bore the same name, date, time, and size. In some instances, Windows XP had been known to indicate that it had copied a file, but because the copying process was interrupted before completion, the copy did not actually contain the file's full data. At the start of this present inquiry, I had the general impression that there existed programs that would be capable of verifying that the copy was indeed a true and accurate one, but I had had very little actual exposure to the workings of any such programs. As a practical matter, the verification component of this inquiry also invoked the question of whether a file's format continued to be readable by currently available software and, if not, how one could become aware of that and respond to it on a timely basis. Working toward Solutions: Backup While corporate users might have more time and money for sophisticated approaches, my own needs and experiences suggested that I divide my data storage needs into three categories: online, offline, and offsite. Offsite was easiest to describe: it was, ideally, a copy of everything that I kept onsite, preserved solely for purposes of emergency backup. Of course, unless I planned to make trips to the offsite location every hour, there would be some slippage between the two. I could remedy that, in part, by using quasi-offsite storage. For example, if I kept my offsite storage at my office, I could put my latest offsite backups into my car and could them drop them at the office the next time I went there. That way, if my house burned down, my car would contain a fairly recent backup of my onsite data -- subject, of course, to the risk that the car might get too hot, too cold, or too humid, or that some thief or passenger would accidentally or intentionally damage or remove it. So offsite storage seemed derivative. It depended on what I was doing onsite. The onsite options, as I say, seemed to be divided between those things that I wanted or needed to keep online, i.e., immediately available on the computer, and those that I could keep offline, i.e., on a hard drive or DVD that I would connect to the computer as needed. For purposes of minimizing the backup and verification chore, the idea of keeping as much data as possible offline had been greatly advanced by the development of large-capacity hard drives and the proliferation of connection technologies. I could now get a large-capacity hard drive at a relatively low cost; could keep it on a shelf; and could connect it to my computer, within a minute or less, using USB, Firewire, or SATA cables. The concept, then, was that I could have my active-duty data on the computer, and my relatively inactive data on an offline (on-the-shelf) drive or DVD, and I could back them both up to another hard drive, normally kept offsite, every now and then -- weekly, or biweekly, or perhaps monthly. It could have made sense, in this scenario, to preserve historical backups in case the absence or corruption of a file was not immediately noticed (or to protect against vulnerability at those times when the offsite drive came back onsite to be updated). Protection against these risks would likely call for having more than one offsite drive. Aside from offsite backups, there would presumably also be onsite backups, made more frequently. Ideally, as I had experienced when I had used a tape drive, the backup software would kick in at scheduled times and would do its work in the background; and when it was full or finished, a reminder would pop up and I would replace the full tape with a new, empty one. It was now possible to do something similar with current backup software, using either online or mostly offline hard drives. (Quick searches at Newegg and Pricegrabber indicated that, at this writing, tape drives having 100GB or more of compressed capacity cost $400 or more, not counting media. Much smaller tape drives were available, but were still not competitive with hard drives or, increasingly, with flash drives.) Thus, a person would seemingly have three distinct sets of data at home. First, there would be online data, primarily stored in hard drives on the computer. Second, there would be offline data, stored in disks on the shelf. Third, there would be backups of the first two categories, made more frequently than was the case with the weekly, biweekly, or monthly offsite backups. And as a possible fourth category, there could be stacks of DVDs or other media containing archival copies of old backups, in case one wanted to refer back to, say, the status of a certain file as of January 1 of the previous year. Working toward Solutions: Verification For purposes of maximum protection against data loss, if it were feasible, a person would ideally undertake some kind of verification effort in each of the backup processes described above, and would also do some verification when files were being copied from one drive to another. Indeed, verification would ideally occur whenever files were duplicated, deleted, transferred, or otherwise altered, as compared to where they had been when someone last checked up on them. At the other extreme, a verification effort might catch most failures even if it was postponed until the last possible date. The last possible date would presumably be the date on which the backup source was no longer available or reliable. Thus, for example, if a person had a stack of CDs that contained a backup of his/her computer as of January 1, 1997, it might still be possible to verify the current data on one's computer against those old CDs, and that possibility might continue to exist for as long as the CDs did not get thrown out or become too old. The old CDs would not provide any insights regarding files that had been modified or created after January 1, 1997; but at least it could indicate whether the files that had remained unchanged since then were still in good condition. Since those old CDs could fade when nobody was looking, a more reasonable intermediate approach might be to go through the closet for old backups; do the comparison of then vs. now; restore old originals to replace corrupted current copies on the computer as appropriate; and then burn a new current backup of the system, to replace the old set in the closet. Before burning that new set, one would probably want to compare as many original sources as possible. That is, since the January 1, 1997 CDs would not contain more recent files, one might want to check the computer's current data, not only against that old set, but also against more recent sets. Current Technologies The foregoing discussion has already provided a partial answer to the question of what technologies, specifically, should one use for purposes of backup and verification of stored data. At present, for many, it may be fastest, most affordable, and most convenient to make backups on hard drives, supplemented for some purposes by DVDs or other increasingly affordable media (e.g., flash drives, dual-layer DVDs, DVD/RWs). Verification, for these purposes, commonly consists of a trust approach based upon the presumed functioning of the file copying setup (as when one transfers files from one computer to another using Windows Explorer) or of the backup software (which may offer the valid or invalid claim that it verifies the files it copies in terms that match the user's expectations). The foregoing discussion also leaves some things unanswered, however. On the backup side, how would one notice if some files accidentally vanished? As long as the data being backed up have not been used since the last backup, the number of files should be the same. Unfortunately, active-duty files do change in number. For instance, a document may divided into several subparts, or subparts may be combined into one file. It may not be feasible to examine files individually to insure that such changes are always authorized. Long-term archival backups on DVDs may be, ultimately, the only insurance against that sort of thing. It may be somewhat more possible, however, to proceed on a directory-by-directory basis, in an effort to provide a list of folders whose contents have changed since the previous backup. That sort of listing, with some filtering (e.g., to focus on user data files, as distinct from program files whose variations are beyond the average user's scrutiny), could provide at least an occasional heads-up as to what is changing in one's file collection. There were several different file verification technologies, at this writing, including Simple File Verification (SFV), Message-Digest algorithm 5 (md5), Secure Hash Algorithm 1 (sha1), and Checksum. A number of computer programs incorporated these technologies, in various ways. The common goal, in general terms, was to use the data within a file to calculate a number. A very simple example, simpler than the one at the foregoing Wikipedia link for Checksum, would arise where one combined the numerical value of letters together. In the string "adb," for instance, the letter "a" would have a value of 1, because it comes first in the alphabet; "d" would have a value of 4; and "b" would have a value of two. If the algorithm in question called for adding the first two letters and subtracting the third, the result would be 1 + 4 - 2 = 3. If one of the letters got changed, the result would not be 3 anymore. A person could store the value of 3, with that "adb" string, and could re-run the calculation at any time to make sure the string's contents had not changed. More complex algorithms, based upon large quantities of data within a file, significantly reduced the possibility that part of a file could change without also changing the resulting value. So, for these technologies, the basic idea was that one would calculate the sum when the file was created; would keep that sum with the file; and would refer back to it whenever one wanted to verify that the file had not changed. From the several technologies just mentioned, a number of Windows-compatible computer programs had been generated. These included FastSum (free command-line version; $14.95 graphical user interface (GUI) version); md5sum (free); sha1sum (free); FSUM (free); MD5Summer (postcardware); Advanced CheckSum Verifier ($14.95); eXpress CheckSum Verifier (free); HyberHasher ($10.00); EF CheckSum Manager (free); AccuHash ($19.95); MD5 CheckSum Verifier ($14.95); MD5 Checker (free); FileCheckMD5 (free); HashCalc (free); Turbo WinMD5 (free); File Ace ($29.95); Chaos MD5 (free); AccuHash ($19.95); GizmoHasher (free). Among those labeled as freeware, the programs bearing a rating of four or five stars from QArchive.org (which did not specify the number of votes in any case) included EF CheckSum Manager (four stars), MD5 Checker (five stars), and HashCalc (five stars). The only one bearing a rating of at least four stars from FreeDownloadManager.org was FSUM (four votes). The only one rating above four stars from CNET's Downloads.com was GizmoHasher (4.5 stars from six voters; 1,172 downloads); next closest was MD5 Checker (3.5 stars from five voters, 16,539 downloads). Note: these several ratings sites did not necessarily carry all of these programs. Finally, free programs rated Excellent at Softpedia.com included Windows Md5 Checksum (4.7, seven votes, 9,093 downloads), MD5 Calculator (5.0, six votes, 4,713 downloads), hkSFV (4.7, 17 votes, 10,283 downloads), and FileAlyzer (4.7, four votes, 2,283 downloads). [These last two paragraphs posted on Wikipedia.] Based upon this review, it appeared that the best free verification programs for Windows might include MD5 Checker, GizmoHasher, and the four from Softpedia. More detailed review of those programs' writeups suggested that GizmoHasher, Windows Md5 Checksum, and hkSFV would be especially useful for dealing with directories and with large numbers of files (as distinct from checking one file at a time). I wasn't able to find an English-language homepage for Windows Md5 Checksum, so as to learn more about its capabilities with multiple files and folders. The creator of GizmoHasher no longer appeared to be offering it, so there was no information there either. I also could not find a homepage for hkSFV. Lacking other clear guiding criteria, I started with hkSFV, the most popular by the measures cited in the previous paragraph. I ran hkSFV on a sample folder. It created a little SFV or MD5 file -- your choice -- in that same folder. My understanding was that I could then check that folder, a month or a year later, and the program would hopefully report if any file had gone bad. To test this, I changed one of the files. I inserted a plain text file in place of an Excel spreadsheet, and gave it the same name as the Excel spreadsheet. I also deleted several files. Then I ran hkSFV again, with cached mode off. Sure enough, it reported the deleted files as being not found. Oddly, unless I had made a mistake, it appeared to recognize that the text file with an .xls extension was actually a text file; I think it renamed it as filename.txt. But when I changed it back to .xls and tried again, the hkSFV program crashed, which was not entirely encouraging. I ran the same procedure again, once again opening my little MD5 file there in that folder, and yes, it definitely did rename the so-called .xls file to be, as it actually was, a .txt file; and it reported that that file was there OK. So while I was not too encouraged by the crash, I thought maybe I had caused it by fiddling with the folder while the program was open. I definitely did like its ability to supply correct file extensions. These steps were only the start of my process. Before I was going to be prepared to generate MD5 checksum for each data folder, I wanted to make sure that the folders contained actual working files, not just duds, and that they were in the proper archival format; and also that I had eliminated duplicates and had otherwise generally prepared the folder not to be changed very often. That would be a project for the future.