A Backup Arrangement with Beyond Compare

I had been using Beyond Compare (BC) for a year or two.  Over that period, I had settled into what seemed like a decent backup arrangement.  This post describes that arrangement.

For a while, I had a spare internal partition to which I would make backups.  The original concept there was that I would use rsync or some other program to make backups on an hourly basis.  That setup had fallen into disrepair, mostly because I didn't quite like how it was working.  So I didn't have an hourly backup at this point.  The arrangement described here is more on the longer-term (e.g., daily, weekly, monthly) level.

My backup took place on external drives.  I had an external enclosure that I would have to open up (removing several screws) to swap drives, and I also had an inexpensive dock that I could just plug an internal SATA drive into.  Both seemed to work equally well.  The enclosure was handy for unplugging the drive and taking it with me.  Now and then -- especially when the tornado alarms sounded -- I visualized myself grabbing it and running for the basement.  I wondered if that would be one of those fateful delays that would cost me my life.

The external enclosure had an eSATA connector, but my previous motherboard had not been able to accommodate eSATA on a hot-swappable basis.  In other words, I had to reboot in order to get the system to recognize it.  It also had a USB connector.  The external dock (i.e., not the enclosure) was also a USB device.  USB was slower but very adaptable.  That was almost always what I used.  Some partitions on the external drive were compressed, to save space.  I had the impression that this did not help with the USB connection -- that the CPU would unpack the file before shipping it across the slow USB cable to the computer, resulting in at least as much data moving along the wire -- but I hadn't verified that.

For my purposes, Beyond Compare offered two key concepts.  First was the workspace.  If I plugged in the external drive that I used for daily backups, then I would open up the DAILY workspace in BC.  If I plugged in a drive that I used for weekly backups, then I would choose the WEEKLY workspace.  I also had a SIMPLE COMPARE workspace that I would use for random tasks -- say, comparing two folders on a one-shot basis.  And I had a NETBOOK workspace that I would use to synchronize my laptop.  GoodSync might have been better for that if I had been using the laptop frequently, but at this point it was mostly a case of keeping the data on the laptop current with the desktop.  That is, I was mostly doing one-way updates, from desktop to laptop.

My workspaces differed in the tabs they made available.  In the DAILY workspace, I had a tab for each day of the week, plus whatever other comparisons I would want BC to make on a daily basis.  Likewise for the WEEKLY and the other workspaces.  In other words, I used a workspace as a place where I would be able to see tabs for each comparison that I wanted BC to make, whenever I plugged in the weekly drive or the laptop or whatever.

I found that the best approach was to start BC first, let the workspace load, and only then turn on or connect the external USB drive.  That way, BC would not try to do complete comparisons for all of the open tabs.  It would do its calculations for the relevant folders on the drives inside the computer, which were already available to it, but on the external drive it would have to wait until I gave it the go-ahead within a particular tab.

Focusing on the DAILY workspace, I was writing these notes on a Friday.  So to guide my remarks, I opened BC at this point.  Somehow, I had arranged for the DAILY workspace to come up by default; or maybe BC just defaulted to the last open workspace.  I wasn't sure how I had arranged that.  When BC was up and running, I turned on the USB drive.  It took that drive a moment to become available.  (I found that AntiRun was useful, not only for protecting my system from autorun malware and such, but also for telling me when a drive really was online or offline, and for giving me a functional way of taking external drives offline.)

I went to the Friday tab.  BC had stalled because the Friday folder on the external drive had been unavailable.  I told it to retry; and now that the USB drive was connected, BC ran its comparison.  (Details on the kinds of comparisons available, and other program capabilities, are available at Beyond Compare's website.  Their forums and other tech support had been very responsive, the few times I had contacted them.)

I had modified my BC toolbar to present the red Mirror button.  This said, basically, just overwrite whatever is in the backup space (in this case, the Friday folder on the USB drive) with whatever is on drive D in the computer.  Drive D was the one that I backed up daily.  So in this case, a number of files had changed since the previous Friday.  Sometimes I would take a look at them; sometimes not.  Usually not.  It seemed pretty rare that a file would be accidentally deleted.  Daily examination of all changing files had seemed to be overkill.

When I say that I would take a look, I mean that BC showed me two panes, one for each of the folders being compared.  To keep things organized, the left-hand pane was almost always the authoritative one.  The left-hand pane would correspond, that is, to a partition inside the computer.  So I was looking at the right-hand pane, corresponding to the backup device.  If I saw a file listed in the right-hand pane, but not in the left-hand pane, that would mean that it was on the system when I made my backup a week ago, but now it was no longer on the system.  BC would also alert me, with a red font, if the file in the right-hand pane was newer.  Generally speaking, that wasn't supposed to happen.

I had an alternating weekly folder on this backup drive.  I used that one on Saturdays.  That's the one I examined more closely.  If I found that something was missing on Saturday, and I didn't think it should be missing, I could then click on the tabs for the other days of the week until I found the last backed-up version, and I could restore it from there.

Drive D contained the things that were in more active use.  I also had a separate partition, drive E, for things that took up a lot of space and didn't change very often.  Videos were the main example.  Because there were so few changes there, it was easier to look at the differences identified in BC, and verify that additions and deletions were desired.

In net terms, I liked this arrangement because it gave me some flexibility to combine automated and manual processes.  I wasn't vulnerable to one of those black-box backup solutions that would seem like they were working just fine until the moment of crisis, when I would painfully discover that I had failed to adjust some essential setting, or that the drive was malfunctioning, or whatever.

In this arrangement, if I was worried that files were missing, I could look down through lists of what was being added and deleted.  If I was confident that everything was fine, I could just click the Mirror button and the backup would happen.  I could also combine both approaches within a single tab, by telling BC to mirror only the selected folder(s).  This would gradually reduce the number of things remaining on the screen (assuming I had BC set, as usual, to Show Differences rather than Show All).  When confronted with what looked like a mess, I could thus eliminate the parts that seemed OK, and focus on the files and folders that didn't seem like they should have been getting added or deleted.

Like most other computer-related matters, my backup approach continued to evolve.  But as I say, I had been using BC for a while, at this point, and I was pretty much satisfied with the combination outlined here.

Robocopy Commands for File and Folder Synchronization

I wanted to know if Robocopy could replace synchronization programs like GoodSync and Allway Sync.  I decided to take a look at its options.  This post describes what I found.

The manual for Robocopy itself seemed to confirm my prior sense that Robocopy was a unidirectional copying utility. That is, you could command it to copy from A to B, or from B to A, but not bidirectionally between A and B, never mind multidirectionally. In particular, manual page 16, discussing Robocopy's /XO option, said this:

The most appropriate use for /XO is to synchronize two directory trees so that they can be updated simultaneously in separate areas. To ensure that the latest files are present in both directory trees, copy with /XO first in one direction and then in the other.

So apparently synchronization with Robocopy would require reciprocal commands: one to copy newer files from A to B, and another, otherwise identical, command to copy from B to A.  Wikipedia noted that Robocopy would not copy open files, but that seemed to be true of GoodSync as well.  I didn't expect it to be a problem for my purposes.

(The information from the manual seemed pretty similar to what I got from a command-line "robocopy /?" command.  A Windows Server webpage provided yet another list of Robocopy commands.  I had noticed, though, that Wikipedia said Robocopy was not entirely consistent between platforms.  It seemed I had better verify that my solutions would work on Windows 7 specifically.)

I wasn't yet sure what kind of analysis /XO or other Robocopy commands would perform.  I assumed, but was not yet positive, that they would take account of time as well as date, down to the minute if not the second.  This thought prompted me to look into something that would improve the present gap of about 20 seconds between the clocks on the two computers -- something more fun than just resetting the clock manually, that is. It seemed I would want one or two minutes' tolerance in timestamp comparisons in any case; but in a separate post, I took a closer look at possible solutions to get the two machines closer to chronological synchronization.  (Later I saw that the manual (p. 17) said, "File-time granularity is 100 nanoseconds on NTFS, and two seconds on FAT.")

The next step seemed to be to peruse the Robocopy command-line options, to see which ones might work for my synchronization project.  The manual (pp. 7-11) provided a concise command-line reference.   (Apparently lowercase versions of the commands worked; I use uppercase here for visibility.)  The options that seemed most relevant were as follows, leaving out those that would be included by default:

/S copy subdirectories, excluding empty ones

/ZB resume a copying task from a point of previous failure (presumably instead of starting over), but switch to backup mode if restartable mode fails (as when using an unreliable network); but Wikipedia said that /Z would slow copying significantly

/MOT:X and /MON:Y   monitor the source directory and run Robocopy again when X number of files have changed and Y minutes have elapsed

/XD   exclude specified directories

/XO   don't overwrite destination files with source files having the same name and an older timestamp

/R   specify the number of retries

/W   specify wait time between retries

/REG   save /R and /W in registry as default settings

/L   list files

/V   produce verbose output

/TS   display source file timestamps in output log

/FP   display full pathnames in output log

/NC   suppress output of Robocopy file class information


/NS   suppress output of file and directory sizes

/NJH and /NJS   turn off logging of job header and summary

/ETA   show estimated time of completion

/LOG:filename or /LOG+:filename   redirect output to named file, overwriting or (+) appending if it already exists

/TEE   display output as well as writing LOG

/JOB:jobfile   read parameters from jobfile; optionally /IF to include files with specified names or paths; optionally /SD:path and /DD:path to specify source and destination directories; optionally /QUIT to view job file contents without any actual copying

/SAVE:jobfile   write current parameter settings to jobfile

This was an intimidating set of potentially relevant options.  There were many things that could go wrong with so many choices.  I decided to look for examples and more information.  As the manual said (p. 22), these options could result in long, unwieldy commands, and it did appear that the JOB options would help with that.  I ran a search with that in mind.  It led most immediately to some general-reference sites (e.g., SS64, PowerCram, FixMyITSystem).  I tried again.  That search led almost nowhere.  Likewise another, seemingly broader search.

Eventually, I found an eHow website that said I could use XCopy to synchronize computers.  Was I trying to use the wrong tool?  It didn't seem so.  I saw indications that Robocopy had more options and better performance, and that use of XCopy was deprecated.  But why was I not seeing tons of obvious references to the use of Robocopy JOB options for synchronization?  (Another possibility with lots of options:  XXCOPY.  But the least expensive version for networked computers cost $100, and its reception at CNET was underwhelming.)

I tried another search, focusing only on Robocopy job options.  Posts at Serverfault and Skonet seemed to suggest that I could work up my desired Robocopy command on the command line itself, and then add the /SAVE option to save it to a file, and that my job options would go into a separate file with an .RCJ extension.  I wasn't sure exactly how all that would work in practice, but at least there seemed to be a starting point.

It appeared that the slash (/) character was the signal, in the job options file and probably also for Robocopy generally, indicating that one option had ended and a new one was starting.  So, for instance, if I gave the /XD command, I could then proceed to list a boatload of directories to exclude, without worrying too much about format -- putting them on the same line, putting each on a different line, or whatever.  Everything I wrote after /XD would be treated as another directory to exclude, until I came to another / command (e.g., /S) or else reached the end of the job options file.

One thing that was not clear to me, from the foregoing options:  what if I had deleted a file on computer B, intending that synchronization should delete it on computer A as well?  GoodSync knew how to interpret such situations.  I suspected that Robocopy, run in a reciprocal setup starting with robocopy A > B, would just restore the file from computer A to computer B.  That, as I found in tinkering, was what Beyond Compare might do.  In other words, there was a real difference between a backup program and a synchronization program.  The former would just make sure that the target contained what was on the source; the latter would keep track of what had been removed from each side, and would decide whether the removal occurred later than the most recent update of those files on the other side.  I could control such things manually, more easily in Beyond Compare than in robocopy, but for an automated solution I would need a program that would keep track of dates and times of deletions as well as of file changes.  I posted a question on this, just to make sure I was understanding Robocopy correctly.  The response confirmed it, and suggested using regular synchronization programs.

Windows 7: Setting and Maintaining Accurate System Time

I wanted to keep two computers' clocks set the same, for purposes of synchronization, so that they would have an accurate sense of whether the version of File X on computer A was newer than the version of File X on computer B.  I had previously installed (or, more accurately, just added a copy of) Judah Levine's portable NISTIME 32 in something of a rush, when installing Windows 7, and, later, had vaguely recognized that it was not working right and/or I had not set it right.  Now I decided to work out the kinks in this function.

NISTIME-32BIT.EXE

I started with the National Institute of Standards and Technology (NIST), from which programs like NISTIME 32 would draw the current time.  It developed that NIST had a program called nistime-32bit.exe.  It turned out to be the same as NISTIME 32, just slightly updated.  The webpage's instructions were to start by going into File > Select Server and then Query Server > Now.  Somewhere I saw advice to choose a server near me.  I was tempted to choose two different ones, one for each computer, so as to have accurate time in case there was some terrible disruption of the national timekeeping system.  Then I realized that this could have the effect of making rivers run upstream, where my files were concerned, to wit:  new could be replaced by old.  Being up-to-date on the latest developments in American chronology suddenly seemed less important than making sure I didn't accidentally overwrite today's crossword puzzle.

When I went to the Query Server > Now menu pick, I got a dialog indicating that NISTIME 32 was prepared to adjust my computer by 0.953 seconds.  I told it to go ahead.  I also went into Query Server > Periodically and told it to update the computer every 12 hours.  Query Server > Server Status confirmed these settings.  File > Help in Choosing Dirs told me to hit File > Save Config to save my settings.  This gave me "File Error:  Cannot open file to save configuration."  That problem may have been caused by nesting the program too deeply in a subfolder.  I moved it elsewhere and tried again. Now it seemed to confirm that it had saved my settings, and it created NISTIMEW.CFG in the same folder as the program's portable executable (nistime-32bit.exe).  I exited and restarted, and it remembered what I had told it.  But I had to remember to hit File > Save Config; it would not remember anything.

But then, when I did go into Query Server > Periodically, specified 12 hours, and hit File > Save Config and then File > Exit, I could not get it back.  The program refused to become visible.  I tried a couple of times, and then looked at Windows Task Manager (Start > Run > taskmgr.exe) > Processes tab.  Taskmgr showed four separate instances of "nistime-32bit.exe *32."  I selected them and clicked End Process, one by one, and then ran nistime-32bit.exe again.  It returned to taskmgr.exe, but not to the screen.  I minimized all windows, one by one, but, no, it was not lurking anywhere.  There didn't seem to be a taskbar or system tray icon for it.  It was here, and yet not here.  I killed the processes again, now that I had started one or two new ones.  I renamed NISTIMEW.CFG to be something else, and now it would start, and it saved new settings in a new NISTIMEW.CFG.  Apparently the config file had gotten corrupted.  I had originally created that file manually in lowercase (nistimew.cfg); possibly something about the program needed the uppercase filename.

But now, same thing again.  Exiting and restarting gave me a hidden program:  visible in Task Manager's Processes tab, but not visible onscreen.  When I right-clicked on nistime-32bit.exe *32 in Task Manager and selected Properties, I got an error:  "Windows cannot find [pathname] nistime-32bit.exe."  I ended the process again.  I created a shortcut to the .exe and tried starting it that way.  I had no reason to think that would make any difference, and in fact it didn't.  I tried moving all of the files from the folder where I had put nistime-32bit.exe, and placed them all instead in C:\Windows, with a shortcut to the executable in my Start Menu.  That wasn't the answer; I still got lurking program sessions that appeared in Task Manager but nowhere else.  I deleted the CFG again and tried again.  Now it ran.  I went directly to File > Save Config without making any changes.  It indicated that it had saved the config file.  I exited and restarted the program.  It ran.

Now I saw something that may have explained the config file problem.  The server list had changed.  The Colorado server that I had selected previously was no longer listed in File > Select Server.  I had previously gone into File > Update Server List, and that had generated a message:  "New server file is C:\Windows\NIST-SRV.LST."  It did that again now, when I designated a new server.  I hit File > Save Config and then File > Exit, and then restarted the program.  Now it was running normally.  I moved the three files (the exe, cfg, and nist-srv.lst files) from C:\Windows back to the folder where I really preferred to have them.  It seemed that the server list had not properly updated when the files were in that folder originally.  I restarted and went through the same steps -- update server list, choose a new server, save config -- and now I was exiting and restarting without a problem.

But no, I spoke too soon.  When I restarted, saved a 12-hour periodic refresh, and exited, it would not restart.  Deleting the config and moving the other files back to C:\Windows did not fix it.  The problem seemed to relate specifically to the attempt to set up recurrent time checks.  I was doing something wrong, or perhaps the program had a bug, or maybe it was not suited for 64-bit Win7.  I went to the NIST webpage cited in the program's Help > More Help and sent an email to the Webmaster link at the bottom of that page, pointing them here.

The Built-In Windows Time Sync Option

I decided to look for an alternative time-updating program.  I ran a search and discovered that there was apparently some kind of automatic time-updating arrangement built into Windows.  The advice there was, however, that "The W32Time service is not a full-featured NTP solution that meets time-sensitive application needs."  That was consistent with the fact that my two computers' timeclocks tended to be somewhat inconsistent with one another.  I had not tried to see how inconsistent they could be, or how long they could remain that way.  I did see an indication somewhere that Windows defaulted to a weekly time update, so maybe it would verify that it was accurate to within a minute, or something, every week or so.

That appeared to be steered by Control Panel > Date and Time.  That dialog could also be opened by right-clicking the clock in the system tray and choosing Adjust Date/Time.  Or, as I now learned from Eric Phelps, it could also run from the command line via "rundll32.exe shell32.dll,Control_RunDLL timedate.cpl."  The latter option would facilitate the option of opening the Date and Time dialog for manual adjustment via, say, a batch file that would open it automatically (to the correct tab) every day, week, or whatever.  (Later, I found a How-To Geek webpage that said I could just run "w32tm /resync" as administrator to resynchronize the clock without even going into the Date and Time dialog.  That, too, could be incorporated into a scheduled batch file.)

The Date and Time dialog > Internet Time tab > Change Settings option gave me a choice of synchronizing with time.nist.gov, which I understood to be the most accurate (though others in that list, not counting time.windows.com, appeared to be cousins of NIST).  I noticed that the dialog told me, here, that "This computer is set to automatically synchronize on a scheduled basis."  The previous sync site, as I saw on the other computer, was time.windows.com."  I wasn't sure how synchronizing with that site could have left my two computers with different times -- differing by seconds, that is, not by minutes -- unless maybe time.windows.com was just not that worried about the seconds.  Or maybe it was trying to synchronize when my router was doing its daily self-restart, and was therefore not getting access to the online clock?  I wasn't sure.  (Note:  Fouzan said that this whole process wouldn't work if the computer was on a domain.)

Curious about the timing, I went into Start > Run > taskschd.msc > Task Scheduler Library.  There were maybe 15 items in the list, and none of them were obvious time sync tasks.  So another possibility was that some bug or tweak, brought into my system somewhere along the line, was preventing the creation or execution of the scheduling function.  Another emerging possibility was that, as stated in a How-To Geek webpage, time.windows.com (which my systems had been using by default) had "a ton of problems with uptime."  So possibly I had already fixed my problem, just by switching the machines to use time.nist.gov in the Date and Time dialog.  (I did notice, as soon as I made that switch and clicked the update button, that both computers' clocks showed exactly the same time.)

Other Possibilities

I ran another search and found a Gizmo recommendation for Dimension 4 as a time correction utility.  It occurred to me, at this point, that possibly I had fixed my problem, just by switching away from time.windows.com (above), and that maybe I should just let things slide for a week or two.  I decided mostly just to record some notes, here, for possible future reference.  So instead of installing Dimension 4, I just dragged the icon for its webpage from my browser's Address bar over to the Time subfolder in my customized Start Menu.  If I ever needed it, I could follow the link at that time.

There also appeared to be more to know than I had realized, regarding Task Scheduler (taskschd.msc).  In Task Scheduler's left-hand pane, I went down the tree into Task Scheduler Library > Microsoft > Windows > Time Synchronization.  Now I saw that my machine was indeed set to synchronize time at 1 AM every Sunday.  I saw advice from Tina Sieber on a way to adjust and improve the scheduling via Task Scheduler.  Tina seemed to believe, however, that using a separate program might be the simpler and more accurate approach.  Tina pointed toward two other programs, Atomic Clock Sync and AtomTime.  The webpage for the latter seemed very old.  I was not sure how it would fare in a 64-bit Windows 7 world.

For now, the solution seemed to be simply to go into the system's clock and change its time source to NIST.  My monthly batch file brought up the NIST/USNO timepage on the first of every month, so I could observe, later, whether my two computers were again diverging from one another and/or from the time on that webpage.  If they didn't stay in line, I would have two options.  One would be to add one of the foregoing command lines to my daily or weekly batch files, to permit manual and/or automatic checking and/or resynchronization.  Another would be to try one of the several freeware utilities just mentioned, particularly Dimension 4 or Atomic Clock Sync.

Windows 7: RAID or Mirror Across Computers?

Where to put the data ... hmm.  I had a home network with two computers running Windows 7.  If the data I needed to work with was on one computer and it went down or had one of those frequent Microsoft maintenance or service interruption needs, I couldn't get to it from the other computer.  But if I put the data on a server, then (a) I had to buy and maintain the server, cables, routers, etc., (b) I had slower access times, (c) the data would then be unavailable to *both* machines (unless I wanted to swap out one or more hard drives) if the server went down, and (d) I had found that, if I accidentally corrupted or deleted the wrong file, a server might not be willing to undelete it.  Not a big deal, assuming you had good backup, but there were painful exceptions.

So it occurred to me:  can you put the data on one computer, so that it can function as a standalone, and also put the data on the other computer, so that it is a standalone too, but then have a constant RAID or mirror arrangement between the computers, so that whatever you do with the data on one computer is immediately duplicated on the other computer?  That way, you've got local speed, no server, and redundancy during downtime on either machine.  Basically, two-way mirroring:  when a file is modified, it checks the other computer, and the two of them figure out which version is newer, and it overwrites the older version on the other machine as well.  All you need is a router, if that.

I figured possibly everybody else already knew the answer to this.  But since I didn't, I started with a search.  Only six hits.  It looked like the concept of "RAID between computers" was a nonstarter.  Alright, a different search.  Wow, "mirror between computers" produced 13 hits.  But, OK, not to complain, it seemed most of those hits were for TreeNetCopy.  Take it out of the equation, and the search produced only five hits.  So TreeNetCopy seemed to provide the path forward.  But it didn't look like CNET, PCMag, or other big-name sites had reviewed it.  I went to the product's home website and found out why:  it was for systems using Windows NT or Windows 2000.

Apparently mirror and RAID were not the concepts I wanted.  How about incremental backup?  You couldn't have it running constantly; it would have to finish one scan of the system's files before it could start on the next one.  So maybe you'd set it to run every 15 or 30 minutes, or however a scan would take to finish, across the network connection.  This wouldn't be nearly as good as software that would detect and propagate changes as soon as they were made, but I wasn't seeing how to find anything like that.  With a 15- or 30-minute delay, you couldn't have someone being able to open the file on computer B as soon as someone else updated it on computer A, unless possibly if you had a script that would somehow be able to run the incremental backup manually for a given folder by just maybe choosing a right-click context menu option.

Alright, a different approach.  I had been using Beyond Compare, a file comparison tool.  It still looked like one of the more capable file comparison tools, so how about using it?  As I was thinking about that possibility, I realized that I didn't like the idea of having to do a right-click or other manual update.  The computer could crash before I got around to that, and then I wouldn't have the current data in the parallel folder on the other computer, and therefore really couldn't just keep right on working where I left off.  I had only used Beyond Compare as a manual comparison tool, where I would start it up and it would run for a while and compare directories and then show me what needed to be mirrored to my backup drive, and then I would click the buttons necessary to do that.  I knew it was possible to write scripts to automate some of this, but it seemed unlikely that scripts would help Beyond Compare remain up-to-the-minute on all of the file changes made on the system.  Most likely, I could set up scripts to run in some frequently used folders, and maybe even to automate the mirroring of those folders, but other folders would be left out in the cold.  Possibly I could have multiple scripts doing comparisons of more- and less-frequently used folders on different schedules, so as to increase the likelihood that most folders would be mirrored relatively often.  But with enough scripts running simultaneous file comparisons, I'd start to take a performance hit.

Lacking a better option, I did a search to learn more about Beyond Compare scripts.  The search came up with a number of interesting concepts, right there among the top ten hits.  One was the concept of automated synchronization.  Duh!  Of course.  Synchronization was the Windows term for what I wanted.  So I did a search for that, dropping Beyond Compare for the moment.  But the only thing that came of it was the discovery of Super Flexible File Synchronizer, which cost $60 for a two-year license (unless, for some bizarre reason, I would think that I could do without the pro version's ability to copy ZIP files!).  It did look like it might have some advantages over Beyond Compare, such as the ability to detect that I had moved a folder, so that it could just repeat the move rather than delete the folder from one location and create it in another (which might involve a lot of copying, if it was a large folder).  It had very good ratings on CNET.  I could download and try it out free for 30 days.  But it was ultimately still a backup program, running on a schedule, not a mirroring program, so I was still basically working with the same scenario:  design a set of backup scripts, profiles, or whatever, and set them to run at different frequencies, backing up what I would consider the most heavily used folders most frequently.

TopTenReviews ranked Super Flexible File Synchronizer eighth in its list of sync programs.  Their comparison page had a number of relevant criteria, including the ability to do bidirectional sync, to mirror files, and to operate across a network.  It actually looked like their number two program, GoodSync ($19.95), had better features for my purposes than their number one choice, Syncables 360 ($39.95).  Their review of GoodSync made it sound good indeed.  They said it couldn't sync or merge Microsoft Outlook files, which was OK because I was using Thunderbird.  (Later, I encountered a review by a user who said s/he was using it for this purpose, so I assumed they had updated the program.)  They said that working over networks could be complicated.  I wasn't sure if they meant that as a generic remark that would be relevant to all kinds of work over networks.  They seemed to rank it number two rather than one because it "lacks some of the advanced features professional users expect."  CNET's review likewise ranked it number two, but in the category of "file management."  I wasn't sure what they considered the number one program; their webpage didn't indicate which criterion they used for that ranking.  But GoodSync was the most frequently downloaded program during the prior week.  GoodSync's awards webpage mostly listed awards and positive reviews that were at least a couple of years old.  So apparently it had been created and was now coasting.  I did a search among its many reviews on CNET, looking for more info about using it on a network.  Unfortunately, CNET's links to specific reviews weren't working for me at that point, so I wasn't able to get details, but what I was able to read from the summary results was positive with the exception of one person for whom GoodSync did not work well.

There were hardware options.  SyncSharp offered a device that would synchronize via USB.  It sounded similar to The Tornado and to the Windows 7 Easy Transfer option.  I didn't want an additional device, and since ethernet was faster and was already in place, I didn't want to use USB.  For purposes of speed and also capacity, not to mention reducing dependency on external data sources, I was obviously not going to be interested in a cloud (i.e., web-based) solution, even if I had found one that offered constant, continuous, real-time synchronization.

It turned out that CNET had another category, for "data transfer & sync software."  As with some other CNET searches, I looked at the top 30 both in terms of downloads last week and user ratings.  Setting aside those that were for special purposes (e.g., Blackberry, Outlook), and focusing on those that were for Windows 7, I found that only two were free:  CopyTo Synchronizer, which had only two votes and which I therefore deemed insufficiently tested, and Microsoft Live Mesh, which had only one vote but which I was willing to assume was better developed.  The nonfree alternatives that came up in this search included BeyondSync, ViceVersa Pro, Easy Computer Sync, and Syncables 360 Premium.  I reduced this set to Live Mesh, Beyond Sync, and ViceVersa.  A search for further information on Live Mesh suggested it was web-based, more like DropBox, leaving me to focus on the other two.  A search led back to a TopTenReviews comparison -- it may have been the same one as before, but a couple of weeks had passed by thie time -- naming GoodSync (above) as No. 2, ViceVersa as No. 5, and BeyondSync as No. 10.  Of the ten, the ones offering bidirectional sync, network synchronizing, and Windows 7 support were these three plus Syncables 360, SugarSync, Laplink, and Super Flexible.  Most of the same names also appeared in a CEOWorld review.  I eliminated SugarSync as another cloud solution.  The TopTen review for Beyond Sync made it sound unappealing.  A dotTech review echoed that. 

I ran a search looking for comparisons of Syncables against the others that also sounded good for bidirectional synchronization.  A couple of reviews alerted me to the feature, evidently present in GoodSync but not all others, of being able to see which changes would overwrite.  I began to get a sense that Syncables was more of a glossy product, designed for people who wanted simple and trouble-free synchronization without necessarily having an option to scrutinize every step of what was happening.  Having been burned by the occasional backup program that would not save (and would also not tell me that it was not saving) files of a certain kind, or nested too deep, or had an umlauted character in their filenames, or were otherwise secretly exempt from what I thought was happening, I had become more inclined to use transparent software.  At least until I gained a lot of trust and experience with a program, I wanted to see what it was doing.  So this feature of GoodSync appealed to me.  I noticed, also, that a review described Liuxz Sync as being "of most use to users that need to carryout real time synchronizations over a network or between hard drives."  As I continued to look at other opinions, ViceVersa still sounded relatively good too.

On this basis, I decided to start with GoodSync, as described in a separate post.  After some days of using it, and comparing its results against an external backup drive via manual comparisons using Beyond Compare (as described in more detail in that other post), I concluded that GoodSync was a good product for this purpose.  I set its sync rules so as to check most frequently those partitions in which I was most likely to make changes.  For practical purposes, I could change files on one computer and I would see those changes on the other computer when I went looking for them.

In short, I wound up using GoodSync to synchronize files on two computers on a home network.  The files were generally available on the other computer within minutes.  I did this without using a server.  That is, the files were available locally on each computer, so that I could keep right on working if the other one went down.  I arranged backup via external drive, and I occasionally checked that external drive against the internal drive manually using Beyond Compare.  I had better performance than on a network, and was not very vulnerable to network problems; presumably I could have set up the same arrangement via crossover cable, without even having a router.  This really felt like a solution that I had been seeking for years.

Synchronizing Two Computers: Detailed Review of GoodSync

As described in another post (posted about the same time as this one), I decided to try GoodSync to keep two desktop computers running Windows 7 in constant synchronization.  This post describes how that worked out.  As with my other posts, this post presents the actual process, in all its chaotic glory.  This seems to be a good way of conveying what it was actually like to install and use the program.

I started by going to GoodSync's homepage to see if their description made the program sound like what I needed.  On their Features page I noticed, among other things, that "Analysis and Synchronization can be started when any file in sync folders changes."  I saw that GoodSync also came in an Enterprise version with appropriate features (e.g., command line) for $10 more.  The download page (for the Pro version) offered versions 8.5, 9 beta, Mac, and portable.  I would have gone with the portable version, if I had had a more consistently good experience with other portable software -- after all, why not avoid having to reinstall a program on each machine?  It was a little confusing -- they seemed to offer a separate trial version -- but on the assumption that I would just have to enter a license key within 30 days, I downloaded Pro version 8.5.  (I later found an indication that the portable version would not start automatically with Windows and could not be scheduled via Windows Scheduler.  There was no user forum on the GoodSync site.)

I had set up a home network, so I knew my computers were seeing each other.  I had also done full backups on both computers to an external drive that was now disconnected, so I was confident that even the worst screwup could not completely trash anything irreplaceable.  In fact, I went ahead, using Beyond Compare, to synchronize those backups with each other.  So both computers, at this point, had identical data. So I could change just one thing and see what happened.  (My impression of Beyond Compare was that it could have done much of the same thing as GoodSync, but would have required me to write scripts to do it.  If I was going to write scripts, I would just use the Windows equivalents of rsync and cron.)

With that in place, I installed GoodSync on one computer and ran it.  It opened a dialog inviting me to set up my first job.  I had a choice of synchronizing or backup.  The interface was like other file comparison software, with two panels side-by-side representing the two computers being synchronized.  I needed to indicate what I wanted to synchronize.  Unfortunately, it wouldn't let me designate all of computer A.  As I confirmed at their FAQs page, the best I could do was to set up separate jobs for each individual folder off the root drive!  So if I created a new root folder (say, D:\Newfolder) and forgot to set up a new GoodSync job for it, it would not be synchronized; likewise if I changed a root folder's name.  Apparently files stored in the root folder could not be synchronized in any case, unless perhaps one went to the trouble of mapping that drive and attempting a sync on the map.  I nearly dropped the program at this point.  I did try synchronizing drive D, but the OK button was grayed out when I selected that.  (Later -- after I had set up and configured individual jobs for each of a dozen different folders -- I went back and tried drive D again, and this time it worked.  I wasn't sure what happened there, but I spent the better part of an hour unnecessarily pursuing that.  When I looked at the FAQs page again at that point, I realized they were talking specifically about drive C.  Apparently they meant that other drives were fair game.)

Anyway, I went ahead with the experiment, creating a job for just one folder.  I called it D-Miscellany, since it would be synchronizing D:\Miscellany.  It was easy enough to indicate that folder on computer A, where I was working:  the list of drives on computer A came up by default.  But how to designate D:\Miscellany, the matching folder, on computer B?  When I clicked the Browse button for the right-hand pane, it showed me just that same list of drives on computer A.  The answer was that I had to look in the left side of the "Right Folder" dialog that came up when I clicked on Browse.  I had to tell the program to look, not on Local, but on LocalNet SMB.  It was greyed out at first, but I guess after computer A detected computer B, the LocalNet SMB option became available.  But when I went there and tried to select computer B, I got an error:  "Access is denied. (error 5)."  So, referring back to the struggles near the end of my post on networking, I applied to computer B the same steps I had applied to computer A, so as to make its drives available to GoodSync.  The starting point here was to right-click on drive D, using Windows Explorer on computer B, and go into Properties and tinker with the Sharing and Security tabs.  In the Sharing tab, it was Advanced Sharing > Permissions > Full Control for Everyone.  In the Security tab, I had to click Edit > Add > type "Everyone" > OK > Full Control (with Everyone selected) > Apply > wait > OK.  That enabled me to put computer B's Miscellany folder in the right-hand pane of GoodSync.  Next, I clicked GoodSync's Analyze button.  It reported that the two were "In Sync."  Good.  Sync!

OK, so would GoodSync automatically detect a new file in D:\Miscellany on computer B, or would I have to click the Analyze button manually?  Perusing the menu options, I saw that perhaps I could just click Job > Synchronize All without having to go through a separate Analyze step, but I wasn't sure about that.  I went into Job > Options.  Lots of things I could adjust if I wanted.  This looked good.  There was an option to do an Analyze or Sync (or both) every ___ hours and ___ minutes (fill in the box).  So conceivably I could set the sucker to run every minute.  I tried that.  Clicking the Sync option there automatically turned on the Analyze option as well, so apparently Analyze was the necessary first step.  Logical.  I set it for zero hours and 1 minute.  But then I deselected that and checked the "On File Change" box instead.  Did this mean it would sync a file as soon as it was changed, on either computer?  Time to try it out.  I selected On File Change, clicked Apply, and then created a little text file on computer B in D:\Miscellany.  I called it Test.txt.  GoodSync on computer A said, "Analyze is blocked by Options or Browse dialog.  Oops, OK, I clicked OK to close that dialog.  The error message didn't go away.  I had to close it manually.  Ten minutes later, it still hadn't done anything.  So I clicked on the Auto button on GoodSync's lower panel.  That just brought up the Options dialog again.  Alright, well, skip that.  I unchecked the Sync and Analyze options for "On File Change" and went back to the option of synchronizing every 1 minute.  That did something!  Almost immediately, I got an indication that it had detected that something had happened with Test.txt on computer A.  I went into computer A's D:\Miscellany folder and, sure enough, there it was.  I changed its contents and closed it.  Within about 20 seconds, Good Sync did something.  I checked on computer B.  Yes, its copy of Test.txt now had the change.  Very good.  I renamed the file on computer B, watched GoodSync react, and checked D:\Miscellany again on computer A.  This was working.  Note that the time on my computers did not precisely agree.  Apparently GoodSync had made a note of that, and was adjusting its calculations accordingly.

As I browsed GoodSync's FAQs, I saw that they also did not automatically allow system and hidden files to be copied.  I got the feeling that I had really better just double-check whatever GoodSync was doing with Beyond Compare, for a while, to make sure it was doing what I thought it was going to be doing.  I didn't know what other exceptions there might be, and that's the point:  I didn't know.  I couldn't be sure.  Even after reading the FAQs, as experience with other backup programs had taught me, there might be something else that the programmers decided not to include.

A couple of reactions, at this point, about the interface. One was that I wished that GoodSync were capable of more compressed viewing. There was a lot of white space among lines, such that relatively few files being compared were visible onscreen at any one time. Also, I wanted the Log Window to be more user-friendly. I think, instead of using up that valuable real estate with tiny print stretching across the full width at the bottom of the screen, showing only the last few lines, it would have been better to provide a more readable version behind a button, and to organize relevant options (e.g., show all errors) around that pop-up.  Eventually, I also found myself wishing that, when I excluded a folder from consideration, it would automatically roll up into one line (i.e., no longer display all its subfolders and files) and change color or otherwise be marked as visibly excluded, so that it was not distracting me and taking up screen real estate.  Also, it was odd not to have a save option of some sort on the Job menu.  It seemed like some items from the Tools menu belonged on the Job menu, and vice versa.  On the Tools list as it was, I assumed at first that "Export Job List" meant that it would save, to a designated file, the whole setup -- all of the jobs and settings that I had opened, exactly as the whole screen was now set up -- but later it occurred to me that maybe this meant, literally, only the list, such that I also had to do a separate save operation for the individual jobs.  This was not something I wanted to get wrong, but it was also not something I wanted to spend more time researching.  I guessed this was like Beyond Compare's idea of saving individual sessions within a workspace, and being able to save and load workspaces, but here it wasn't as clear.

For the time being, I closed the GoodSync window.  It retired to the system tray.  I could right-click its icon down there to reopen it.  I did that.  I adjusted its Tools > Program Options.  There were pop-up help dialogs for the various settings, and these were generally pretty good.  They needed to be gone over by an English major, though, with some attention to clarity of references.  For instance, one said, "If checked then GoodSync will create left/right sync folders if they are not present."  I had no idea what a left/right sync folder was, and there was no link there.  I didn't want to lose my place, as I was going down through that help dialog, so I ran a Google search to get an explanation.  The only search result with any information was for the GoodSync Manual, but it just repeated what was in that help dialog (or maybe it was the other way around).  There was no explanation anywhere, as far as I could tell, of what a sync folder was.  I gathered, from several oblique references, that GoodSync would optionally create a sync folder in each folder that it was synchronizing.  To test this, I turned on the option to create sync folders, added a new file to D:\Miscellany, and sat back to watch.  Sure enough, both computers now had a system folder called D:\Miscellany\_gsdata_.  Needless to say, I didn't want that clutter.  I turned off the option and made a change to my test text file.  GoodSync mirrored it without complaint.  This did not mesh with the manual's indication that there would be an error message if the sync folder was not found.  Oh, but now I saw that was probably because unchecking the option did not remove the sync folder. OK, well, then, with the option off, I deleted _gsdata_ from computer A's D:\Miscellany folder and then changed Test.txt again.  Whoa.  _gsdata_ was back.  Alright, what if I deleted it on both machines?

This called for a pause to reflect.  Each synchronized folder would have a _gsdata_ folder.  It would not be hidden from me, since I normally displayed hidden folders.  But its presence would be a relatively minor irritant.  After all, I had overlooked the intrusions of desktop.ini and Thumbs.db for years.  I was not sure why GoodSync could not store all of that synchronization data in a cache, in a location of my choosing, as other programs (e.g., Copernic Desktop Search) would do.  But if I had a good synchronization solution here otherwise, I would much rather have these headaches than deal with a program that did not do its job as well.  Another concern was that I wasn't entirely confident that GoodSync was going to do the job right.  I was not sensing a commitment to give me a transparent, thorough synchronization that I could trust without having to check up on.  Still, it wasn't a backup program per se, so possibly this would work out OK.  I would probably just have to use it for a while, to see how things turned out in practice.  (If I do not comment on this program in future posts, it is likely that I stopped using it.)  (This concern about the _gsdata_ folder was reduced, later, when I was able to synchronize whole drives rather than individual folders.  At that point, the only _gsdata_ folders were in the drive root, i.e., at D:\ and E:\.)

I decided to continue.  There wasn't much else to comment on in Tools > Program Options at this point, except to say that they did have a global Filters section there, so they surely could have allowed for folder filtering.  Over in Job > Options, I noticed that they had an option to save one or more versions of deleted files for as long as the user might wish, or until disk space was gone.  There was another, job-specific Filters option here.  I didn't want exceptions popping up in my Beyond Compare double-check, so I didn't set filters; Beyond Compare would notify me about everything.  (Somewhere along the way, I noticed that the contents of the help or explanation window at the bottom of these options dialogs would change as I moused over individual settings.  This was very practical and helpful.)

Moving on, I was now back at the Job > Options > Auto section.  The option of running a synchronization every ___ minutes seemed sufficient, so I didn't go with any of the other scheduling options.  This D:\Miscellany folder rarely changed.  I could probably set it to sync every couple of weeks, and just use GoodSync's manual Sync button if I happened to think of it after I did make a change in that folder.  There was an option not to do a sync if more than a specified percentage of files changed.  This, they felt, could be a sign that something was wrong and that manual intervention was appropriate.  I decided to leave that at their default 10% setting, to see if that worked for me.  Finally, there was an option to "Auto Clear the tree after Sync or Analyze with no changes."  I had no idea what this was about, probably because I didn't know what tree they were referring to, and the help file did not provide an explanation.  (I have pointed GoodSync to this post.  Hopefully they will read it and address some of these concerns.  If so, perhaps they will even post a comment to say so.)

Job > Options also offered Scripts and Advanced sections.  In the Advanced section, I checked some things they had not checked, for added file integrity.  In a case of "ask and it shall be given," I saw that they did have a job-specific "No _gsdata_ folder" option.  The context-specific help note was too big for the little window at the bottom of the dialog, but I found it repeated in the manual.  It said this:

No _gsdata_ folder (un-checked by default)

If checked then do not create _GSDATA_ folder in the Right (remote) sync folder of the job, instead create _GSDATA_ folder in the GoodSync profile folder on this computer.
If not checked then create _GSDATA_ folder in both sync folders of this job.
Still use _gsdata_ folder on the left side of the job.
Use this option only for folders that are not sync centers (all jobs only read from this folder or only write to this folder).
If _gsdata_ folder is disabled then these options are disabled too: Save Previous/Past Version.

This was not entirely clear.  It seemed to be saying, for one thing, that if I did opt to save deleted copies of a file, those copies would be saved in _gsdata_.  Possibly this was the reason for putting the _gsdata_ folders within the individual folders rather than in a separate cache folder:  copying or moving them from their folder of origin to a cache folder could take a lot of time, in the case of many or large files.  It still seemed like something that users should be able to choose, though.  This message also seemed to say that, if I did opt out of _gsdata_ folders, my drive C would swell as I larded it down with data files from synchronized folders on other drives.  Even after opting out, the program would apparently create a _gsdata_ folder in computer A nonetheless.  I didn't know what a sync center was, it wasn't explained in the manual, and the parenthetical explanation here wasn't clear.  I decided not to opt out of _gsdata_.  Finally, I changed their folder and file links options to "as is."

Whew.  OK.  I had set the options for the D-Miscellany job.  Now I had about ten more to folder jobs to set up.  That was going to be a pain, and the odds that I would get all the settings right without overlooking anything were just about nil.  I would have to adjust timing for different folders, but otherwise most of their settings, it seemed, should be the same.  I wished GoodSync had an option, like Beyond Compare, to set default settings for all jobs, so that I wouldn't have to do the same things for each job individually.  Then I noticed an option to create a job template.  Just what the doctor ordered.  I wanted to save D-Miscellany as a template, where all I would have to do would be to change the folders being synchronized.  To do this, I went to Job > Save as template.  The location was C:\Users\Administrator\AppData\Roaming\GoodSync\Templates.  I assumed I would have to make a manual copy of the resulting Default Template.gst file to my D:\Installation\Saved Settings folder if I wanted it to survive a Windows reinstallation.  To use the template, I went to Job > New > From template.  I soon found I could just click the yellow plus symbol at the left end of the tab bar to invoke this dialog.  I was able to drag tabs into a preferred order, which was a nice touch.  Specifying the folders to be synchronized took time:  the computer apparently had to re-connect with the network each time I clicked on a Browse button.  Each recognition step, two per job, took about 10 seconds on the left side and about 20-30 seconds on the right.  Not much in the grand scheme of life, but a drag when you're creating a dozen or more jobs.  This made me wish I had been able to just right-click on a static (refreshable) list of folders to create a job.  Then I went back into each job and changed how frequently it would do its auto-sync.  I decided to try again with the auto sync "on file change," reasoning that surely that part of the program must be functional, if not instantaneous.  What I found was that it would check for file changes frequently -- every minute, from the looks of it -- so this was not the way to minimize system resource usage.  So I changed them all back to a specified number of minutes after all.  When I was done with this, I went to Tools > Export Job List and saved the .tix file to D, again to survive a Win7 reinstallation.

While I was doing all that, of course, those newly created jobs, modeled on D-Miscellany, were all springing into action to synchronize computer A with computer B.  I had done a few file moves on computer A since using Beyond Compare to synchronize computer A with an external drive.  So in a few moments I would go into Beyond Compare and compare computer A against that external drive again.

But first, something weird was happening.  I had finished setting up the folder comparisons on the folders in drive D, but had neglected a couple of folders I had on drive E.  When I began to set those up, I discovered that GoodSync would, in fact, allow me to do a comparison of the entire drive, rather than having to designate individual folders.  This was what I wanted, but it was what they had said they couldn't do and what I had not actually been able to do on drive D.  I went back and reviewed the FAQ page statement cited above.  Well, it said that I could not back up all of drive C.  It didn't say anything about drives D, E, ... So had I made a lot of extra work for myself -- was there actually a way of doing all of drive D at once, some way that I had just overlooked?

At this point, I discovered that GoodSync did have a right-click option to exclude the Recycle Bin and other items.  Unfortunately, when I used that option, it ignored me.  There continued to be a yellow exclamation mark on the tab for that drive, and I continued to get error messages indicating that access to the recycle bin was denied.  A yellow banner said, "Sync has finished with Errors or Unresolved Conflicts," and the log said, "Click Errors button to see sync errors."  But there was no Errors button.  I stopped GoodSync during a sync, so as to hunt for the log, but that made the program crash.  It sent off a crash report to the GoodSync people.

Alright, then, this was a good time to do that comparison with Beyond Compare.  I was not pleased with the results.  Changes that I had made on computer A, after synchronizing via external drive with computer B, had been undone.  That is, computer A had been restored to the earlier condition represented by computer B.  The specific problem seemed to be that files I had deleted on computer A had been restored to computer A from computer B, instead of being deleted from computer B.  Those may have been the only sync problems.  I restarted GoodSync and checked the file deletion settings.  In Job > Options > General, I had already checked "Propagate Deletions," which should have taken care of it.  In Job > Options > Auto, in response to "Automatically resolve conflicts," I had designated "Do Not Copy" instead of "Left to Right," "Right to Left," or "Newer File Wins."  I changed this to "Newer File Wins," which was my preference, now that I thought of it, but this did not seem to explain the problem.  I had not checked "Auto Clear the tree after Sync," and was irritated that the list of changes would disappear from the screen before I had a chance to look at them (if, that is, it ran another sync before I looked at the program again).  Oddly, it seemed that computer B, where I was *not* running GoodSync, wound up with the correct files.

In short, my impression at that point was that there were some bugs in the program.  At this point, I was wishing that Beyond Compare offered a sync program.  But after that first scare, GoodSync seemed to behave more as I had expected.  Hopefully this did not mean that it finally won and was able to delete the file versions that it did not like, that I happened to prefer. 

I was not sure exactly what happened in that first encounter.  It may somehow have been my fault.  During the next 48 hours, GoodSync won my confidence.  And then I used it to compare an existing partition with a new one, where I had added just a few programs on the new one via USB jump drive, and a similar thing happened again:  the Analyze screen indicated that it was prepared to wipe out the existing partition and replace it with the new one.  Then I realized that, of course, it should:  the new partition had been changed more recently.  It didn't go through with it, though, because it detected that more than 10% of the partition was going to change.  So this was more like a warning to me, to not treat GoodSync as a magic wand that would automatically make everything right.

After several days of usage, with careful comparisons of all changes against an external backup using Beyond Compare, GoodSync seemed to be a solid, reliable way to keep two computers synchronized.