Robocopy Commands for File and Folder Synchronization

I wanted to know if Robocopy could replace synchronization programs like GoodSync and Allway Sync.  I decided to take a look at its options.  This post describes what I found.

The manual for Robocopy itself seemed to confirm my prior sense that Robocopy was a unidirectional copying utility. That is, you could command it to copy from A to B, or from B to A, but not bidirectionally between A and B, never mind multidirectionally. In particular, manual page 16, discussing Robocopy's /XO option, said this:

The most appropriate use for /XO is to synchronize two directory trees so that they can be updated simultaneously in separate areas. To ensure that the latest files are present in both directory trees, copy with /XO first in one direction and then in the other.

So apparently synchronization with Robocopy would require reciprocal commands: one to copy newer files from A to B, and another, otherwise identical, command to copy from B to A.  Wikipedia noted that Robocopy would not copy open files, but that seemed to be true of GoodSync as well.  I didn't expect it to be a problem for my purposes.

(The information from the manual seemed pretty similar to what I got from a command-line "robocopy /?" command.  A Windows Server webpage provided yet another list of Robocopy commands.  I had noticed, though, that Wikipedia said Robocopy was not entirely consistent between platforms.  It seemed I had better verify that my solutions would work on Windows 7 specifically.)

I wasn't yet sure what kind of analysis /XO or other Robocopy commands would perform.  I assumed, but was not yet positive, that they would take account of time as well as date, down to the minute if not the second.  This thought prompted me to look into something that would improve the present gap of about 20 seconds between the clocks on the two computers -- something more fun than just resetting the clock manually, that is. It seemed I would want one or two minutes' tolerance in timestamp comparisons in any case; but in a separate post, I took a closer look at possible solutions to get the two machines closer to chronological synchronization.  (Later I saw that the manual (p. 17) said, "File-time granularity is 100 nanoseconds on NTFS, and two seconds on FAT.")

The next step seemed to be to peruse the Robocopy command-line options, to see which ones might work for my synchronization project.  The manual (pp. 7-11) provided a concise command-line reference.   (Apparently lowercase versions of the commands worked; I use uppercase here for visibility.)  The options that seemed most relevant were as follows, leaving out those that would be included by default:

/S copy subdirectories, excluding empty ones

/ZB resume a copying task from a point of previous failure (presumably instead of starting over), but switch to backup mode if restartable mode fails (as when using an unreliable network); but Wikipedia said that /Z would slow copying significantly

/MOT:X and /MON:Y   monitor the source directory and run Robocopy again when X number of files have changed and Y minutes have elapsed

/XD   exclude specified directories

/XO   don't overwrite destination files with source files having the same name and an older timestamp

/R   specify the number of retries

/W   specify wait time between retries

/REG   save /R and /W in registry as default settings

/L   list files

/V   produce verbose output

/TS   display source file timestamps in output log

/FP   display full pathnames in output log

/NC   suppress output of Robocopy file class information


/NS   suppress output of file and directory sizes

/NJH and /NJS   turn off logging of job header and summary

/ETA   show estimated time of completion

/LOG:filename or /LOG+:filename   redirect output to named file, overwriting or (+) appending if it already exists

/TEE   display output as well as writing LOG

/JOB:jobfile   read parameters from jobfile; optionally /IF to include files with specified names or paths; optionally /SD:path and /DD:path to specify source and destination directories; optionally /QUIT to view job file contents without any actual copying

/SAVE:jobfile   write current parameter settings to jobfile

This was an intimidating set of potentially relevant options.  There were many things that could go wrong with so many choices.  I decided to look for examples and more information.  As the manual said (p. 22), these options could result in long, unwieldy commands, and it did appear that the JOB options would help with that.  I ran a search with that in mind.  It led most immediately to some general-reference sites (e.g., SS64, PowerCram, FixMyITSystem).  I tried again.  That search led almost nowhere.  Likewise another, seemingly broader search.

Eventually, I found an eHow website that said I could use XCopy to synchronize computers.  Was I trying to use the wrong tool?  It didn't seem so.  I saw indications that Robocopy had more options and better performance, and that use of XCopy was deprecated.  But why was I not seeing tons of obvious references to the use of Robocopy JOB options for synchronization?  (Another possibility with lots of options:  XXCOPY.  But the least expensive version for networked computers cost $100, and its reception at CNET was underwhelming.)

I tried another search, focusing only on Robocopy job options.  Posts at Serverfault and Skonet seemed to suggest that I could work up my desired Robocopy command on the command line itself, and then add the /SAVE option to save it to a file, and that my job options would go into a separate file with an .RCJ extension.  I wasn't sure exactly how all that would work in practice, but at least there seemed to be a starting point.

It appeared that the slash (/) character was the signal, in the job options file and probably also for Robocopy generally, indicating that one option had ended and a new one was starting.  So, for instance, if I gave the /XD command, I could then proceed to list a boatload of directories to exclude, without worrying too much about format -- putting them on the same line, putting each on a different line, or whatever.  Everything I wrote after /XD would be treated as another directory to exclude, until I came to another / command (e.g., /S) or else reached the end of the job options file.

One thing that was not clear to me, from the foregoing options:  what if I had deleted a file on computer B, intending that synchronization should delete it on computer A as well?  GoodSync knew how to interpret such situations.  I suspected that Robocopy, run in a reciprocal setup starting with robocopy A > B, would just restore the file from computer A to computer B.  That, as I found in tinkering, was what Beyond Compare might do.  In other words, there was a real difference between a backup program and a synchronization program.  The former would just make sure that the target contained what was on the source; the latter would keep track of what had been removed from each side, and would decide whether the removal occurred later than the most recent update of those files on the other side.  I could control such things manually, more easily in Beyond Compare than in robocopy, but for an automated solution I would need a program that would keep track of dates and times of deletions as well as of file changes.  I posted a question on this, just to make sure I was understanding Robocopy correctly.  The response confirmed it, and suggested using regular synchronization programs.

Transitioning Away from Windows Toward Ubuntu: The Next Step

In September 2008, I reached a point of relative stasis in the development of my computer setup.  I was using two computers, each with its own monitor, but with a common mouse and keyboard via KVM switch.  Both computers were dual-boot setups, so that I could have gone into Windows or Ubuntu on either one; but I tended to just use one or the other.  Specifically, on one computer, I installed Ubuntu and then used VMware Workstation to run Windows XP in virtual machines (VMs).  On the other computer, I was almost always running Windows XP, rarely going into Ubuntu.

I had assumed, there at the end of summer 2008, that I would be revisiting this layout in summer 2009.  Generally, that didn't happen.  Instead, two years passed.  At this point in 2010, however, I found that there had been some developments, such that this system could evolve.

One development was that, sometime in the intervening two years, I set up a third desktop computer.  It was mostly a collection of hand-me-down parts, but it ran Windows XP well enough.  If I really needed to do something in XP, I probably could do it there.  Moreover, I had acquired a laptop, for a bargain price of around $350 -- plus another $250 or so that I didn't entirely expect to spend, when I went to the store, for an extended warranty, laptop sleeve, screen protector, wireless mouse, etc.  Such a deal!  Anyway, neither the laptop nor the bucket-shop computer were anything to write home about, but they would serve in a pinch.

Another development was that I had run into several systems problems on the primary Windows XP machine.  It still ran stably, but I was getting occasional flaky problems.  Probably there would have been a list of them, if I had taken a minute to list the ways in which the machine was not performing up to snuff.

Probably the most worrisome such problem was that I had reached a dead-end in my efforts to install Windows updates on that primary machine.  It just wouldn't install them.  I had revisited the problem repeatedly.  At this point, it would probably have been more effective to just reinstall WinXP from scratch on that machine.

That update situation had been persisting for a while.  It hadn't bothered me much.  Recently, though, something else had happened.  I had somehow started using McAfee antivirus software, and I had just discovered that McAfee had been piling up gigabytes of stuff in .bup files.  I was concerned that this feat would have been possible only by mixing in data files; therefore, I had begun an effort to compare against an old backup and figure out what, if anything, might have gone missing over the past several months or longer.  So that was the end of McAfee, for me, but it was also a wake-up call to take computer security more seriously.  That meant keeping updates installed, but perhaps it also meant it was time to continue my migration away from Windows.

For such reasons, I thought it might be time to consider running both of my primary computers under the same kind of Ubuntu - VMware - WinXP VM setup.  If I needed XP, I could still drop back into the dual-boot, or just use that hand-me-down computer.  If I needed Vista or Windows 7, I could use the laptop, which was presently running the one but which apparently qualified for a free upgrade to the other.

In addition to those developments on the Windows side, things had also been happening on the Ubuntu side of the equation.  First, the good things.  Ubuntu was looking good and running well.  I had learned a bit more about Wine.  Generally, I was continuing to become more familiar and comfortable with the world beyond Windows.  I still had occasional issues with VMware, but generally nothing lethal.  As an additional consideration, Oracle had created the impression that it might be positioning VirtualBox to compete effectively with VMware.  Even without that, it was still nice that I could leave a WinXP VM running in Ubuntu for a week without needing to reboot it, while that would just never work on the native WinXP machine.

At the same time, after these several years of experimenting with Ubuntu, I had to agree with someone who had said that Linux distributions still tended to be terribly unpolished in comparison with Windows.  Somewhat contrary to my expectations, I was not finding many instances in which Ubuntu programs were delivering superior functionality and reliability.  For instance, I had bought a copy of Beyond Compare, a file synchronization program.  (I subsequently realized that I probably could have gotten by with a freeware alternative, but whatever.)  There were Windows and Linux versions of Beyond Compare.  The Linux version did not seem to be very actively developed, and it was having problems that I wasn't having in the Windows version.  The same was true elsewhere.  I was still using IrfanView, which did not yet have a Linux version; I was still using CoolEdit 2000, because it had features that Audacity did not provide.  Generally, I was finding that Ubuntu was great as an operating system; I was finding it useful as an underlying layer, to handle tasks that WinXP couldn't handle (e.g., delete files that WinXP couldn't delete); I found that WinXP running in a VM on Ubuntu was more stable (although slower) than a native WinXP installation.  But at the point of application, for my purposes, Ubuntu wasn't a serious competitor against Windows XP.  And I was increasingly unwilling to invest the time to learn how to do everything in two or more different ways.

My conclusion, at this point, was that the best of both worlds called for running Windows XP within virtual machines (in VMware or otherwise) on an Ubuntu operating system base, on an Ubuntu/XP dual-boot computer.  I had already worked through many of the issues in this sort of setup, and could therefore hope to be efficient and preserve multiple troubleshooting options without too much of a time investment.  If Windows 7, Ubuntu, or some other operating system (OS) began to display capabilities that I badly needed, I would hopefully be able to incorporate those OSs into my setup, one computer at a time, without too much disruption overall.