Showing posts with label computers. Show all posts
Showing posts with label computers. Show all posts

Saturday, February 25, 2012

Robocopy Commands for File and Folder Synchronization

I wanted to know if Robocopy could replace synchronization programs like GoodSync and Allway Sync.  I decided to take a look at its options.  This post describes what I found.

The manual for Robocopy itself seemed to confirm my prior sense that Robocopy was a unidirectional copying utility. That is, you could command it to copy from A to B, or from B to A, but not bidirectionally between A and B, never mind multidirectionally. In particular, manual page 16, discussing Robocopy's /XO option, said this:

The most appropriate use for /XO is to synchronize two directory trees so that they can be updated simultaneously in separate areas. To ensure that the latest files are present in both directory trees, copy with /XO first in one direction and then in the other.
So apparently synchronization with Robocopy would require reciprocal commands: one to copy newer files from A to B, and another, otherwise identical, command to copy from B to A.  Wikipedia noted that Robocopy would not copy open files, but that seemed to be true of GoodSync as well.  I didn't expect it to be a problem for my purposes.

(The information from the manual seemed pretty similar to what I got from a command-line "robocopy /?" command.  A Windows Server webpage provided yet another list of Robocopy commands.  I had noticed, though, that Wikipedia said Robocopy was not entirely consistent between platforms.  It seemed I had better verify that my solutions would work on Windows 7 specifically.)

I wasn't yet sure what kind of analysis /XO or other Robocopy commands would perform.  I assumed, but was not yet positive, that they would take account of time as well as date, down to the minute if not the second.  This thought prompted me to look into something that would improve the present gap of about 20 seconds between the clocks on the two computers -- something more fun than just resetting the clock manually, that is. It seemed I would want one or two minutes' tolerance in timestamp comparisons in any case; but in a separate post, I took a closer look at possible solutions to get the two machines closer to chronological synchronization.  (Later I saw that the manual (p. 17) said, "File-time granularity is 100 nanoseconds on NTFS, and two seconds on FAT.")

The next step seemed to be to peruse the Robocopy command-line options, to see which ones might work for my synchronization project.  The manual (pp. 7-11) provided a concise command-line reference.   (Apparently lowercase versions of the commands worked; I use uppercase here for visibility.)  The options that seemed most relevant were as follows, leaving out those that would be included by default:
/S copy subdirectories, excluding empty ones

/ZB resume a copying task from a point of previous failure (presumably instead of starting over), but switch to backup mode if restartable mode fails (as when using an unreliable network); but Wikipedia said that /Z would slow copying significantly

/MOT:X and /MON:Y   monitor the source directory and run Robocopy again when X number of files have changed and Y minutes have elapsed

/XD   exclude specified directories

/XO   don't overwrite destination files with source files having the same name and an older timestamp

/R   specify the number of retries

/W   specify wait time between retries

/REG   save /R and /W in registry as default settings

/L   list files

/V   produce verbose output

/TS   display source file timestamps in output log

/FP   display full pathnames in output log

/NC   suppress output of Robocopy file class information


/NS   suppress output of file and directory sizes

/NJH and /NJS   turn off logging of job header and summary

/ETA   show estimated time of completion

/LOG:filename or /LOG+:filename   redirect output to named file, overwriting or (+) appending if it already exists

/TEE   display output as well as writing LOG

/JOB:jobfile   read parameters from jobfile; optionally /IF to include files with specified names or paths; optionally /SD:path and /DD:path to specify source and destination directories; optionally /QUIT to view job file contents without any actual copying

/SAVE:jobfile   write current parameter settings to jobfile
This was an intimidating set of potentially relevant options.  There were many things that could go wrong with so many choices.  I decided to look for examples and more information.  As the manual said (p. 22), these options could result in long, unwieldy commands, and it did appear that the JOB options would help with that.  I ran a search with that in mind.  It led most immediately to some general-reference sites (e.g., SS64, PowerCram, FixMyITSystem).  I tried again.  That search led almost nowhere.  Likewise another, seemingly broader search.

Eventually, I found an eHow website that said I could use XCopy to synchronize computers.  Was I trying to use the wrong tool?  It didn't seem so.  I saw indications that Robocopy had more options and better performance, and that use of XCopy was deprecated.  But why was I not seeing tons of obvious references to the use of Robocopy JOB options for synchronization?  (Another possibility with lots of options:  XXCOPY.  But the least expensive version for networked computers cost $100, and its reception at CNET was underwhelming.)

I tried another search, focusing only on Robocopy job options.  Posts at Serverfault and Skonet seemed to suggest that I could work up my desired Robocopy command on the command line itself, and then add the /SAVE option to save it to a file, and that my job options would go into a separate file with an .RCJ extension.  I wasn't sure exactly how all that would work in practice, but at least there seemed to be a starting point.

It appeared that the slash (/) character was the signal, in the job options file and probably also for Robocopy generally, indicating that one option had ended and a new one was starting.  So, for instance, if I gave the /XD command, I could then proceed to list a boatload of directories to exclude, without worrying too much about format -- putting them on the same line, putting each on a different line, or whatever.  Everything I wrote after /XD would be treated as another directory to exclude, until I came to another / command (e.g., /S) or else reached the end of the job options file.

One thing that was not clear to me, from the foregoing options:  what if I had deleted a file on computer B, intending that synchronization should delete it on computer A as well?  GoodSync knew how to interpret such situations.  I suspected that Robocopy, run in a reciprocal setup starting with robocopy A > B, would just restore the file from computer A to computer B.  That, as I found in tinkering, was what Beyond Compare might do.  In other words, there was a real difference between a backup program and a synchronization program.  The former would just make sure that the target contained what was on the source; the latter would keep track of what had been removed from each side, and would decide whether the removal occurred later than the most recent update of those files on the other side.  I could control such things manually, more easily in Beyond Compare than in robocopy, but for an automated solution I would need a program that would keep track of dates and times of deletions as well as of file changes.  I posted a question on this, just to make sure I was understanding Robocopy correctly.  The response confirmed it, and suggested using regular synchronization programs.

Saturday, January 22, 2011

Synchronizing Two Computers: Detailed Review of GoodSync

As described in another post (posted about the same time as this one), I decided to try GoodSync to keep two desktop computers running Windows 7 in constant synchronization.  This post describes how that worked out.  As with my other posts, this post presents the actual process, in all its chaotic glory.  This seems to be a good way of conveying what it was actually like to install and use the program.

I started by going to GoodSync's homepage to see if their description made the program sound like what I needed.  On their Features page I noticed, among other things, that "Analysis and Synchronization can be started when any file in sync folders changes."  I saw that GoodSync also came in an Enterprise version with appropriate features (e.g., command line) for $10 more.  The download page (for the Pro version) offered versions 8.5, 9 beta, Mac, and portable.  I would have gone with the portable version, if I had had a more consistently good experience with other portable software -- after all, why not avoid having to reinstall a program on each machine?  It was a little confusing -- they seemed to offer a separate trial version -- but on the assumption that I would just have to enter a license key within 30 days, I downloaded Pro version 8.5.  (I later found an indication that the portable version would not start automatically with Windows and could not be scheduled via Windows Scheduler.  There was no user forum on the GoodSync site.)

I had set up a home network, so I knew my computers were seeing each other.  I had also done full backups on both computers to an external drive that was now disconnected, so I was confident that even the worst screwup could not completely trash anything irreplaceable.  In fact, I went ahead, using Beyond Compare, to synchronize those backups with each other.  So both computers, at this point, had identical data. So I could change just one thing and see what happened.  (My impression of Beyond Compare was that it could have done much of the same thing as GoodSync, but would have required me to write scripts to do it.  If I was going to write scripts, I would just use the Windows equivalents of rsync and cron.)

With that in place, I installed GoodSync on one computer and ran it.  It opened a dialog inviting me to set up my first job.  I had a choice of synchronizing or backup.  The interface was like other file comparison software, with two panels side-by-side representing the two computers being synchronized.  I needed to indicate what I wanted to synchronize.  Unfortunately, it wouldn't let me designate all of computer A.  As I confirmed at their FAQs page, the best I could do was to set up separate jobs for each individual folder off the root drive!  So if I created a new root folder (say, D:\Newfolder) and forgot to set up a new GoodSync job for it, it would not be synchronized; likewise if I changed a root folder's name.  Apparently files stored in the root folder could not be synchronized in any case, unless perhaps one went to the trouble of mapping that drive and attempting a sync on the map.  I nearly dropped the program at this point.  I did try synchronizing drive D, but the OK button was grayed out when I selected that.  (Later -- after I had set up and configured individual jobs for each of a dozen different folders -- I went back and tried drive D again, and this time it worked.  I wasn't sure what happened there, but I spent the better part of an hour unnecessarily pursuing that.  When I looked at the FAQs page again at that point, I realized they were talking specifically about drive C.  Apparently they meant that other drives were fair game.)

Anyway, I went ahead with the experiment, creating a job for just one folder.  I called it D-Miscellany, since it would be synchronizing D:\Miscellany.  It was easy enough to indicate that folder on computer A, where I was working:  the list of drives on computer A came up by default.  But how to designate D:\Miscellany, the matching folder, on computer B?  When I clicked the Browse button for the right-hand pane, it showed me just that same list of drives on computer A.  The answer was that I had to look in the left side of the "Right Folder" dialog that came up when I clicked on Browse.  I had to tell the program to look, not on Local, but on LocalNet SMB.  It was greyed out at first, but I guess after computer A detected computer B, the LocalNet SMB option became available.  But when I went there and tried to select computer B, I got an error:  "Access is denied. (error 5)."  So, referring back to the struggles near the end of my post on networking, I applied to computer B the same steps I had applied to computer A, so as to make its drives available to GoodSync.  The starting point here was to right-click on drive D, using Windows Explorer on computer B, and go into Properties and tinker with the Sharing and Security tabs.  In the Sharing tab, it was Advanced Sharing > Permissions > Full Control for Everyone.  In the Security tab, I had to click Edit > Add > type "Everyone" > OK > Full Control (with Everyone selected) > Apply > wait > OK.  That enabled me to put computer B's Miscellany folder in the right-hand pane of GoodSync.  Next, I clicked GoodSync's Analyze button.  It reported that the two were "In Sync."  Good.  Sync!

OK, so would GoodSync automatically detect a new file in D:\Miscellany on computer B, or would I have to click the Analyze button manually?  Perusing the menu options, I saw that perhaps I could just click Job > Synchronize All without having to go through a separate Analyze step, but I wasn't sure about that.  I went into Job > Options.  Lots of things I could adjust if I wanted.  This looked good.  There was an option to do an Analyze or Sync (or both) every ___ hours and ___ minutes (fill in the box).  So conceivably I could set the sucker to run every minute.  I tried that.  Clicking the Sync option there automatically turned on the Analyze option as well, so apparently Analyze was the necessary first step.  Logical.  I set it for zero hours and 1 minute.  But then I deselected that and checked the "On File Change" box instead.  Did this mean it would sync a file as soon as it was changed, on either computer?  Time to try it out.  I selected On File Change, clicked Apply, and then created a little text file on computer B in D:\Miscellany.  I called it Test.txt.  GoodSync on computer A said, "Analyze is blocked by Options or Browse dialog.  Oops, OK, I clicked OK to close that dialog.  The error message didn't go away.  I had to close it manually.  Ten minutes later, it still hadn't done anything.  So I clicked on the Auto button on GoodSync's lower panel.  That just brought up the Options dialog again.  Alright, well, skip that.  I unchecked the Sync and Analyze options for "On File Change" and went back to the option of synchronizing every 1 minute.  That did something!  Almost immediately, I got an indication that it had detected that something had happened with Test.txt on computer A.  I went into computer A's D:\Miscellany folder and, sure enough, there it was.  I changed its contents and closed it.  Within about 20 seconds, Good Sync did something.  I checked on computer B.  Yes, its copy of Test.txt now had the change.  Very good.  I renamed the file on computer B, watched GoodSync react, and checked D:\Miscellany again on computer A.  This was working.  Note that the time on my computers did not precisely agree.  Apparently GoodSync had made a note of that, and was adjusting its calculations accordingly.

As I browsed GoodSync's FAQs, I saw that they also did not automatically allow system and hidden files to be copied.  I got the feeling that I had really better just double-check whatever GoodSync was doing with Beyond Compare, for a while, to make sure it was doing what I thought it was going to be doing.  I didn't know what other exceptions there might be, and that's the point:  I didn't know.  I couldn't be sure.  Even after reading the FAQs, as experience with other backup programs had taught me, there might be something else that the programmers decided not to include.

A couple of reactions, at this point, about the interface. One was that I wished that GoodSync were capable of more compressed viewing. There was a lot of white space among lines, such that relatively few files being compared were visible onscreen at any one time. Also, I wanted the Log Window to be more user-friendly. I think, instead of using up that valuable real estate with tiny print stretching across the full width at the bottom of the screen, showing only the last few lines, it would have been better to provide a more readable version behind a button, and to organize relevant options (e.g., show all errors) around that pop-up.  Eventually, I also found myself wishing that, when I excluded a folder from consideration, it would automatically roll up into one line (i.e., no longer display all its subfolders and files) and change color or otherwise be marked as visibly excluded, so that it was not distracting me and taking up screen real estate.  Also, it was odd not to have a save option of some sort on the Job menu.  It seemed like some items from the Tools menu belonged on the Job menu, and vice versa.  On the Tools list as it was, I assumed at first that "Export Job List" meant that it would save, to a designated file, the whole setup -- all of the jobs and settings that I had opened, exactly as the whole screen was now set up -- but later it occurred to me that maybe this meant, literally, only the list, such that I also had to do a separate save operation for the individual jobs.  This was not something I wanted to get wrong, but it was also not something I wanted to spend more time researching.  I guessed this was like Beyond Compare's idea of saving individual sessions within a workspace, and being able to save and load workspaces, but here it wasn't as clear.

For the time being, I closed the GoodSync window.  It retired to the system tray.  I could right-click its icon down there to reopen it.  I did that.  I adjusted its Tools > Program Options.  There were pop-up help dialogs for the various settings, and these were generally pretty good.  They needed to be gone over by an English major, though, with some attention to clarity of references.  For instance, one said, "If checked then GoodSync will create left/right sync folders if they are not present."  I had no idea what a left/right sync folder was, and there was no link there.  I didn't want to lose my place, as I was going down through that help dialog, so I ran a Google search to get an explanation.  The only search result with any information was for the GoodSync Manual, but it just repeated what was in that help dialog (or maybe it was the other way around).  There was no explanation anywhere, as far as I could tell, of what a sync folder was.  I gathered, from several oblique references, that GoodSync would optionally create a sync folder in each folder that it was synchronizing.  To test this, I turned on the option to create sync folders, added a new file to D:\Miscellany, and sat back to watch.  Sure enough, both computers now had a system folder called D:\Miscellany\_gsdata_.  Needless to say, I didn't want that clutter.  I turned off the option and made a change to my test text file.  GoodSync mirrored it without complaint.  This did not mesh with the manual's indication that there would be an error message if the sync folder was not found.  Oh, but now I saw that was probably because unchecking the option did not remove the sync folder. OK, well, then, with the option off, I deleted _gsdata_ from computer A's D:\Miscellany folder and then changed Test.txt again.  Whoa.  _gsdata_ was back.  Alright, what if I deleted it on both machines?

This called for a pause to reflect.  Each synchronized folder would have a _gsdata_ folder.  It would not be hidden from me, since I normally displayed hidden folders.  But its presence would be a relatively minor irritant.  After all, I had overlooked the intrusions of desktop.ini and Thumbs.db for years.  I was not sure why GoodSync could not store all of that synchronization data in a cache, in a location of my choosing, as other programs (e.g., Copernic Desktop Search) would do.  But if I had a good synchronization solution here otherwise, I would much rather have these headaches than deal with a program that did not do its job as well.  Another concern was that I wasn't entirely confident that GoodSync was going to do the job right.  I was not sensing a commitment to give me a transparent, thorough synchronization that I could trust without having to check up on.  Still, it wasn't a backup program per se, so possibly this would work out OK.  I would probably just have to use it for a while, to see how things turned out in practice.  (If I do not comment on this program in future posts, it is likely that I stopped using it.)  (This concern about the _gsdata_ folder was reduced, later, when I was able to synchronize whole drives rather than individual folders.  At that point, the only _gsdata_ folders were in the drive root, i.e., at D:\ and E:\.)

I decided to continue.  There wasn't much else to comment on in Tools > Program Options at this point, except to say that they did have a global Filters section there, so they surely could have allowed for folder filtering.  Over in Job > Options, I noticed that they had an option to save one or more versions of deleted files for as long as the user might wish, or until disk space was gone.  There was another, job-specific Filters option here.  I didn't want exceptions popping up in my Beyond Compare double-check, so I didn't set filters; Beyond Compare would notify me about everything.  (Somewhere along the way, I noticed that the contents of the help or explanation window at the bottom of these options dialogs would change as I moused over individual settings.  This was very practical and helpful.)

Moving on, I was now back at the Job > Options > Auto section.  The option of running a synchronization every ___ minutes seemed sufficient, so I didn't go with any of the other scheduling options.  This D:\Miscellany folder rarely changed.  I could probably set it to sync every couple of weeks, and just use GoodSync's manual Sync button if I happened to think of it after I did make a change in that folder.  There was an option not to do a sync if more than a specified percentage of files changed.  This, they felt, could be a sign that something was wrong and that manual intervention was appropriate.  I decided to leave that at their default 10% setting, to see if that worked for me.  Finally, there was an option to "Auto Clear the tree after Sync or Analyze with no changes."  I had no idea what this was about, probably because I didn't know what tree they were referring to, and the help file did not provide an explanation.  (I have pointed GoodSync to this post.  Hopefully they will read it and address some of these concerns.  If so, perhaps they will even post a comment to say so.)

Job > Options also offered Scripts and Advanced sections.  In the Advanced section, I checked some things they had not checked, for added file integrity.  In a case of "ask and it shall be given," I saw that they did have a job-specific "No _gsdata_ folder" option.  The context-specific help note was too big for the little window at the bottom of the dialog, but I found it repeated in the manual.  It said this:

No _gsdata_ folder (un-checked by default)

If checked then do not create _GSDATA_ folder in the Right (remote) sync folder of the job, instead create _GSDATA_ folder in the GoodSync profile folder on this computer.
If not checked then create _GSDATA_ folder in both sync folders of this job.
Still use _gsdata_ folder on the left side of the job.
Use this option only for folders that are not sync centers (all jobs only read from this folder or only write to this folder).
If _gsdata_ folder is disabled then these options are disabled too: Save Previous/Past Version.
This was not entirely clear.  It seemed to be saying, for one thing, that if I did opt to save deleted copies of a file, those copies would be saved in _gsdata_.  Possibly this was the reason for putting the _gsdata_ folders within the individual folders rather than in a separate cache folder:  copying or moving them from their folder of origin to a cache folder could take a lot of time, in the case of many or large files.  It still seemed like something that users should be able to choose, though.  This message also seemed to say that, if I did opt out of _gsdata_ folders, my drive C would swell as I larded it down with data files from synchronized folders on other drives.  Even after opting out, the program would apparently create a _gsdata_ folder in computer A nonetheless.  I didn't know what a sync center was, it wasn't explained in the manual, and the parenthetical explanation here wasn't clear.  I decided not to opt out of _gsdata_.  Finally, I changed their folder and file links options to "as is."

Whew.  OK.  I had set the options for the D-Miscellany job.  Now I had about ten more to folder jobs to set up.  That was going to be a pain, and the odds that I would get all the settings right without overlooking anything were just about nil.  I would have to adjust timing for different folders, but otherwise most of their settings, it seemed, should be the same.  I wished GoodSync had an option, like Beyond Compare, to set default settings for all jobs, so that I wouldn't have to do the same things for each job individually.  Then I noticed an option to create a job template.  Just what the doctor ordered.  I wanted to save D-Miscellany as a template, where all I would have to do would be to change the folders being synchronized.  To do this, I went to Job > Save as template.  The location was C:\Users\Administrator\AppData\Roaming\GoodSync\Templates.  I assumed I would have to make a manual copy of the resulting Default Template.gst file to my D:\Installation\Saved Settings folder if I wanted it to survive a Windows reinstallation.  To use the template, I went to Job > New > From template.  I soon found I could just click the yellow plus symbol at the left end of the tab bar to invoke this dialog.  I was able to drag tabs into a preferred order, which was a nice touch.  Specifying the folders to be synchronized took time:  the computer apparently had to re-connect with the network each time I clicked on a Browse button.  Each recognition step, two per job, took about 10 seconds on the left side and about 20-30 seconds on the right.  Not much in the grand scheme of life, but a drag when you're creating a dozen or more jobs.  This made me wish I had been able to just right-click on a static (refreshable) list of folders to create a job.  Then I went back into each job and changed how frequently it would do its auto-sync.  I decided to try again with the auto sync "on file change," reasoning that surely that part of the program must be functional, if not instantaneous.  What I found was that it would check for file changes frequently -- every minute, from the looks of it -- so this was not the way to minimize system resource usage.  So I changed them all back to a specified number of minutes after all.  When I was done with this, I went to Tools > Export Job List and saved the .tix file to D, again to survive a Win7 reinstallation.

While I was doing all that, of course, those newly created jobs, modeled on D-Miscellany, were all springing into action to synchronize computer A with computer B.  I had done a few file moves on computer A since using Beyond Compare to synchronize computer A with an external drive.  So in a few moments I would go into Beyond Compare and compare computer A against that external drive again.

But first, something weird was happening.  I had finished setting up the folder comparisons on the folders in drive D, but had neglected a couple of folders I had on drive E.  When I began to set those up, I discovered that GoodSync would, in fact, allow me to do a comparison of the entire drive, rather than having to designate individual folders.  This was what I wanted, but it was what they had said they couldn't do and what I had not actually been able to do on drive D.  I went back and reviewed the FAQ page statement cited above.  Well, it said that I could not back up all of drive C.  It didn't say anything about drives D, E, ... So had I made a lot of extra work for myself -- was there actually a way of doing all of drive D at once, some way that I had just overlooked?

At this point, I discovered that GoodSync did have a right-click option to exclude the Recycle Bin and other items.  Unfortunately, when I used that option, it ignored me.  There continued to be a yellow exclamation mark on the tab for that drive, and I continued to get error messages indicating that access to the recycle bin was denied.  A yellow banner said, "Sync has finished with Errors or Unresolved Conflicts," and the log said, "Click Errors button to see sync errors."  But there was no Errors button.  I stopped GoodSync during a sync, so as to hunt for the log, but that made the program crash.  It sent off a crash report to the GoodSync people.

Alright, then, this was a good time to do that comparison with Beyond Compare.  I was not pleased with the results.  Changes that I had made on computer A, after synchronizing via external drive with computer B, had been undone.  That is, computer A had been restored to the earlier condition represented by computer B.  The specific problem seemed to be that files I had deleted on computer A had been restored to computer A from computer B, instead of being deleted from computer B.  Those may have been the only sync problems.  I restarted GoodSync and checked the file deletion settings.  In Job > Options > General, I had already checked "Propagate Deletions," which should have taken care of it.  In Job > Options > Auto, in response to "Automatically resolve conflicts," I had designated "Do Not Copy" instead of "Left to Right," "Right to Left," or "Newer File Wins."  I changed this to "Newer File Wins," which was my preference, now that I thought of it, but this did not seem to explain the problem.  I had not checked "Auto Clear the tree after Sync," and was irritated that the list of changes would disappear from the screen before I had a chance to look at them (if, that is, it ran another sync before I looked at the program again).  Oddly, it seemed that computer B, where I was *not* running GoodSync, wound up with the correct files.

In short, my impression at that point was that there were some bugs in the program.  At this point, I was wishing that Beyond Compare offered a sync program.  But after that first scare, GoodSync seemed to behave more as I had expected.  Hopefully this did not mean that it finally won and was able to delete the file versions that it did not like, that I happened to prefer. 

I was not sure exactly what happened in that first encounter.  It may somehow have been my fault.  During the next 48 hours, GoodSync won my confidence.  And then I used it to compare an existing partition with a new one, where I had added just a few programs on the new one via USB jump drive, and a similar thing happened again:  the Analyze screen indicated that it was prepared to wipe out the existing partition and replace it with the new one.  Then I realized that, of course, it should:  the new partition had been changed more recently.  It didn't go through with it, though, because it detected that more than 10% of the partition was going to change.  So this was more like a warning to me, to not treat GoodSync as a magic wand that would automatically make everything right.

After several days of usage, with careful comparisons of all changes against an external backup using Beyond Compare, GoodSync seemed to be a solid, reliable way to keep two computers synchronized.