Ubuntu 10.04: "rsync: failed to set permissions on [folder]: Operation not permitted" -- and Beyond Compare

Using rsync to automate backups on Ubuntu 10.04 (Lucid Lynx), I got a million iterations of this message:  "rsync: failed to set permissions on [target drive & folder]: Operation not permitted."

I had previously had a somewhat similar problem with an external USB drive.  The solution there had been to reformat the drive.  I didn't want to have to do that again if it wasn't necessary.  I ran a search and, following what i understood from one post, I set out to change ownership on the target folder so as to match the ownership of the source folder.  To do that, I went into Ubuntu's Terminal and typed "sudo nautilus," and then navigated Nautilus to File System/media/[target drive].  (It could instead have been in File System/mnt/.)  I right-clicked on that and got Properties > Permissions.  The owner was root.  When I tried to change it, it changed back.

I exited Nautilus and followed instructions in Ubuntu Community Documentation, combined with advice from another post and the Ubuntu manual page for chown.  I typed "sudo chown -R ray /media/[target drive]."  That ran for five or ten minutes, as the chown (change ownership) command recursed (i.e., -R) all of the subdirectories on the target drive.  But that, by itself, did not fix the problem; indeed, it didn't seem to change anything.  Root was still the owner of the target drive.  Then again, the properties of the target allowed others to create and delete files as well, so why was rsync complaining?

There appeared to be another way.  In Terminal, I typed "sudo gedit /etc/fstab."  I already had a line for the target drive, which I'll call simply TARGET.  That line was as follows:

UUID=[UUID for the drive] /media/TARGET ntfs-3g defaults,umask=000 0 0

The Ubuntu documentation I was now looking at said, "Permissions are set at the time of mounting the partition with umask, dmask, and fmask and can not be changed with commands such as chown or chmod."  So possibly that's why my chown command wasn't doing anything.  Following the advice, I changed it to dmask=027,fmask=137, since this was going to be just a backup drive anyway.  I also saw that "defaults" meant these options:  rw,suid,dev,exec,auto,nouser,async.  All of those were fine, as far as I knew, except for this information from the documentation:

user - Permit any user to mount the filesystem. This automatically implies noexec, nosuid,nodev unless overridden.

nouser - Only permit root to mount the filesystem. This is also a default setting.

I wanted user, not nouser.  I decided it might be easier to spot this sort of thing, in future visits to fstab, if I just changed all occurrences of "defaults" to make these options explicit.  Besides, in light of the statements just quoted, it seemed that I would have to override several defaults anyway.  So the resulting fstab line looked like this:

UUID=[UUID for the drive] /media/TARGET ntfs-3g rw,suid,dev,exec,auto,user,async,dmask=027,fmask=137 0 0

I saved fstab and, in Terminal, typed "sudo mount -a" to mount the partitions according to these new instructions.  Then I ran the rsync command again.  Sadly, these changes made no visible difference.  I still got a million error messages.  So, OK.  I had been overlooking the first three error lines in the log output of my rsync command.  The second and third lines followed the pattern of this first line, but were for subdirectories:

rsync: failed to set times on /media/TARGET/.": Operation not permitted (1)

I hadn't previously investigated the "failed to set times" error, so now I did.  What I got out of the forums I came across was that possibly I had a syntax error in my rsync command.  Here's how it went:

rsync -qhlEtrip --progress --delete-after --ignore-errors --force --exclude=/.Trash-0 --exclude=/.Trash-1000/ --exclude=/lost+found/ /media/SOURCE /media/TARGET 2> /media/SOURCE/Backuplog.log

This was identical to the perfectly working rsync command that I was occasionally using to back up SOURCE to a different partition, with one exception:  in the one shown above, I didn't have a slash after Trash-0.  So I changed that to --exclude=/.Trash-0/ and re-ran it.  That didn't solve it.  I then noticed (I was using two desktops) that I had gotten a message telling me that the system was not able to mount the partition:  "Device or resource busy."  So possibly that fstab step still needed to be tested.  I rebooted the system.  None of the the parittions that I had adjusted in fstab were mounted.  I went back, deleted the dmask and fmask parts, and returned it to umask=000, and rebooted.  Now the partitions were all mounted.  I ran the rsync command again.  The problem had not changed.

I found an informative thread in which bscbrit suggested trying to use cp to copy a single file from the source partition to the target partition.  If it worked, s/he said, the problem was not with permissions; it was just with trying to use rsync.  So in SOURCE I copied a file, renamed it "x.txt," and typed "cp /media/SOURCE/x.txt /media/TARGET," and it worked:  there was a copy of x.txt on TARGET.  It sounded like there could be a kind of mismatch between the ext3 source drive and the ntfs target.  I also came across an informative post by djgrandmarquis that said rsync could have problems with Windows file systems.  That was in response to a question about very slow backup times, which  I had also experienced.  I needed Windows file systems because I was backing up to an offsite drive that I sometimes shared with a Windows system.

I had previously investigated Beyond Compare (BC) as another way of synchronizing drives.  BC had drawn  a lot of praise.  Since my previous writeup, I had bought a copy and had been using it for several months on Ubuntu, WinXP, and Linux.  I had subsequently found a comparison of many such programs; but since I had already bought BC and was using it with good results, I didn't explore that comparison.  I could verify that BC was fast and trouble-free when compared to rsync, especially when going between ext3 and ntfs drives.  I also liked the visible information about what was being changed and backed up.  So I decided to stop using rsync and start relying only on BC for my backups.

There were a few differences between the Windows and Linux versions of BC; but in an impressively thorough reply to my post, Craig from BC said that the portable version of BC ran well on Wine in Ubuntu.  I had already installed Wine, and I had also already been using the portable version of BC3; I preferred portable versions to save time during reinstallations, as I was able to save them elsewhere than drive C and therefore they weren't wiped out during a Windows reinstall.  So to try this out, I copied over the BC3 portable installation and tried using a funky procedure involving PortableApps to try the Windows version.  But now it seemed I had not properly understood or used that procedure:  PortableApps now appeared to be looking, not for .exe files, but for ".paf exe" files.  Meanwhile, it looked like Beyond Compare 3 would run fairly well under Wine, so I gave it a try.  Following the steps described in that previous post, in Terminal I navigated to the folder where I had the original BC installer, typed "ls" to get the exact filename, and typed "wine BCompare-3.1.10.11626.exe" to install that file.  Terminal reported some "fixme" and "err" messages; nonetheless, the Beyond Compare 3 Setup Wizard ran.  I had a licensed version, so I entered the license key, and followed the instruction to restart.  It worked, but now I came to the new realization that Windows identifies drives by letter (C: etc.), while Ubuntu identifies them by name (/media/DATA, etc.).  So of course a Windows version of Beyond Compare, looking for Windows-type folder paths, running in Ubuntu, is going to find only a limited number of Ubuntu locations.  I didn't pursue that problem.

This left at least one issue unresolved.  While I was OK with doing manual comparisons for now, I was still not at the point of being able to write scripts to automate BC backup processes.  I started another post on that.