Ubuntu 10.04: "rsync: failed to set permissions on [folder]: Operation not permitted" -- and Beyond Compare

Using rsync to automate backups on Ubuntu 10.04 (Lucid Lynx), I got a million iterations of this message:  "rsync: failed to set permissions on [target drive & folder]: Operation not permitted."

I had previously had a somewhat similar problem with an external USB drive.  The solution there had been to reformat the drive.  I didn't want to have to do that again if it wasn't necessary.  I ran a search and, following what i understood from one post, I set out to change ownership on the target folder so as to match the ownership of the source folder.  To do that, I went into Ubuntu's Terminal and typed "sudo nautilus," and then navigated Nautilus to File System/media/[target drive].  (It could instead have been in File System/mnt/.)  I right-clicked on that and got Properties > Permissions.  The owner was root.  When I tried to change it, it changed back.

I exited Nautilus and followed instructions in Ubuntu Community Documentation, combined with advice from another post and the Ubuntu manual page for chown.  I typed "sudo chown -R ray /media/[target drive]."  That ran for five or ten minutes, as the chown (change ownership) command recursed (i.e., -R) all of the subdirectories on the target drive.  But that, by itself, did not fix the problem; indeed, it didn't seem to change anything.  Root was still the owner of the target drive.  Then again, the properties of the target allowed others to create and delete files as well, so why was rsync complaining?

There appeared to be another way.  In Terminal, I typed "sudo gedit /etc/fstab."  I already had a line for the target drive, which I'll call simply TARGET.  That line was as follows:

UUID=[UUID for the drive] /media/TARGET ntfs-3g defaults,umask=000 0 0

The Ubuntu documentation I was now looking at said, "Permissions are set at the time of mounting the partition with umask, dmask, and fmask and can not be changed with commands such as chown or chmod."  So possibly that's why my chown command wasn't doing anything.  Following the advice, I changed it to dmask=027,fmask=137, since this was going to be just a backup drive anyway.  I also saw that "defaults" meant these options:  rw,suid,dev,exec,auto,nouser,async.  All of those were fine, as far as I knew, except for this information from the documentation:

user - Permit any user to mount the filesystem. This automatically implies noexec, nosuid,nodev unless overridden.

nouser - Only permit root to mount the filesystem. This is also a default setting.

I wanted user, not nouser.  I decided it might be easier to spot this sort of thing, in future visits to fstab, if I just changed all occurrences of "defaults" to make these options explicit.  Besides, in light of the statements just quoted, it seemed that I would have to override several defaults anyway.  So the resulting fstab line looked like this:

UUID=[UUID for the drive] /media/TARGET ntfs-3g rw,suid,dev,exec,auto,user,async,dmask=027,fmask=137 0 0

I saved fstab and, in Terminal, typed "sudo mount -a" to mount the partitions according to these new instructions.  Then I ran the rsync command again.  Sadly, these changes made no visible difference.  I still got a million error messages.  So, OK.  I had been overlooking the first three error lines in the log output of my rsync command.  The second and third lines followed the pattern of this first line, but were for subdirectories:

rsync: failed to set times on /media/TARGET/.": Operation not permitted (1)

I hadn't previously investigated the "failed to set times" error, so now I did.  What I got out of the forums I came across was that possibly I had a syntax error in my rsync command.  Here's how it went:

rsync -qhlEtrip --progress --delete-after --ignore-errors --force --exclude=/.Trash-0 --exclude=/.Trash-1000/ --exclude=/lost+found/ /media/SOURCE /media/TARGET 2> /media/SOURCE/Backuplog.log

This was identical to the perfectly working rsync command that I was occasionally using to back up SOURCE to a different partition, with one exception:  in the one shown above, I didn't have a slash after Trash-0.  So I changed that to --exclude=/.Trash-0/ and re-ran it.  That didn't solve it.  I then noticed (I was using two desktops) that I had gotten a message telling me that the system was not able to mount the partition:  "Device or resource busy."  So possibly that fstab step still needed to be tested.  I rebooted the system.  None of the the parittions that I had adjusted in fstab were mounted.  I went back, deleted the dmask and fmask parts, and returned it to umask=000, and rebooted.  Now the partitions were all mounted.  I ran the rsync command again.  The problem had not changed.

I found an informative thread in which bscbrit suggested trying to use cp to copy a single file from the source partition to the target partition.  If it worked, s/he said, the problem was not with permissions; it was just with trying to use rsync.  So in SOURCE I copied a file, renamed it "x.txt," and typed "cp /media/SOURCE/x.txt /media/TARGET," and it worked:  there was a copy of x.txt on TARGET.  It sounded like there could be a kind of mismatch between the ext3 source drive and the ntfs target.  I also came across an informative post by djgrandmarquis that said rsync could have problems with Windows file systems.  That was in response to a question about very slow backup times, which  I had also experienced.  I needed Windows file systems because I was backing up to an offsite drive that I sometimes shared with a Windows system.

I had previously investigated Beyond Compare (BC) as another way of synchronizing drives.  BC had drawn  a lot of praise.  Since my previous writeup, I had bought a copy and had been using it for several months on Ubuntu, WinXP, and Linux.  I had subsequently found a comparison of many such programs; but since I had already bought BC and was using it with good results, I didn't explore that comparison.  I could verify that BC was fast and trouble-free when compared to rsync, especially when going between ext3 and ntfs drives.  I also liked the visible information about what was being changed and backed up.  So I decided to stop using rsync and start relying only on BC for my backups.

There were a few differences between the Windows and Linux versions of BC; but in an impressively thorough reply to my post, Craig from BC said that the portable version of BC ran well on Wine in Ubuntu.  I had already installed Wine, and I had also already been using the portable version of BC3; I preferred portable versions to save time during reinstallations, as I was able to save them elsewhere than drive C and therefore they weren't wiped out during a Windows reinstall.  So to try this out, I copied over the BC3 portable installation and tried using a funky procedure involving PortableApps to try the Windows version.  But now it seemed I had not properly understood or used that procedure:  PortableApps now appeared to be looking, not for .exe files, but for ".paf exe" files.  Meanwhile, it looked like Beyond Compare 3 would run fairly well under Wine, so I gave it a try.  Following the steps described in that previous post, in Terminal I navigated to the folder where I had the original BC installer, typed "ls" to get the exact filename, and typed "wine BCompare-3.1.10.11626.exe" to install that file.  Terminal reported some "fixme" and "err" messages; nonetheless, the Beyond Compare 3 Setup Wizard ran.  I had a licensed version, so I entered the license key, and followed the instruction to restart.  It worked, but now I came to the new realization that Windows identifies drives by letter (C: etc.), while Ubuntu identifies them by name (/media/DATA, etc.).  So of course a Windows version of Beyond Compare, looking for Windows-type folder paths, running in Ubuntu, is going to find only a limited number of Ubuntu locations.  I didn't pursue that problem.

This left at least one issue unresolved.  While I was OK with doing manual comparisons for now, I was still not at the point of being able to write scripts to automate BC backup processes.  I started another post on that.

VMware Workstation 7 Installation: Permission Denied

I had been using VMware Workstation on Ubuntu for several years.  VMware put out a new release of Workstation 7, so I downloaded it.  I tried using the familiar steps I had used to upgrade VMware.  It had been just a matter of entering these two commands:

sudo chmod +x VMware-Workstation-Full-7.1.0-261024.i386.bundle 

sudo ./VMware-Workstation-Full-7.1.0-261024.i386.bundle

This time, though, it didn't work.  The first one went OK; but after the second one, I got this:

bash: ./VMware-Workstation-Full-7.1.0-261024.i386.bundle: Permission denied

Malcolm Lewis explained that I could use ./ to execute the file after running chmod, but I could skip the chmod command and just use one command:  sh [filename].  So I tried that.  And, you know, that worked.  Or at least it went further than before.  So apparently my chmod +x command was not sufficient to make the thing executable.  Possibly I needed to use chmod 777 or something.

I say it went further because I didn't type "sudo sh"; I just typed "sh" followed by the downloaded VMware bundle file.  So after it said, "Extracting VMware Installer...done," it stopped with an error:  "root access is required for the operations you have chosen."  I did it again with sudo, and this time it ran.  End of problem!

Basic Ubuntu (Bash) Shell Scripts

In Ubuntu 9.04, I wanted to write a script that would execute an rsync command, so that I could put a brief reference to the script into my crontab file, instead of putting the whole long rsync command there.

For a brief, tiny moment, I was almost tempted to consider learning the GAMBAS (Gambas Almost Means BASIC) programming language, just because (pre-Visual) BASIC was the only programming language I ever learned.  Instead, I moved toward basic instructions on writing a shell script.  Here's what I wrote:

#!/bin/bash
# This is backup-hour.sh
# It backs up CURRENT to CURRBACKUP every few hours
rsync -qhlEtrip --progress --delete-after --ignore-errors --force --exclude=/.Trash-1000/ --exclude=/lost+found/ /media/CURRENT/ /media/CURRBACKUP

As the instructions said, the first line was essential to tell the computer to use BASH to interpret the following lines.  The next two lines were comments, and the final line (wrapping over onto multiple lines here) was exactly the rsync line I'd been using to do the backup.  In other words, learning how to write the command was almost all I needed to write the script.

The next step was to save it somewhere.  I had previously heard, and the instructions said, that the common place to put it is in your bin folder.  I went with that, but made a note that I would need to be sure to back up my bin folder, because I didn't want to go to the trouble of writing all these scripts and then see them vanish.

The usual location for the user's bin folder is at /home/[username]/bin.  In my case, that's /home/ray/bin.  Getting there in Nautilus (i.e., Ubuntu's File Browser, also started by typing "nautilus" in Terminal) can be confusing:  you can get there via the Home Folder option (assuming you're showing Tree rather than Places or something else (or nothing) at the left side of the file browser), or you can go to File System/home/[username]/bin.  Same thing, either way.  So I saved the script (above) in my bin folder as backup-hour.sh.  That turned the comment lines (beginning with #) blue in gedit.

Next, the instructions said, I needed to set permissions.  This was a confusing aspect of Ubuntu.  The documentation seemed to say that there were ten permissions that a person could set.  These were represented by ten hyphens or minus signs:  ----------.  I couldn't tell what the first one was for, but the remaining nine were divided into three sets of three.  The first three belonged to the owner, the second three to the group, and the last three to "other."  Within each set of three, the first one was for read, the second was for write, and the third was for execute (i.e., run it as a program).  So if you set the owner's permissions to read (r), write (w), and execute (x), your ten hyphens would now change to this:  -rwx------.  If you set all three parties (i.e., owner, group, and other) the same, they would look like this:  -rwxrwxrwx.

You could set the permissions using the chmod command.  I found an Ubuntu manual page on chmod.  It was not really that complicated, but it looked like it was going to require a time investment to make sure I had it right, and at this point I was getting impatient.  The basic idea seemed to be that you could use chmod to enter values of 4 (for read permission), 2 (for write permission), and/or 1 (for execute permission).  So, for example, you could type "chmod 755" and that would give a value of 7 to the first of the three users mentioned above (i.e., the owner), a value of 5 to the second of the three (i.e., the group), and a value of five to the third of the three (i.e., other).  The 7 would mean that you gave read + write + execute (4 + 2 + 1) permissions to the owner, whereas the 5 would mean that you gave only read + execute (4 + 1) permissions to the rest.  Since that's what the instructions suggested, I went with that.  To set the script with those permissions, I typed "chmod 755 backup-hour.sh."

I wasn't too sure of who the owner was (i.e., me or root), not to mention the group or other.  I mean, this was for my home computer.  Not a lot of people milling around, waiting to take my hard drive for a spin.  These kinds of options seemed to be set up for networked computers, where the "accounting" department might be a group that would own a file.  I found what looked like a good tutorial on file owners, and another interesting (yawn!) page about permissions, but fortunately I did not have time to work through them.

When I typed "chmod 755 backup-hour.sh," I got "cannot access 'backup-hour.sh': No such file or directory."  One solution was to use a change directory (cd) command to get the Terminal prompt into the bin subfolder, so it would see what I was looking for.  But since I planned to put more scripts into that folder, and anyway since I wanted cron or other programs to know right away what I was talking about when I referred to something like backup-hour.sh, I decided to figure out how to put the bin folder in my "path."  The path is the list of folders where the operating system looks for guidance on what a command means.  To change my path so that the system would always know to look in bin, they said I needed to find and edit my .bash_profile file.  Unfortunately, they didn't say where it was.  It wasn't easy to find.  I ran searches in Nautilus (both user and root), but while those were grinding away, I found that I could just type "locate .bash_profile."  That turned up nothing, but very quickly.  Then I got some advice that, if it didn't exist, I could create it by using "touch ~/.bash_profile."  So I did that, and then tried again with "chmod 755 backup-hour.sh."  Still no joy.  Ah, but maybe that was because I hadn't rebooted; backup-hour.sh would run only on startup.  OK, so I used the other approach after all:  I changed directory to the bin folder and tried again.  Now I got "Permission denied."  What if I gave everybody full permissions with chmod 777?  I tried that instead of chmod 755.  That seemed to do it.  The hard drive was doing its thing now.

I wanted to see what was going on, so I decided to create a log file.  I wanted it to store only the error messages, not to list the thousands of files that backup-hour.sh was backing up successfully, so I put this on the end of (that is, on the same command line as) my rsync command in backup-hour.sh (above):

2> /media/CURRENT/backup-hour.log

The log filename thus matched the script filename.  I put "backup" first so that I could see all of my backup scripts in the same part of the folder's directory listing, and then I set up a backup-day.sh script along the same lines.  New problem:  these backup scripts would generate empty log files if there were no errors, and I didn't want to have to delete them manually.  So I found a forum post with advice on how to delete them automatically, using the "find" command.  In my version, it looked like this:

find /media/CURRENT/ -name "*.log" -size 0c -exec rm -f {} \;

I put that at the end of the backup-day.sh script, and it seemed to work.  It said, basically, look in the CURRENT folder for files whose name ends with .log and have zero bytes; and if you find any files like that, execute the "remove" command without asking for permission.  I didn't know what that ending punctuation is about, but that's what the advisor suggested.

In my backup-day.sh (not backup-hour.sh) script, I included the instructions for updating my USB jump drive (above).  I also included a set of commands to save my e-mail (I was using Thunderbird in Ubuntu) as a .tar compressed file. Actually, as seven .tar files, one for each day of the week.  That part of backup-day.sh looked like this:

# Assign Thunderbird mail & profile to be backed up
backup_files="/home/ray/.mozilla-thunderbird"
dest="/media/BACKROOM/Backups/Tbird"
# Create archive filename
day=$(date +%A)
hostname=$(hostname -s)
archive_file="$hostname-$day.tgz"
# Back up to a tgz file
tar zcvf $dest/$archive_file $backup_files 2>> /media/CURRENT/A-INCOMING/T-bird-Backup.log

So these seemed to be the basic kinds of tools I needed to set up rsync scripts and crontab entries.