Ubuntu: Backup with Rsync

In a previous post, I got as far as concluding that rsync was the tool of choice for backing up my computer in Ubuntu 9.04. I didn't pursue it because I was short on time and patience for writing scripts at that point. But eventually the need for a regular backup system became acute. So this post logs the steps I took to make rsync and cron work for me.

First, here's what I wrote previously:

As an alternative to rdiff-backup, what people had actually mentioned more frequently was rsync. It did not have the incremental backup features of rdiff-backup, to my knowledge, but it seemed to be an established tool for backup purposes. So for now, at least, I thought I might try that instead. Once again, I did a Google search and got a package details page with no apparent link to any help files. Eventually I found what looked like the official rsync webpage and, after looking at their FAQs and some other pages, landed on their Examples page. It was intimidating.

This time, I went to their Documentation page. This gave me links to, among other things, Michael Holve's rsync tutorial. The tutorial said, "You must set up one machine or another of a pair to be an "rsync server" by running rsync in a daemon mode." I was curious, so I did a Google search for "what is daemon mode" and I got back, would you believe, exactly one page. One webpage in the entire known planet answered the question, "What is daemon mode?" Except it didn't really answer it. It just said, "It makes wget put standard output into a log file and not bug you while downloading." Accepting that as the best available answer (and ten points to the answerer!), I typed "rsync --daemon" in Ubuntu's Terminal and proceeded to the next step, "Setting Up a Server." After reading it, I decided it didn't seem to apply to me. It was for people who wanted to back up files between computers. I just wanted to back up to another drive.

So I went on to the tutorial's "Using Rsync Itself" section. Since I wasn't sure what daemon mode did, or if it was necessary, I killed that Terminal session and started another. I didn't know if that would shut off daemon mode, or if doing so was what I should do. I read the section and then checked another source of documentation, the rsync man page ("man" being short for "manual"). The man page would ordinarily be output in response to a Terminal command, but someone had put it here in html form, so that's what I used. It reminded me, first, to check Ubuntu's System > Administration > Synaptic Package Manager to make sure I had rsync already installed, here on my secondary computer. I searched Synaptic for rsync and got back a couple dozen listed programs; rsync was among them and was shown as being installed. I looked partway into the man page and got an answer to one question I had from the tutorial. So here's how I translated what the tutorial was telling me. First, the tutorial listed these lines:

rsync --verbose --progress --stats --compress --rsh=/usr/local/bin/ssh --recursive --times --perms --links --delete \
--exclude "*bak" --exclude "*~" \
/www/* webserver:simple_path_name

The first thing to know was that this all represented a single command line. It was too long to fit on one line, though, so apparently the trailing backslash said, "This line continues on the next line." The command would be typed into a file and saved as a script, not typed directly into Terminal. I didn't know why the first line didn't end with a backslash. I decided I would want to experiment with this in a relatively safe place -- with a junk directory on my secondary computer, perhaps -- to see what it was doing.

So as the tutorial explained it, the first line of this example told rsync how to proceed: verbosely (i.e., with lots of information about what it was doing), showing a progress report, with statistics. The rsh part was for encryption, to be used optionally if you were sending your stuff online to another computer. I wasn't, so I decided to try leaving that off. I also didn't want to compress the output, because that made the process slower and required more attention from the CPU.

The second line of the example, above, told rsync to recurse -- to work through all of my directories and subdirectories under the folder that I would be naming. It also told rsync to preserve file timestamps and file permissions -- so if, for example, a file was readable only by root on the source drive, it would be the same way on the target drive. The -- links command was an instruction to preserve symbolic links -- not sure what that meant -- and the --delete command, as I understood it, would tell rsync to delete anything on the target that wasn't on the source. So you'd have a mirror, and not just an accumulation of backups of files that you have deliberately trimmed out of your file collection.

The third line of the example told rsync not to bother copying some kinds of files. I liked the sound of that at first, but then I decided I would rather be able to do a Properties comparison of source and target and verify that both had exactly the same number of files. So I decided to leave out this line when I used rsync.

The fourth line of the example named the source and target locations. I wasn't going to be using it with a remote source or target, so mine was going to look somewhat different from this.

On that basis, here's what I assembled as a test version of the rsync example from above:

rsync --verbose --progress --stats \
--recursive --times --perms --links --delete \
/media/DATA/Source /media/DATA/Target

I created a Source subfolder in my DATA folder and put a TestFile.txt file into it. I also created a Target subfolder in DATA. Then I copied those three rsync lines into a file in Ubuntu's Text Editor (gedit) and saved it to Desktop as TestRun. To make it executable, I went into Terminal, typed "cd /home/ray/Desktop" and then "chmod +x TestRun" and then double-clicked on TestRun and said Run. And, you know, it worked. Just like that. Not exactly as intended -- I had not only TestFile.txt but also the whole Source folder underneath my Target folder -- but, yeah, there it was. I deleted the Target folder and ran it again and, sure enough, it created the Target folder and then inserted a copy of the Source folder into it. Excellent!

Now it was time to try something a little bolder. I wanted to see how it worked if I tried to copy the whole DATA folder to an external drive. This part was a little confusing. The external drive seemed to have two different names. If I looked in /media, its name was simply "disk." But if I hit the Computer icon in File Browser, it came up as "193.8 GB Media." I decided the latter sounded more specific, so I would try that first. So now the third line of my TestRun file read like this:

/media/DATA "/media/193.8 GB Media"

I used quotation marks because there were spaces in the name. I saved TestRun and double-clicked it again on the Desktop. It didn't seem to do anything. I realized that I had probably made a mistake in that line, and tried again like this:

/media/DATA "/193.8 GB Media"

That didn't do it either, so I tried again, without the leading slash:

/media/DATA "193.8 GB Media"

That still didn't work, so I tried the other approach:

/media/DATA /media/disk

Still nothing. I went into System > Administration > Partition Editor (GPartEd), wiped out the target partition, and recreated it as a FAT32 partition. Now the drive was totally invisible to Ubuntu. I went back into GPartEd and reformatted it as an ext3 partition. Then I realized: it was an IDE drive, so apparently it would not be recognized until I rebooted. I decided to reboot into Windows (I had a dual-boot system) and format it as NTFS. I named it 186GB (which seemed to be the net amount of space available in NTFS format) and rebooted into Ubuntu. I revised TestRun's last line again:

/media/DATA /media/186GB

and ran it again. This time, it seemed to be working -- the external 186GB drive was making noise -- but I wondered why I wasn't getting a verbose indication of what was going on. I guessed that, if I wanted the verbose information, I would have to execute rsync on the command line, not in an executable script. While I was rooting around for an answer to that question, I was reminded that I could also use the shorthand versions of these commands. So instead of typing --verbose into the script, I could just type -v and whatever other letters I needed. In this approach, the final contents of TestRun, which were as follows:

rsync --verbose --progress --stats \
--recursive --times --perms --links --delete \
/media/DATA /media/186GB

could instead be expressed like this, if I understood the man page's Options Summary section correctly:

rsync -vshlEPtrip --del --delete-excluded --force \
--exclude RECYCLER \
--exclude "System Volume Information" \
/media/DATA /media/186GB

In that version, I added a couple other options that seemed appropriate, and also told rsync to exclude (i.e., don't copy) those extra folders that Windows XP seemed to put on every drive. I revised TestRun along these lines and, when the external drive settled down and it looked like the foregoing TestRun process had ended, I ran it again. But it didn't seem to make any difference. The extra folders were still there. I had understood that it would delete them. Part of the problem seemed to be that I had not used the syntax correctly. I was supposed to use an equals sign: --exclude=RECYCLER. But another part of the problem was that it was not clear whether the "exclude" command was supposed to work with directories. It didn't seem so. The man page just referred to excluding files. I tried again with equals signs, but still no change. I posted a question on it, but the kind response was unfortunately not able to resolve the issue.

Next, I tried a modified version of TestRun on the primary computer. I went through several revisions and wound up with this version, which seemed to work:

rsync -vchlEtrip --progress --del --ignore-errors --force /media/CURRENT/ "/media/OFFSITE/P4 CURRENT"

The partition being backed up, in this case, was an ext3 partition named CURRENT, and the target to which it was being backed up was a USB external drive named OFFSITE. (Some weeks have passed since I started this post, so there may be some discontinuity in my writing at this point.)

I did not run this command as a script within a file called TestRun that I would start by double-clicking on it, because I discovered that you would only get the detailed output if you entered the rsync command on the command line. I was able to enter all of the foregoing rsync command on one line. I did not need the "exclude" commands because this was not an NTFS drive formatted by Windows. I still had ext3 "lost+found" and Trash folders that got copied over in this way, but they were small, so it was OK.

As I think I may have said before, I got the selected rsync parameters by typing "man rsync" at the command line. It did take some trial and error to get this particular set. The resulting backup, when checked by right-clicking and selecting Properties, seemed to be virtually identical.

When I ran that rsync command, it showed me lots of detail on what it was doing. It concluded with this message:

rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9]

Eventually, however, I did figure out how to do it.  Here is an example of an rsync command that worked for me:

rsync -qhlEtrip --progress --delete-after --ignore-errors --force --exclude=/.Trash-1000/ --exclude=/lost+found/ /media/CURRENT/ /media/CURRBACKUP

This one would back up what Windows sees as drive D (named CURRENT) to a partition that Windows sees as drive G (named CURRBACKUP).  Both partitions had to be mounted in Ubuntu before this would work.  I used a similar command to copy a folder on CURRENT to a USB jump drive named KINGSTON.  That gave me a portable copy of the current state of that folder, ready to take along.

The next thing I needed to do was to back up my blogs.  I started by just wanting to be able to back up a webpage.  I had discovered that all of the posts on a Blogger (i.e., Blogspot) blog like this one could be displayed in a single webpage, at least if you had less than 1,000 posts.  To do that, you just needed to go to this URL:  http://blogname.blogspot.com/search?max-results=1000.  I wasn't sure what would happen if you entered a larger number than 1000.  So now that I had that webpage, I wanted to know how to save a copy of it automatically.  Strangely, at this point, Google searches for any of these sets of terms

ubuntu "back up a webpage"
rsync "back up a webpage"
rsync "copy a webpage"

produced zero hits.  Eventually, it started to look like this was because I was barking up the wrong tree.  As described in a separate post, it seemed that what I wanted for this purpose might be wget, not rsync.

Recognizing Partitions Automatically in Ubuntu 9.04

In Ubuntu 9.04 (Jaunty Jackalope), disk partitions normally remain unmounted until you mount them manually. One way to do this is to start Ubuntu's File Browser (also known as Nautilus). You can do this by choosing the Places > Computer menu option. In File Browser, just double-click on the drive you want to mount, and its name will appear under the File System > Media folder there in File Browser. You can also mount drives by typing commands into Ubuntu's command-line window (Applications > Accessories > Terminal). First, type "sudo mkdir /media/[drivename]" to create a mount point. After doing this for all unrecognized drives, type "sudo mount -a".

But instead of doing this sort of thing, I wanted the system to recognize my drives on bootup, so that I would not have to mount them manually each time I rebooted. In my first approach to this problem, I typed "sudo ntfs-config." This approach worked OK on one computer, where I was doing a clean installation -- where, that is, I had formatted the partition before installing 9.04 on it. But on another computer, I installed 9.04 on top of 8.04 without formatting the partition. This was advantageous in that it preserved lots of files and settings from before. The drawback was that I was running into funky problems with ntfs-config.

On this system, I got an error message when I ran ntfs-config. The message was, literally, "Error : An error occured when trying to configure [partition name], please retry. Thanks." (For those who are searching with the correct spelling, let me retype that error message: "Error: An error occurred when trying to configure . . . ." When I tried to examine the partition in GParted (i.e., System > Administration > Partition Editor (see above)), I got this: "Warning: Unable to read the contents of this file system!" The partition had been available a minute earlier, when I had booted into Windows XP, and I had shut down WinXP normally before rebooting into Ubuntu. So I didn't think the partition was truly hosed.

I found a very, very long thread on the subject, but post no. 2015 in that thread said the thread's advice was outdated for purposes of newer versions of Ubuntu. In another thread, someone said they got this error message after installing a newer version without formatting the partition, and solved it by deleting the mount points. I had a fresh backup of the partition in question, so I tried taking that approach. First, I opened File Browser and went to File System > Media > CURRENT. File Browser said that partition was empty. When I right-clicked on it, however, File Browser didn't give me an option of deleting it. Its Properties said it was owned by root, so I logged in as root (see above), typed "cd /media", typed "dir" to make sure CURRENT was there, typed "dir CURRENT" to make sure it was empty, and then typed "rm -r CURRENT" to delete it. I restarted the system, started File Browser, went to Computer (in the Tools bar), double-clicked on CURRENT, and it was there, with all its contents. CURRENT was formatted in ext3 (i.e., Linux). I had another partition that was having the same problem. It was formatted in NTFS (i.e., Windows). But now that I had rebooted, it too was available and looking fine.

I dropped this issue for the time being. Next day, when I rebooted, I found that, predictably, the various partitions were not mounted. In Terminal, I typed "sudo gedit /etc/fstab." My fstab file did not show the partitions I wanted to be mounted automatically. I reviewed one of my old posts on ntfs-3g. It seemed to say that ntfs-config was the solution. I tried running ntfs-config again. This time, it ran. It recognized two NTFS partitions. Now I just had two ext3 partitions to set up for automatic mounting. I went back into fstab and saw the lines that ntfs-config had added. I hadn't thought to save the fstab file from my previous installation; it would have been handy to copy over the relevant lines. From the previous post, I gathered that the trick was to write something like this in fstab:

/dev/sda3 /media/VMS ext3 defaults 0 0

for the VMS partition, and likewise for the other ext3 partition. I double-checked that I had the right device addresses by checking them in System > Administration > Partition Editor (i.e., GParted). I rebooted, but the ext3 partitions were still not automounted, though the NTFS partitions continued to be automounted nicely. I didn't know why the line just shown, for the VMS ext3 partition, did not work in fstab now; according to the old post, it had worked in Ubuntu 8.04. Ubuntu documentation said something about pysdm, so I looked into that. They suggested typing this into Terminal:

sudo apt-get install pysdm

but I saw it was available in Synaptic (i.e., Ubuntu's System > Administration > Synaptic Package Manager menu pick), so I used that instead to install pysdm. I typed "pysdm" at the Terminal prompt, but got an error message indicating that I had to have root privileges, so I tried again with "sudo pysdm." (Also available via System > Administration > Storage Device Manager.) That gave me a Storage Device Manager (SDM) window that listed some, but not all, of my partitions. I noticed that, if I started pysdm while I still had the foregoing /dev/sda3 line for my VMS partition in my fstab file, SDM would not give me an option of configuring the VMS partition; but if I deleted that line from fstab and then started pysdm and clicked on sda3, SDM would give me a "Configure now?" dialog with the message, "/dev/sda3 hasn't been configured. Do you want to configure it now?" I said OK. I told it to set the mount point to File System > Media > VMS. Then I clicked on the Assistant button and saw that the option was checked that said "The file system is mounted at boot time." I also checked the option that said, "Check file system at mount time" on the options dialog's Mounting tab. I didn't change any other options. When I checked OK to leave Assistant, I saw an error message in Terminal: "wrong fs type, bad option, bad superblock on /dev/sda3, missing codepage or helper program, or other error." I went back into Assistant and removed the "Check file system" option I had just checked. This time, when I left Assistant, I got Warning messages that said, "Unknown option: rw" and "nouser is not suitable for user" and "Unknown option: async." I clicked on the "Set defaults" button in SDM and then clicked Apply. This produced no additional Warning messages in Terminal. Trying once more, I just went into Assistant and immediately executed, and this did produce the same Warning messages, plus an additional one that said, "Unknown option: defaults." I tried just emptying out the Options box altogether in SDM. Then I clicked Apply. I did the same with the other ext3 partition that was not yet mounting automatically. I restarted the system.

It occurred to me that I may not have restarted the system previously, but may merely have logged out and logged back in. Ubuntu 9.04 seemed to require two steps to reboot, and it seemed that I had forgotten to take that second step last time around. This occurred to me right around the time when I tried to restart the system and found that it froze with a black screen. Evidently my fstab file was not presently in an acceptable form. I punched the computer's reset button and rebooted with the Ubuntu installation CD, chose the "Try Ubuntu" option, and edited fstab as above. (Note that there's an fstab on the CD too; make sure you're editing the one on the hard drive.) I saw that I had failed to add the preferred mountpoint for one of my ext3 partitions: it said "/media/sdc5" instead of "/media/CURRENT." I fixed that and rebooted without the CD. This time, the system booted OK. But those two ext3 partitions were still not automounted.

In File System > media (i.e., /media), I noticed that there was a mount point for the VMS partition, but it had nothing in it. I typed "dir" to make sure there was nothing there, and then typed "sudo rm -r VMS" to delete it. I also deleted the "sc5" mountpoint mentioned above. In File Browser, I clicked on the Computer button at the top and then double-clicked on each of these two partitions (i.e., VMS and CURRENT) to create mountpoints (i.e., /media folders) for them. I tried rebooting again. They still weren't automounting, and the mount points I had created for them were gone, as I should have expected. Incidentally, at this point I discovered that the custom was apparently to mount fixed hard drives at /mnt, and use /media for removable hard drives.

About this time, I discovered that I had commented out the lines added by SDM (i.e., I put # signs (also called hash marks) in front of each of those lines). I forgot. I had done that to troubleshoot fstab, and then didn't go that route after all, and forgot to remove them. So I removed the hash marks now and rebooted. But that wasn't the solution. I re-ran SDM, went through more or less the steps described above, and rebooted. This time, it worked: the partitions were automounted, and their contents were visible. Not sure exactly what was different this time, but something was. So I guess the conclusion is, if it doesn't work in SDM the first time, tinker around for a while, reboot a couple of times, try SDM again, and reboot.