Ubuntu 10.04: "rsync: failed to set permissions on [folder]: Operation not permitted" -- and Beyond Compare

Using rsync to automate backups on Ubuntu 10.04 (Lucid Lynx), I got a million iterations of this message:  "rsync: failed to set permissions on [target drive & folder]: Operation not permitted."

I had previously had a somewhat similar problem with an external USB drive.  The solution there had been to reformat the drive.  I didn't want to have to do that again if it wasn't necessary.  I ran a search and, following what i understood from one post, I set out to change ownership on the target folder so as to match the ownership of the source folder.  To do that, I went into Ubuntu's Terminal and typed "sudo nautilus," and then navigated Nautilus to File System/media/[target drive].  (It could instead have been in File System/mnt/.)  I right-clicked on that and got Properties > Permissions.  The owner was root.  When I tried to change it, it changed back.

I exited Nautilus and followed instructions in Ubuntu Community Documentation, combined with advice from another post and the Ubuntu manual page for chown.  I typed "sudo chown -R ray /media/[target drive]."  That ran for five or ten minutes, as the chown (change ownership) command recursed (i.e., -R) all of the subdirectories on the target drive.  But that, by itself, did not fix the problem; indeed, it didn't seem to change anything.  Root was still the owner of the target drive.  Then again, the properties of the target allowed others to create and delete files as well, so why was rsync complaining?

There appeared to be another way.  In Terminal, I typed "sudo gedit /etc/fstab."  I already had a line for the target drive, which I'll call simply TARGET.  That line was as follows:

UUID=[UUID for the drive] /media/TARGET ntfs-3g defaults,umask=000 0 0

The Ubuntu documentation I was now looking at said, "Permissions are set at the time of mounting the partition with umask, dmask, and fmask and can not be changed with commands such as chown or chmod."  So possibly that's why my chown command wasn't doing anything.  Following the advice, I changed it to dmask=027,fmask=137, since this was going to be just a backup drive anyway.  I also saw that "defaults" meant these options:  rw,suid,dev,exec,auto,nouser,async.  All of those were fine, as far as I knew, except for this information from the documentation:

user - Permit any user to mount the filesystem. This automatically implies noexec, nosuid,nodev unless overridden.

nouser - Only permit root to mount the filesystem. This is also a default setting.

I wanted user, not nouser.  I decided it might be easier to spot this sort of thing, in future visits to fstab, if I just changed all occurrences of "defaults" to make these options explicit.  Besides, in light of the statements just quoted, it seemed that I would have to override several defaults anyway.  So the resulting fstab line looked like this:

UUID=[UUID for the drive] /media/TARGET ntfs-3g rw,suid,dev,exec,auto,user,async,dmask=027,fmask=137 0 0

I saved fstab and, in Terminal, typed "sudo mount -a" to mount the partitions according to these new instructions.  Then I ran the rsync command again.  Sadly, these changes made no visible difference.  I still got a million error messages.  So, OK.  I had been overlooking the first three error lines in the log output of my rsync command.  The second and third lines followed the pattern of this first line, but were for subdirectories:

rsync: failed to set times on /media/TARGET/.": Operation not permitted (1)

I hadn't previously investigated the "failed to set times" error, so now I did.  What I got out of the forums I came across was that possibly I had a syntax error in my rsync command.  Here's how it went:

rsync -qhlEtrip --progress --delete-after --ignore-errors --force --exclude=/.Trash-0 --exclude=/.Trash-1000/ --exclude=/lost+found/ /media/SOURCE /media/TARGET 2> /media/SOURCE/Backuplog.log

This was identical to the perfectly working rsync command that I was occasionally using to back up SOURCE to a different partition, with one exception:  in the one shown above, I didn't have a slash after Trash-0.  So I changed that to --exclude=/.Trash-0/ and re-ran it.  That didn't solve it.  I then noticed (I was using two desktops) that I had gotten a message telling me that the system was not able to mount the partition:  "Device or resource busy."  So possibly that fstab step still needed to be tested.  I rebooted the system.  None of the the parittions that I had adjusted in fstab were mounted.  I went back, deleted the dmask and fmask parts, and returned it to umask=000, and rebooted.  Now the partitions were all mounted.  I ran the rsync command again.  The problem had not changed.

I found an informative thread in which bscbrit suggested trying to use cp to copy a single file from the source partition to the target partition.  If it worked, s/he said, the problem was not with permissions; it was just with trying to use rsync.  So in SOURCE I copied a file, renamed it "x.txt," and typed "cp /media/SOURCE/x.txt /media/TARGET," and it worked:  there was a copy of x.txt on TARGET.  It sounded like there could be a kind of mismatch between the ext3 source drive and the ntfs target.  I also came across an informative post by djgrandmarquis that said rsync could have problems with Windows file systems.  That was in response to a question about very slow backup times, which  I had also experienced.  I needed Windows file systems because I was backing up to an offsite drive that I sometimes shared with a Windows system.

I had previously investigated Beyond Compare (BC) as another way of synchronizing drives.  BC had drawn  a lot of praise.  Since my previous writeup, I had bought a copy and had been using it for several months on Ubuntu, WinXP, and Linux.  I had subsequently found a comparison of many such programs; but since I had already bought BC and was using it with good results, I didn't explore that comparison.  I could verify that BC was fast and trouble-free when compared to rsync, especially when going between ext3 and ntfs drives.  I also liked the visible information about what was being changed and backed up.  So I decided to stop using rsync and start relying only on BC for my backups.

There were a few differences between the Windows and Linux versions of BC; but in an impressively thorough reply to my post, Craig from BC said that the portable version of BC ran well on Wine in Ubuntu.  I had already installed Wine, and I had also already been using the portable version of BC3; I preferred portable versions to save time during reinstallations, as I was able to save them elsewhere than drive C and therefore they weren't wiped out during a Windows reinstall.  So to try this out, I copied over the BC3 portable installation and tried using a funky procedure involving PortableApps to try the Windows version.  But now it seemed I had not properly understood or used that procedure:  PortableApps now appeared to be looking, not for .exe files, but for ".paf exe" files.  Meanwhile, it looked like Beyond Compare 3 would run fairly well under Wine, so I gave it a try.  Following the steps described in that previous post, in Terminal I navigated to the folder where I had the original BC installer, typed "ls" to get the exact filename, and typed "wine BCompare-3.1.10.11626.exe" to install that file.  Terminal reported some "fixme" and "err" messages; nonetheless, the Beyond Compare 3 Setup Wizard ran.  I had a licensed version, so I entered the license key, and followed the instruction to restart.  It worked, but now I came to the new realization that Windows identifies drives by letter (C: etc.), while Ubuntu identifies them by name (/media/DATA, etc.).  So of course a Windows version of Beyond Compare, looking for Windows-type folder paths, running in Ubuntu, is going to find only a limited number of Ubuntu locations.  I didn't pursue that problem.

This left at least one issue unresolved.  While I was OK with doing manual comparisons for now, I was still not at the point of being able to write scripts to automate BC backup processes.  I started another post on that.

Ubuntu 9.04: Backing Up and Copying Webpages and Websites

As described in a previous post, I had been using rsync to make backups of various files.  This strategy was not working so well in the case of webpages and websites, or at least I wasn't finding much guidance that I could understand.  (Incidentally, I had also tried the Windows program HTTrack Website Copier, but had found it to be complicated and frustrating.  It seemed to want either to download the entire Internet or nothing at all.)

The immediate need driving this investigation was that I wanted to know how to back up a blog.  I used the blog on which I am posting this note as my test bed.

Eventually, I discovered that maybe what I needed to use was wget, not rsync.  The wget manual seemed thorough if a bit longwinded and complex, so I tried the Wikipedia entry.  That, and another source, gave me the parts of the command I used first:

wget -r -l1 -np -A.html -N -w5 http://raywoodcockslatest.blogspot.com/search?max-results=1000 --directory-prefix=/media/Partition1/BlogBackup1

The parts of this wget command have the following meanings:

• -r means that wget should recurse, i.e., it should go through the starting folder and all folders beneath it (e.g., www.website.com/topfolder and also www.website.com/topfolder/sub1 and sub2 and sub3 . . .)
  
• -l1 (that's an L-one) means stay at level number one.  That is, don't download linked pages.
• -np means "no parent" (i.e., stay at this level or below; don't go up to the parent directory)
• -A.html means Accept only files with this extension (i.e., only .html files)
• -N is short for Newer (i.e., only download files that are newer than what's already been downloaded).  In other words, it turns on timestamping
• -w5 means wait five seconds between files.  This is because large downloads can overload the servers you are downloading from, in which case an irritated administrator may penalize you
  
• The URL shown in this command is the URL of this very blog, plus the additional information needed to download all of my posts in one html file.  But it didn't work that way.  What I got, with this command, was each of the posts as a separate html file, which is what I preferred anyway
• --directory-prefix indicates where I want to put the download.  If you don't use this option, everything will go into the folder where wget is running from.  I came across a couple of suggestions on what to do if your path has spaces in it, but I hadn't gotten that far yet
  

Incidentally, I also ran across another possibility that I didn't intend to use now, but that seemed potentially useful for the future.  Someone asked if there was a way to save each file with a unique name, so that every time  you run the wget script, you get the current state of the webpage.  One answer involved using mktemp.  Also, it seemed potentially useful to know that I could download all of the .jpg files from a webpage by using something like this:  wget -e robots=off -r -l1 --no-parent -A.jpg http://www.server.com/dir/

The first download was pretty good, but I had learned some more things in the meantime, and had some questions, so I decided to try again.  Here's the script I used for my second try:

wget -A.html --level=1 -N -np -p -r -w5 http://raywoodcockslatest.blogspot.com --directory-prefix=/media/Partition1/BlogBackup2

This time, I arranged the options (or at least the short ones) in alphabetical order.  The -p option indicated that images and style sheets would be downloaded too.  I wasn't sure I needed this -- the basic html pages looked pretty good in my download as they were -- but I thought it might be interesting to see how much larger that kind of download would be.  I used a shorter version of the source URL and I designated a different output directory.

I could have added -k (long form:  --convert-links) so that the links among the downloaded html pages would be modified to refer to the other downloaded pages, not to the webpage where I had downloaded them from; but then I decided that the purpose of the download was to give me a backup, not a local copy with full functionality; that is, I wanted the links to work properly when posted as webpages online, not necessarily when backed up on my hard drive.  I used the long form for the "level" option, just to make things clearer.  Likewise, with a bit of learning, I decided against using the -erobots=off option.  There were probably a million other options I could have considered, in the long description of wget in the official manual, but these were the ones that others seemed to mention most.

The results of this second try were mixed.  For one thing, I was getting a lot of messages of this form:

2010-01-01 01:43:03 (137 KB/s) - `/[target directory]/index.html?widgetType=BlogArchive&widgetId=BlogArchive1&action=toggle&dir=open&toggle=MONTHLY-1196485200000&toggleopen=MONTHLY-1259643600000' saved [70188]

Removing /[target directory]/index.html?widgetType=BlogArchive&widgetId=BlogArchive1&action=toggle&dir=open&toggle=MONTHLY-1196485200000&toggleopen=MONTHLY-1259643600000 since it should be rejected.

I didn't know what this meant, or why I hadn't gotten these kinds of messages when I ran the first version of the command (above).  It didn't seem likely that the mere rearrangement of options on the wget command line would be responsible.  To find out, I put it out of its misery (i.e., I ran "pkill wget" in a separate Terminal session) and took a closer look.

Things got a little confused at this point.  Blame it on the late hour.  I thought, for a moment, that I had found the answer.  A quick glance at the first forum that came up in response to my search led me to recognize that, of course, my command was contradictory:  it told wget to download style sheets (-p), but it also said that only html files would be accepted (-A.html).  But then, unless I muddled it somehow, it appeared that, in fact, I had not included the -p option after all.  I tried re-running version 2 of the command (above), this time definitely excluding the -p option.  And no, that wasn't it; I still got those same funky messages (above) about removing index.html.  So the -p option was not the culprit.

I tried again.  This time, I reverted to using exactly the command I had used in the first try (above), changing only the output directory.  Oh, and somewhere in this process, I shortened the target URL.  This gave me none of those funky messages.  So it seemed that the order of options on the command line did matter, and that the order used in the first version (above) was superior to that in the second version.  To sum up, then, the command that worked best for me, for purposes of backing up my Blogger.com (blogspot) blog, was this:

wget -r -l1 -np -A.html -N -w5 http://raywoodcockslatest.blogspot.com --directory-prefix=/media/Partition1/BlogBackup1

Since there are other blog hosts out there, I wanted to see if exactly the same approach would work elsewhere.  I also had a WordPress blog.  I tried the first version of the wget command (above), changing only the source URL and target folder, as follows:

wget -r -l1 -np -A.html -N -w5 http://raywoodcock.wordpress.com/ --directory-prefix=/media/Partition1/WordPressBackup

This did not work too well.  The script repeatedly produced messages saying "Last-modified header missing -- time-stamps turned off," so then wget would download the page again.  As far as I could tell from the pages I examined in a search, there was no way around this; apparently WordPress did not maintain time stamps.

The other problem was that it did not download all of the pages.  It would download only one index.html file for each month.  That index.html file would contain an actual post, which was good, but what about all the other posts from that month?  I modified the command to specify the year and month (e.g., http://raywoodcock.wordpress.com/2009/03/).  This worked.  Now the index.html file at the top of the subtree (e.g., http://raywoodcock.wordpress.com/2009/03/index.html) would display all of the posts from that month, and beneath it (in e.g., .../2009/03/01) I had named subfolders for each post, each of which contained the index.html file displaying that particular post.  So at this rate, I would have to write wget lines for each month in which I had posted blog entries.  But then I found that removing the -A.html option solved the problem.  But if I ran it at the year level, it worked only for some months, and skipped others.  I tried what appeared to be the suggestion of running it twice at the year level (i.e., at .../wordpress.com/ with an ending slash), with --save-cookies=cookies.txt --load-cookies=cookies.txt --keep-session-cookies.  That didn't seem to make a difference.  So the best I could do with a WordPress blog, at this point, was to enter separate wget commands for each month, like this:

wget -r -l1 -np -N -A.html -w5 http://raywoodcock.wordpress.com/2009/01 --directory-prefix=/media/Partition1/WordPressBackup

I added back the -A.html option, as shown, because it didn't seem to hurt anything; html pages were the only ones that had been downloaded anyway.

Since these monthly commands would re-download everything, I would run the older ones only occasionally, to pick up the infrequent revision of an older post.  I created a bunch of these, for the past and also for some months into the future.  I put the historical ones in a script called backup-hist.sh, which I planned to run only occasionally, and I put the current and future ones into my backup-day.sh, to run daily.

But, ah, not so fast.  When I tried this on another, unrelated WordPress blog, it did not consistently download all posts for each month.  I also noticed that it duplicated some posts, in the sense that the higher-level (e.g., month-level) index.html file seemed to contain everything that would appear on the month-level webpage on WordPress.  So, for example, if you had your WordPress blog set up to show a maximum of three posts per page, this higher-level webpage would show all three of those.  The pages looked good; it was just that I was not sure how I would use this mix in an effective backup-and-restore operation.  This raised the question for my own blog:  if I ever did have to restore my blog, was I going to examine the HTML for each webpage manually, to re-post only those portions of text and code that belonged on a given blog page?

I decided to combine approaches.  First, since it was yearend, I made a special-case backup of all posts in each blog.  I did this by setting the blogs to display 999 posts on one page, and then printed that page as a yearend backup PDF.  Second, I noticed that rerunning these scripts seemed to catch additional posts on the subsquent passes.  So instead of separating the current and historical posts, I decided to stay with the original idea of running one command to download each WordPress post.  I would hope that this got most of them, and for any that fell through the crack, I would refer to the most recent PDF-style copy of the posts.  The command I decided to use for this purpose was of this form:

wget -r -l1 -np -N -A.html -w5 [URL] --directory-prefix=/media/Backups/Blogs/WordPress

I had recently started one other blog.  This one was on Livejournal.com.  I tried the following command with that:

wget -r -l1 -np -N -A.html -w5 http://rwclippings.livejournal.com/ --directory-prefix=/media/Backups/Blogs//LiveJournal

This was as far as I was able to get into this process at this point.

Ubuntu: Schedule Items with Cron

I wanted to schedule regular backups in Ubuntu 9.10.  I had already worked out the rsync commands I wanted to use; now it was a matter of running them automatically at certain times or on certain days.  I began by seeing what was already scheduled in my crontab (i.e., my chronological table).  Actually, I had two of them:  one for me, and one for the root (i.e., administrator).  I checked them with "crontab -l" (that's a small L) and "sudo crontab -l" and both say "no crontab."  This supposedly meant that there were no crontab files in /var/spool/cron/crontabs.  I verified that via "sudo nautilus."  It seemed like that would apply to the root's cron, but I wasn't able to find any different location where the user's cron should be, so I just moved on to the next step.

The next step was to edit the crontab by using "crontab -e."  This seemed to be creating a crontab for me, as distinct from root:  it said "no crontab for ray - using an empty one."  To confirm that, I tried in a separate Terminal session with "sudo crontab -e."  It seemed to flash the same choice as had appeared for me, showing a choice of editors; but then it went directly into nano, which the other Terminal session was describing as the "easiest" of the three available editors.  So, OK, since I ordinarily ran my rsync backups as me, user, not as root, I figured I would want to set up my own crontab, not a root crontab.  (Later, I found some statements that it was a bad idea to edit root's crontab.)  So I killed that nano session and went back to the first Terminal session.  There, I chose no. 3, nano, as my editor.

The top of the nano screen was showing me "# m h  dom mon dow   command."  This was my cue for the things that I needed to enter on a line, in order to schedule a cron job:  minute, hour, day of month, month, day of week, and command.  (The leading # was to indicate that this sample line was just a comment and should not be executed.)

According to About.com, up through the week level, permissible values began at zero:  that is, minutes of the hour ran from 0 to 59, hours of the day ran from 0 to 23, and days of the week ran from 0 to 7 (where Sunday was both 0 and 7, as you prefer).  Beyond that, days of the month ran from 1 to 31, and months of the year ran from 1 to 12.  If they had names, you could use their first three letters (e.g., "Mon" and "Jul" but not "minute 23"). Cron uses the union (not the intersection) of the two day commands.  That is, if you specify a day of the week (e.g., Fri) and also a day of the month (e.g., 15), the command will run on both days (e.g., every Friday, and also the 15th of every month).  You could use an asterisk to indicate "every"; for example, * * * * * would indicate that you want to run the command every minute of every hour of every day of every month.

There were additional options for numbers below the date level; that is, these wouldn't work on days of the week or of the month.  One of these options was to use fractions:  for instance, */4 would mean "every fourth" minute or hour or whatever.  You could also use a range:  40-45 would mean it should run every minute of the hour from 40 to 45 minutes (i.e., 12:40 AM, 12:41 AM, 12:42 AM . . . 12:45 AM, 1:40 AM . . . 11:45 PM).  You could use lists, separating items with commas, so that 40,41 would mean that it should run only on the 40th and 41st minutes of the hour. So designating the hour as 0-11/2 would mean that it should run every other hour, in the morning only.  You could use a list of ranges; for example, an hour designation of 0-1,10-11 would indicate that the command should run in the first two and also in the last two hours of the morning.  Range and list commands start on the first number; for instance, 2-6/2 runs at 2 AM, 4 AM, and 6 AM (at whatever minute you specify).

There was one other category of entry:  special words.  These words would replace all five numbers.  In other words, if you wanted the precise control offered by the numbers for minute, hour, etc., use the numbers; but if you want the convenience of just entering one word without having to think much about what it means, use the word.  These special words were @reboot (run at reboot), @yearly or @annually (once a year), @monthly, @weekly, @daily or @midnight (run once a day), and @hourly.  These all run as soon as the time period starts (e.g., January 1, 12:00 midnight).  There was more to know about these commands, in the official documentation.

I decided the first thing to schedule was a backup of my CURRENT partition to a backup internal hard drive.  I wanted this backup to run several times a day.  I hadn't set it up as a RAID array because I didn't want it to happen immediately; I wanted to allow some time in case I accidentally deleted something, or made some other stupid mistake.  The more frequently it ran, the more likely it would contain the most recent version of the relevant folder - which could mean it would be more likely to have the version that existed *after* my stupid mistake.  My compromise was to set it up to run every two hours.  The cron line I used, then, was this:

0   */2   *   *   *   [rsync command]

In crontab, I put several spaces between the numbers for readability.  Here, in this blog posting, I had to use nonbreaking spaces for the display shown above, because plain old spaces tend to get ignored in HTML.  I haven't reproduced, here, the long command that I want executed, because I want to focus on the cron parts of the line.  (The rsync command of choice is shown in the previous post.)

Then it occurred to me that, instead of putting that long command in cron, where I would have to do some minor translation every time I wondered what it meant, I could probably write a basic Ubuntu shell script that would do the same thing and would allow me to add explanatory notes and other commands.  So I took a brief detour into the land of scripts.  By the time I returned and finished my look at rsync, I had two scripts.  One was called backup-hour.sh, to be run every few hours.  The other was called backup-day.sh.  I put them into /home/ray/bin and wrote the following cron lines for them:

0   */2   *   *   *   ./home/ray/bin/backup-hour.sh

0   2   *   *   *   ./home/ray/bin/backup-day.sh

The first one would hopefully run backup-hour.sh every two hours, all day and all night.  The second one was intended to run backup-day.sh every day at 2 AM.

To put these lines into crontab, I typed crontab -e.  It all looked good.  But nothing was happening.  A couple of days went by, and cron didn't run.  The problem, I suspected, was with those periods I had put at the start of my path names.  I had thought that was part of a command, but nobody else was using them in their cron files.  So I deleted those and waited until the next even-numbered hour, to see if backup-hour.sh would run.

Then I wondered whether I was saving crontab in the right place.  I noticed that nano, my default crontab editor, was saving it to /tmp/crontab.zfItNF/crontab.  Somehow, that didn't look right.  I did a quick search and found a variety of theories on where crontab should be, and none of them involved the /tmp folder.  Someone suggested typing "which crontab" at the prompt.  That came back with /usr/bin/crontab, which wasn't one of the options those other people had suggested.  I tried to save this crontab to /usr/bin/crontab and got a message that the file already existed.  I tried "gedit /usr/bin/crontab," but even with sudo I got a message that the file could not be opened.  I decided to pass on that one for the time being and, selecting a seemingly knowledgeable opinion, I thought about saving it in /var/spool/cron.  It turned out that there was a subdirectory there, and more specifically we had a file called /var/spool/cron/crontabs/ray.  When I looked in that, I saw a copy of my crontab file, the one that nano had been trying to save in a /tmp folder, except that it began with the line, "DO NOT EDIT THIS FILE - edit the master and reinstall."  So it seemed that maybe nano knew what it was doing after all.  So I let nano save crontab in that /tmp folder after all.  Then, following some advice, I decided to output the error messages, if any, to a crontab.log file.  So the first complete line in my crontab looked like this:

0 */2 * * * /home/ray/bin/backup-hour.sh 2>> /Folder1/crontab-errors.log

Unfortunately, at 4:00 PM, nothing happened.  I decided to follow the suggestion that it is much easier to use the gnome-schedule package (though there were problems for those who upgraded from Ubuntu 9.04 to 9.10 rather than doing a fresh install), so I installed that in Synaptic.  This gave me a new option at Applications > System Tools > Scheduled Tasks.  (Scheduled Tasks was apparently a simple front end for cron.)  But before Scheduled Tasks would work, I had to make sure that crontab and a program called At were installed.  Both were marked as installed in Synaptic.  Cron was for recurrent tasks, and At was for one-time jobs (e.g., run this "At" startup).

The basic idea, it developed, was that crontab -e would add scheduling files to the /var/spool/cron/crontabs folder, but it was apparently advisable to just let crontab -e do that, and not try to find and edit those files directly.  I tried typing "sudo /etc/init.d/cron restart," but that gave me a suggestion:  "Rather than invoking init scripts through /etc/init.d, use the service(8) utility, e.g. service cron restart."  So, OK, I tried that.  This gave me a long message that started with "restart: Rejected send message."  That didn't sound good, so I tried "ps -eaf | grep cron" and that gave me these three lines:

ray       8381  8343  0 19:46 pts/0    00:00:00 man 5 crontab
root      8409     1  0 19:50 ?        00:00:00 cron
ray       8416  8343  0 19:52 pts/0    00:00:00 grep --color=auto cron

When I had tried running that command previously, I had gotten only the second and third lines, not the first.  So perhaps the problem had been that my cron had not been running when I had tried to run it previously, and now it was running.  On that assumption, I could have just gone back to the command-line approach at this point, but I liked the added option of using At to schedule one-time events. But as I checked further, "crontab -l" said "no crontab for ray," so apparently crontab was *still* not running.  But no, one source said, "The output: no crontab for [username] means crontab is installed."

I found a long thread that told me I could use gedit instead of nano, to edit crontab, by typing this:

export EDITOR=gedit && crontab -e

(The "&&" part combined separate commands; apparently I could have achieved the same thing by typing these two on separate lines.)  Someone said I could make this permanent by putting the command in my .bashrc file, which they said I would find in /home/ray, which was true.  But if .bashrc hadn't been there, apparently I could have used gedit to create it, with something like this:

# .bashrc - bash config file #
# export variables
export EDITOR=gedit

Since .bashrc was already there, I just added those last two lines to the end of it.  But anyway, back at the cron issue, someone in that long thread said I could change my cron line to look like this:

* * * * * export DISPLAY:=0 && xterm [command]

if I wanted to see something on the screen when the command was executing.  But this got me back to the realization that, for all of the flexibility I was seeing as I worked my way through page 9 of that very long thread, I would probably prefer, right now, to just get something working.  So I guessed that Scheduled Tasks would work just fine if plain old crontab was working.  So, as others had done, I wrote up a simple command to test crontab.  The entry looked like this:

* * * * * export DISPLAY:=0 && xterm dir

Sadly, this did nothing.  I opened Scheduled Tasks, thinking I would try something similar there, and instead I saw that my crontab line was already there, albeit in ugly form: 

Recurrent    At every minute     export DISPLAY:=0 &amp&amp dir

Anyway, that didn't seem to be running, so I deleted it, in Scheduled Tasks, and tried doing similar as a one-time task.  Here's what I ran:

dir > /home/ray/DIRDIRDIR

and it worked!  I got a text file named DIRDIRDIR that contained a directory listing.  So it seemed the one-time part of Scheduled Tasks was working properly.  So I returned to the question of recurrent tasks.  I remembered that, in Ubuntu, we use "ls" rather than "dir."  So in Scheduled Tasks, I used more or less the same command to append updated directory listings each minute, showing the time when DIRDIRDIR had been last updated:

ls -l >> /home/ray/DIRDIRDIR

and that worked too.  So now I felt I should try again with the lines I had attempted earlier, as revised.  This time, I entered them into Scheduled Tasks rather than into crontab, so I didn't need the * * * * * parts of the entries.  So here are the commands I entered in Scheduled Tasks:

/home/ray/bin/backup-hour.sh 2>> /Folder1/crontab-errors.log
/home/ray/bin/backup-day.sh 2>> /Folder1/crontab-errors.log

I didn't actually enter them both right away; I started with the first one, and made it run just once, at nine minutes past the hour (which was about two minutes ahead of when I was working on it).  When the time came, it ran, or it seemed to:  there was now a file named "crontab-errors.log" with 0 bytes in Folder1.  That was good enough for now.  I tried another line, this time telling Scheduled Tasks to make it an "X application" rather than "Default behavior."  That didn't seem to do anything, but whatever.  It looked I had what I needed, and I could add more knowledge later.

Basic Ubuntu (Bash) Shell Scripts

In Ubuntu 9.04, I wanted to write a script that would execute an rsync command, so that I could put a brief reference to the script into my crontab file, instead of putting the whole long rsync command there.

For a brief, tiny moment, I was almost tempted to consider learning the GAMBAS (Gambas Almost Means BASIC) programming language, just because (pre-Visual) BASIC was the only programming language I ever learned.  Instead, I moved toward basic instructions on writing a shell script.  Here's what I wrote:

#!/bin/bash
# This is backup-hour.sh
# It backs up CURRENT to CURRBACKUP every few hours
rsync -qhlEtrip --progress --delete-after --ignore-errors --force --exclude=/.Trash-1000/ --exclude=/lost+found/ /media/CURRENT/ /media/CURRBACKUP

As the instructions said, the first line was essential to tell the computer to use BASH to interpret the following lines.  The next two lines were comments, and the final line (wrapping over onto multiple lines here) was exactly the rsync line I'd been using to do the backup.  In other words, learning how to write the command was almost all I needed to write the script.

The next step was to save it somewhere.  I had previously heard, and the instructions said, that the common place to put it is in your bin folder.  I went with that, but made a note that I would need to be sure to back up my bin folder, because I didn't want to go to the trouble of writing all these scripts and then see them vanish.

The usual location for the user's bin folder is at /home/[username]/bin.  In my case, that's /home/ray/bin.  Getting there in Nautilus (i.e., Ubuntu's File Browser, also started by typing "nautilus" in Terminal) can be confusing:  you can get there via the Home Folder option (assuming you're showing Tree rather than Places or something else (or nothing) at the left side of the file browser), or you can go to File System/home/[username]/bin.  Same thing, either way.  So I saved the script (above) in my bin folder as backup-hour.sh.  That turned the comment lines (beginning with #) blue in gedit.

Next, the instructions said, I needed to set permissions.  This was a confusing aspect of Ubuntu.  The documentation seemed to say that there were ten permissions that a person could set.  These were represented by ten hyphens or minus signs:  ----------.  I couldn't tell what the first one was for, but the remaining nine were divided into three sets of three.  The first three belonged to the owner, the second three to the group, and the last three to "other."  Within each set of three, the first one was for read, the second was for write, and the third was for execute (i.e., run it as a program).  So if you set the owner's permissions to read (r), write (w), and execute (x), your ten hyphens would now change to this:  -rwx------.  If you set all three parties (i.e., owner, group, and other) the same, they would look like this:  -rwxrwxrwx.

You could set the permissions using the chmod command.  I found an Ubuntu manual page on chmod.  It was not really that complicated, but it looked like it was going to require a time investment to make sure I had it right, and at this point I was getting impatient.  The basic idea seemed to be that you could use chmod to enter values of 4 (for read permission), 2 (for write permission), and/or 1 (for execute permission).  So, for example, you could type "chmod 755" and that would give a value of 7 to the first of the three users mentioned above (i.e., the owner), a value of 5 to the second of the three (i.e., the group), and a value of five to the third of the three (i.e., other).  The 7 would mean that you gave read + write + execute (4 + 2 + 1) permissions to the owner, whereas the 5 would mean that you gave only read + execute (4 + 1) permissions to the rest.  Since that's what the instructions suggested, I went with that.  To set the script with those permissions, I typed "chmod 755 backup-hour.sh."

I wasn't too sure of who the owner was (i.e., me or root), not to mention the group or other.  I mean, this was for my home computer.  Not a lot of people milling around, waiting to take my hard drive for a spin.  These kinds of options seemed to be set up for networked computers, where the "accounting" department might be a group that would own a file.  I found what looked like a good tutorial on file owners, and another interesting (yawn!) page about permissions, but fortunately I did not have time to work through them.

When I typed "chmod 755 backup-hour.sh," I got "cannot access 'backup-hour.sh': No such file or directory."  One solution was to use a change directory (cd) command to get the Terminal prompt into the bin subfolder, so it would see what I was looking for.  But since I planned to put more scripts into that folder, and anyway since I wanted cron or other programs to know right away what I was talking about when I referred to something like backup-hour.sh, I decided to figure out how to put the bin folder in my "path."  The path is the list of folders where the operating system looks for guidance on what a command means.  To change my path so that the system would always know to look in bin, they said I needed to find and edit my .bash_profile file.  Unfortunately, they didn't say where it was.  It wasn't easy to find.  I ran searches in Nautilus (both user and root), but while those were grinding away, I found that I could just type "locate .bash_profile."  That turned up nothing, but very quickly.  Then I got some advice that, if it didn't exist, I could create it by using "touch ~/.bash_profile."  So I did that, and then tried again with "chmod 755 backup-hour.sh."  Still no joy.  Ah, but maybe that was because I hadn't rebooted; backup-hour.sh would run only on startup.  OK, so I used the other approach after all:  I changed directory to the bin folder and tried again.  Now I got "Permission denied."  What if I gave everybody full permissions with chmod 777?  I tried that instead of chmod 755.  That seemed to do it.  The hard drive was doing its thing now.

I wanted to see what was going on, so I decided to create a log file.  I wanted it to store only the error messages, not to list the thousands of files that backup-hour.sh was backing up successfully, so I put this on the end of (that is, on the same command line as) my rsync command in backup-hour.sh (above):

2> /media/CURRENT/backup-hour.log

The log filename thus matched the script filename.  I put "backup" first so that I could see all of my backup scripts in the same part of the folder's directory listing, and then I set up a backup-day.sh script along the same lines.  New problem:  these backup scripts would generate empty log files if there were no errors, and I didn't want to have to delete them manually.  So I found a forum post with advice on how to delete them automatically, using the "find" command.  In my version, it looked like this:

find /media/CURRENT/ -name "*.log" -size 0c -exec rm -f {} \;

I put that at the end of the backup-day.sh script, and it seemed to work.  It said, basically, look in the CURRENT folder for files whose name ends with .log and have zero bytes; and if you find any files like that, execute the "remove" command without asking for permission.  I didn't know what that ending punctuation is about, but that's what the advisor suggested.

In my backup-day.sh (not backup-hour.sh) script, I included the instructions for updating my USB jump drive (above).  I also included a set of commands to save my e-mail (I was using Thunderbird in Ubuntu) as a .tar compressed file. Actually, as seven .tar files, one for each day of the week.  That part of backup-day.sh looked like this:

# Assign Thunderbird mail & profile to be backed up
backup_files="/home/ray/.mozilla-thunderbird"
dest="/media/BACKROOM/Backups/Tbird"
# Create archive filename
day=$(date +%A)
hostname=$(hostname -s)
archive_file="$hostname-$day.tgz"
# Back up to a tgz file
tar zcvf $dest/$archive_file $backup_files 2>> /media/CURRENT/A-INCOMING/T-bird-Backup.log

So these seemed to be the basic kinds of tools I needed to set up rsync scripts and crontab entries.

Ubuntu: Backup with Rsync

In a previous post, I got as far as concluding that rsync was the tool of choice for backing up my computer in Ubuntu 9.04. I didn't pursue it because I was short on time and patience for writing scripts at that point. But eventually the need for a regular backup system became acute. So this post logs the steps I took to make rsync and cron work for me.

First, here's what I wrote previously:

As an alternative to rdiff-backup, what people had actually mentioned more frequently was rsync. It did not have the incremental backup features of rdiff-backup, to my knowledge, but it seemed to be an established tool for backup purposes. So for now, at least, I thought I might try that instead. Once again, I did a Google search and got a package details page with no apparent link to any help files. Eventually I found what looked like the official rsync webpage and, after looking at their FAQs and some other pages, landed on their Examples page. It was intimidating.

This time, I went to their Documentation page. This gave me links to, among other things, Michael Holve's rsync tutorial. The tutorial said, "You must set up one machine or another of a pair to be an "rsync server" by running rsync in a daemon mode." I was curious, so I did a Google search for "what is daemon mode" and I got back, would you believe, exactly one page. One webpage in the entire known planet answered the question, "What is daemon mode?" Except it didn't really answer it. It just said, "It makes wget put standard output into a log file and not bug you while downloading." Accepting that as the best available answer (and ten points to the answerer!), I typed "rsync --daemon" in Ubuntu's Terminal and proceeded to the next step, "Setting Up a Server." After reading it, I decided it didn't seem to apply to me. It was for people who wanted to back up files between computers. I just wanted to back up to another drive.

So I went on to the tutorial's "Using Rsync Itself" section. Since I wasn't sure what daemon mode did, or if it was necessary, I killed that Terminal session and started another. I didn't know if that would shut off daemon mode, or if doing so was what I should do. I read the section and then checked another source of documentation, the rsync man page ("man" being short for "manual"). The man page would ordinarily be output in response to a Terminal command, but someone had put it here in html form, so that's what I used. It reminded me, first, to check Ubuntu's System > Administration > Synaptic Package Manager to make sure I had rsync already installed, here on my secondary computer. I searched Synaptic for rsync and got back a couple dozen listed programs; rsync was among them and was shown as being installed. I looked partway into the man page and got an answer to one question I had from the tutorial. So here's how I translated what the tutorial was telling me. First, the tutorial listed these lines:

rsync --verbose --progress --stats --compress --rsh=/usr/local/bin/ssh --recursive --times --perms --links --delete \
--exclude "*bak" --exclude "*~" \
/www/* webserver:simple_path_name

The first thing to know was that this all represented a single command line. It was too long to fit on one line, though, so apparently the trailing backslash said, "This line continues on the next line." The command would be typed into a file and saved as a script, not typed directly into Terminal. I didn't know why the first line didn't end with a backslash. I decided I would want to experiment with this in a relatively safe place -- with a junk directory on my secondary computer, perhaps -- to see what it was doing.

So as the tutorial explained it, the first line of this example told rsync how to proceed: verbosely (i.e., with lots of information about what it was doing), showing a progress report, with statistics. The rsh part was for encryption, to be used optionally if you were sending your stuff online to another computer. I wasn't, so I decided to try leaving that off. I also didn't want to compress the output, because that made the process slower and required more attention from the CPU.

The second line of the example, above, told rsync to recurse -- to work through all of my directories and subdirectories under the folder that I would be naming. It also told rsync to preserve file timestamps and file permissions -- so if, for example, a file was readable only by root on the source drive, it would be the same way on the target drive. The -- links command was an instruction to preserve symbolic links -- not sure what that meant -- and the --delete command, as I understood it, would tell rsync to delete anything on the target that wasn't on the source. So you'd have a mirror, and not just an accumulation of backups of files that you have deliberately trimmed out of your file collection.

The third line of the example told rsync not to bother copying some kinds of files. I liked the sound of that at first, but then I decided I would rather be able to do a Properties comparison of source and target and verify that both had exactly the same number of files. So I decided to leave out this line when I used rsync.

The fourth line of the example named the source and target locations. I wasn't going to be using it with a remote source or target, so mine was going to look somewhat different from this.

On that basis, here's what I assembled as a test version of the rsync example from above:

rsync --verbose --progress --stats \
--recursive --times --perms --links --delete \
/media/DATA/Source /media/DATA/Target

I created a Source subfolder in my DATA folder and put a TestFile.txt file into it. I also created a Target subfolder in DATA. Then I copied those three rsync lines into a file in Ubuntu's Text Editor (gedit) and saved it to Desktop as TestRun. To make it executable, I went into Terminal, typed "cd /home/ray/Desktop" and then "chmod +x TestRun" and then double-clicked on TestRun and said Run. And, you know, it worked. Just like that. Not exactly as intended -- I had not only TestFile.txt but also the whole Source folder underneath my Target folder -- but, yeah, there it was. I deleted the Target folder and ran it again and, sure enough, it created the Target folder and then inserted a copy of the Source folder into it. Excellent!

Now it was time to try something a little bolder. I wanted to see how it worked if I tried to copy the whole DATA folder to an external drive. This part was a little confusing. The external drive seemed to have two different names. If I looked in /media, its name was simply "disk." But if I hit the Computer icon in File Browser, it came up as "193.8 GB Media." I decided the latter sounded more specific, so I would try that first. So now the third line of my TestRun file read like this:

/media/DATA "/media/193.8 GB Media"

I used quotation marks because there were spaces in the name. I saved TestRun and double-clicked it again on the Desktop. It didn't seem to do anything. I realized that I had probably made a mistake in that line, and tried again like this:

/media/DATA "/193.8 GB Media"

That didn't do it either, so I tried again, without the leading slash:

/media/DATA "193.8 GB Media"

That still didn't work, so I tried the other approach:

/media/DATA /media/disk

Still nothing. I went into System > Administration > Partition Editor (GPartEd), wiped out the target partition, and recreated it as a FAT32 partition. Now the drive was totally invisible to Ubuntu. I went back into GPartEd and reformatted it as an ext3 partition. Then I realized: it was an IDE drive, so apparently it would not be recognized until I rebooted. I decided to reboot into Windows (I had a dual-boot system) and format it as NTFS. I named it 186GB (which seemed to be the net amount of space available in NTFS format) and rebooted into Ubuntu. I revised TestRun's last line again:

/media/DATA /media/186GB

and ran it again. This time, it seemed to be working -- the external 186GB drive was making noise -- but I wondered why I wasn't getting a verbose indication of what was going on. I guessed that, if I wanted the verbose information, I would have to execute rsync on the command line, not in an executable script. While I was rooting around for an answer to that question, I was reminded that I could also use the shorthand versions of these commands. So instead of typing --verbose into the script, I could just type -v and whatever other letters I needed. In this approach, the final contents of TestRun, which were as follows:

rsync --verbose --progress --stats \
--recursive --times --perms --links --delete \
/media/DATA /media/186GB

could instead be expressed like this, if I understood the man page's Options Summary section correctly:

rsync -vshlEPtrip --del --delete-excluded --force \
--exclude RECYCLER \
--exclude "System Volume Information" \
/media/DATA /media/186GB

In that version, I added a couple other options that seemed appropriate, and also told rsync to exclude (i.e., don't copy) those extra folders that Windows XP seemed to put on every drive. I revised TestRun along these lines and, when the external drive settled down and it looked like the foregoing TestRun process had ended, I ran it again. But it didn't seem to make any difference. The extra folders were still there. I had understood that it would delete them. Part of the problem seemed to be that I had not used the syntax correctly. I was supposed to use an equals sign: --exclude=RECYCLER. But another part of the problem was that it was not clear whether the "exclude" command was supposed to work with directories. It didn't seem so. The man page just referred to excluding files. I tried again with equals signs, but still no change. I posted a question on it, but the kind response was unfortunately not able to resolve the issue.

Next, I tried a modified version of TestRun on the primary computer. I went through several revisions and wound up with this version, which seemed to work:

rsync -vchlEtrip --progress --del --ignore-errors --force /media/CURRENT/ "/media/OFFSITE/P4 CURRENT"

The partition being backed up, in this case, was an ext3 partition named CURRENT, and the target to which it was being backed up was a USB external drive named OFFSITE. (Some weeks have passed since I started this post, so there may be some discontinuity in my writing at this point.)

I did not run this command as a script within a file called TestRun that I would start by double-clicking on it, because I discovered that you would only get the detailed output if you entered the rsync command on the command line. I was able to enter all of the foregoing rsync command on one line. I did not need the "exclude" commands because this was not an NTFS drive formatted by Windows. I still had ext3 "lost+found" and Trash folders that got copied over in this way, but they were small, so it was OK.

As I think I may have said before, I got the selected rsync parameters by typing "man rsync" at the command line. It did take some trial and error to get this particular set. The resulting backup, when checked by right-clicking and selecting Properties, seemed to be virtually identical.

When I ran that rsync command, it showed me lots of detail on what it was doing. It concluded with this message:

rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9]

Eventually, however, I did figure out how to do it.  Here is an example of an rsync command that worked for me:

rsync -qhlEtrip --progress --delete-after --ignore-errors --force --exclude=/.Trash-1000/ --exclude=/lost+found/ /media/CURRENT/ /media/CURRBACKUP

This one would back up what Windows sees as drive D (named CURRENT) to a partition that Windows sees as drive G (named CURRBACKUP).  Both partitions had to be mounted in Ubuntu before this would work.  I used a similar command to copy a folder on CURRENT to a USB jump drive named KINGSTON.  That gave me a portable copy of the current state of that folder, ready to take along.

The next thing I needed to do was to back up my blogs.  I started by just wanting to be able to back up a webpage.  I had discovered that all of the posts on a Blogger (i.e., Blogspot) blog like this one could be displayed in a single webpage, at least if you had less than 1,000 posts.  To do that, you just needed to go to this URL:  http://blogname.blogspot.com/search?max-results=1000.  I wasn't sure what would happen if you entered a larger number than 1000.  So now that I had that webpage, I wanted to know how to save a copy of it automatically.  Strangely, at this point, Google searches for any of these sets of terms

ubuntu "back up a webpage"
rsync "back up a webpage"
rsync "copy a webpage"

produced zero hits.  Eventually, it started to look like this was because I was barking up the wrong tree.  As described in a separate post, it seemed that what I wanted for this purpose might be wget, not rsync.

System Files Backup in Ubuntu 9.04

A quick note on the rsync command I used to make a backup of my system files in Ubuntu 9.04.   I already had a TIB file backup made with Acronis True Image, but it was a month old and I figured the individual files would be more accessible anyway if I just copied them straight over to another drive. The command I used for this was as follows:

sudo rsync -qhlEtrip --progress --delete-after --ignore-errors --force / "/media/BACKUP/Ubuntu System Files"

where BACKUP was the name of a hard drive partition.

rsync, USB Drive, Error 30: "Read-Only File System"

I was regularly copying a folder to a USB drive in Ubuntu using rsync.  Suddenly, for no apparent reason, the rsync process started producing these error messages:

rsync:  failed to set permissions on [USB drive/folder/filename]:  Read-only file system (30)

rsync:  mkstemp [USB drive/folder/filename] failed: Read-only file system (30)

rsync:  failed to set times on [USB drive/folder/filename]: Read-only file system (30)

rsync:  delete_file:  unlink [USB drive/folder/filename] failed: Read-only file system (30)

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1058) [sender=3.0.5]

The fix, for me, was to reformat the USB drive in Windows.  It would probably work even if Windows was not the cause of the problem.

Ubuntu and VMware: Miscellaneous Fixes

This is the latest installment in a series of long posts that log my successes and failures in the effort to replace Windows XP with Ubuntu. Starting with several posts in July 2008, and continuing for the better part of two months thereafter, I had developed what seemed to be, for me, the best solution. Introductions to those various posts provide more details of the two computers on which I was making this effort, as well as other aspects of the situation. Briefly, I had a primary computer and a secondary computer. Both were dual-booting Windows XP and 64-bit Ubuntu Hardy Heron 8.04. I was using VMware Workstation 6 to run Windows programs, when I could not find an acceptable Ubuntu alternative; and my further fallback was to run Windows programs in native WinXP by dual-booting the computers (preferably, whenever possible, the secondary computer). I had now received and installed an EVGA 256-P2-N751-TR GeForce 8600 GT 256MB video card to replace my previous EVGA 256-P2-N624-AR GeForce 7900 GS card. Both, no doubt, were fine cards, but I had invested a huge amount of time trying and failing to get it to work properly with my Ubuntu installation. I had even considered dialing back to the 32-bit version of Ubuntu but, for better or worse, had already spent $170 on the 64-bit version of VMware Workstation. So there was some motivation to try to make it work this way, if possible -- hoping, of course, that 64-bit was the future anyway. This saga within a saga had begun with my attempt to get multiple monitors working properly. I was not sure where to begin, at this point, but the first thing I looked at was Ubuntu's System > Administration > Hardware Drivers, there on the primary computer. I saw, "No proprietary drivers are in use on this system," followed by a list consisting of just one item: "NVIDIA accelerated graphics driver (latest cards)" and an indication that that driver was "Not in use." So, OK, what should I do about that? The truly wise thing, I decided, would be, first of all, to reboot the computer and make a backup of Ubuntu installation as it now stood. The backup tool I had chosen for this purpose was Acronis True Image 11 Home, so I stuck in that CD and did the deed. Acronis had succeeded for me in one recent use and failed in another, so I was not sure I wanted to rely solely on it. But then I thought maybe I hadn't chosen the disk image option for the second one, so I let it slide and moved on with the NVIDIA project. I decided that the driver listed in Hardware Drivers might not be the right one for the new video card. But as I was reading the endless gyrations that I had gone through in one of my first stabs at this problem, I realized that I wasn't exactly sure when I had reinstalled Ubuntu or what screwed-up situation might be left over from that. I decided the best policy might just be the most straightforward one, so I went into Hardware Drivers and clicked to enable the latest NVIDIA driver. It needed a computer restart, so we did that. On reboot, I went back into Hardware Drivers and now it said the drivers were in use. Encouraged, I tried running Second Life again. And, you know, it worked. So changing video cards may have been the answer to the whole graphics hassle. Rebooting also seemed to install the Compiz feature described in the previous post; I was now able to rotate between Ubuntu desktops by rolling the mouse wheel on both computers. The graphics card issue had also been a problem in my previous attempts to install Google Earth and to use multiple monitors in VMware Workstation. But before I could see whether the new card would resolve those issues, I needed to fix something else. I was doing some of my actual work on the primary computer, and I needed to gain access to a university computer via Virtual Private Network (VPN) on the secondary computer. That is, I wanted to be viewing some PDF articles on the right-hand monitor, which I'll call monitor no. 2, which was connected to the secondary computer, while writing about those articles on the left-hand monitor, no. 1, which was connected to the primary computer. This was not the only arrangement I could have used for this task. I could have expanded the Windows desktop, in the virtual machine on monitor 1, to extend to monitor 2 as well. I was running the second monitor through a KVM switch (separate from the KVM switch used for mouse and keyboard), and could thus have just punched the button to bring both monitors within the primary computer's desktop; and then I could have used a web browser within the VM to access the university website via VPN. I had already configured VPN in the VM, so that part would have been easy. The problem with this approach was that it would perpetuate my dependence on Windows. I was not sure whether it would be better from a security perspective, since I did not know the exact security status of Windows within VMware Workstation. It would also have its own complexities, insofar as I had not actually set up the multiple monitor arrangement yet. And in any case, I wanted both of my computers to have as much capability as possible, so that I could sail right on with productive work if one of them was down, going through some maintenance process, or otherwise preoccupied. Another thing I had to take care of was to update the firmware on my router. I had been meaning to get around to it since I had first installed it, because I had the general impression that updated firmware was important in making sure the router functioned effectively as a hardware firewall. I went to the firmware download page for the Linksys WRT54GL router and downloaded the latest firmware. It landed on my desktop. I unzipped it. This gave me a .bin file. Then I looked at the installation instructions. They led me to the page on how to access the router's web-based setup page. For that, I needed the router's IP address, so I went to the instructions for finding that, but they were for Windows. I found some advice that said to just type ifconfig in Terminal. That gave me several numbers, but it sounded like I was supposed to go with the value under eth0. That one said "inet addr:192.168.2.101." I entered that into the address bar in Firefox and got "Unable to connect: Firefox can't establish a connection to the server." I went back and looked at the advice again. They said the router would end in either 0 or 1. So I tried 192.168.2.0. Unable to connect. But 192.168.2.1 got me a login dialog. Now I had to remember my router ID and password. But I couldn't. I found some advice that said the default was to enter no username at all, and enter "admin" as the password; and if that failed, use the reset button. But that worked, and I was in. Before anything else, I went to Administration, there in the router's internal website, and changed the password. I went to the Wireless and Security settings and made sure they were saved the way I wanted. Then, under Status, I saw that the firmware version was 4.30.11. The one I had downloaded was 4.30.12, and it had a newer date. I looked at the release notes for my version and saw that they had apparently updated some security stuff, so I decided I had better go ahead with the firmware update. The next thing Linksys advised was to back up the router's settings. To do this, in the router's webpage, I went to Administration > Config Management > Backup Configuration > Backup > Save File. That went to my desktop. The next step was Administration > Firmware Upgrade > Browse. I browsed to the desktop and indicated the .bin file that I had unzipped from the firmware download. Then I clicked Upgrade. This gave me Upgrade Is Successful. To complete the upgrade, the instructions said, I had to close my browser and reset the router by pressing its Reset button (on back) for 30 seconds. But another way to reset, they said, was to go into the router's webpage and choose Administration > Factory Defaults > Restore Factory Defaults > Yes > Save Settings. That sounded easier, so I tried that. It said I would be returned to the previous webpage after several seconds, but instead I got "Unable to connect." It seemed to be trying to reach the router at 192.168.1.1. I typed 192.168.2.1 in the browser's address bar. That didn't work either, nor did 192.168.2.0 or 192.168.2.101. The inet addr value was still the same in ifconfig in Terminal. I was now unable to save these notes on Blogger or to reach Hotmail. I tried pressing the router's reset button for 30 seconds. Still unable to connect to any of those 192.168 numbers. Of course, I was now also unable to reach any Linksys or other webpages for instructions. As I recalled, there was another option for resetting, which involved (if I wasn't mistaken) holding in the router's reset button for 30 seconds while unplugging and replugging the router's power. Before doing that, it occurred to me to try to connect using the other computer. But no, it wasn't just the computer; neither of them could connect now. I tried that alternate reset approach. No joy. The router lights were flashing; I just didn't have any use of the thing. I unplugged its power and its Internet connection and plugged this secondary computer directly into the DSL modem. Now I could connect. I went back to the notes I had kept two months earlier, when I had previously wrestled with this same problem. But it looked like I had done everything I had written there. I wondered if maybe I had to restart Firefox or the computer to make it work. Before restarting the computer, I tried accessing the router using Opera. To do this, as I quickly discovered, I would have to plug the router back in and reconnect it. But Opera got nowhere on this. Next, I tried leaving the secondary computer connected to the router, disconnecting the router from the modem, and plugging the primary computer directly into the modem. Now the primary computer was online. In Firefox on the secondary computer, I typed http://192.168.1.1. Now it worked! I got the password screen, entered "admin" as the default password, and I was in. The Linksys instructions now told me to restore the router's settings by going, on the router's internal webpage, to Administration > Config Management > Restore Configuration > Browse. I browsed to the configuration file I had saved on my desktop. They called it a Config.bin file, but Ubuntu had saved it as WRT54GLV1_v4.30.11.cfg. I selected that and clicked Restore and got Upgrade Is Successful. I connected the computers back to the router and the router back to the modem. Now I got "Unable to connect" when I tried 192.168.1.0 or 1.1 or 2.0 or 2.1 on the secondary computer. It also could not access Hotmail. But the primary computer was now able to go online with no problem. I restarted Firefox on the secondary computer. Firefox didn't seem to want to restart right away, so I rebooted the computer while I was at it. That solved it. I got back into the router at 192.168.2.1. Apparently the restored configuration had restored that as its address; not sure why Firefox wasn't able to go to it without restarting/rebooting. I checked the settings I had recently changed, there in the router's internal webpage, and everything looked good. I clicked Save Settings just to be sure, re-entered my new password once more for the road, got back in, and killed that webpage. I was able to browse the Internet once again on both computers. Firmware upgrade completed. I had used dvgrab to capture video from my video camera in Ubuntu. It had converted a 40-minute (or so) tape into ten .avi files, each 1GB in size. After fooling with Ubuntu alternatives, I had decided that I would just continue to do my video editing in Adobe Premiere Elements (APE) in WinXP for the time being. I had found that APE did not run smoothly in VMware, so I decided I would do this video editing in a native WinXP dual boot. I decided to do this on the secondary computer, so as not to interrupt my work in VMware on the primary computer. (I had set up five different virtual machines, each for a different project, and I generally kept four of them suspended at any one time, with Windows Explorer and other programs and documents all set up, ready to go back to work on that project whenever I had the time or other materials or information I needed to advance that project a bit further.) So now it came time to edit some video on the secondary computer, dual-booted into WinXP. But -- what's this? All I could get, from those ten .avi files, was static. The audio was pretty much OK, except for a light, rapid clicking sound, but the video was totally screwed up. It was just static, like you would get if you tuned into a TV station that was off the air. My first guess was that dvgrab had produced the wrong kind of .avi file. I used VirtualDub to patch together those 1GB AVIs into two larger files, and then tried importing the resulting .avi files into APE. But the audio was missing on the first one. So I tried again, converting each .avi separately, without combining. Audio was good for each of these individually. I noticed that the new AVIs were now about 8GB each, replacing the 1GB .avi files that dvgrab had produced. Now the audio was OK and I was able to work with them. This doesn't mean that native Windows was any better-behaved than before. I could run Ubuntu endlessly on the secondary computer without a problem, whereas Windows kept shutting down the system at random. Ah, the bad old days of Windows ... Had a new problem. I was trying to watch videos, from YouTube and elsewhere, on the primary computer. Firefox wasn't playing them because I didn't have the Flash player installed and, at last time I had tried, I could not get it to work. So I had resorted to watching them in Internet Explorer, within a VMware virtual machine. But now I would sometimes get an error message from VMware:

Failed to open sound device /dev/ dsp: Device or resource busy Sound will not be available

According to PinoyTux Weblog, "this means the sound device in your host machine is currently in use and the VMware client is not able to access the device." The solution recommended there was to type "killall esd" in Terminal and then maybe restart VMware. I didn't want to go through the time and hassle of shutting down and restarting my several open virtual machines if I didn't have to. I was running a couple of programs in my host Ubuntu operating system, so I tried closing them down, one at a time, and then checking current status in VMware Workstation. The first program I shut down was Second Life, and that did it. As soon as Second Life was off, I could watch and hear videos in Internet Explorer with no problem. I restarted Second Life while a video was playing and still no problem. The problem seemed to be solved. Next. I had a bunch of MP3s on the secondary computer. I wanted something like Winamp to play songs at random (i.e., "shuffle play") from a songlist or playlist. I found a thread that said Audacious was the closest thing to it for Ubuntu Hardy 8.04. The Audacious downloads webpage said version 1.5.1 was the current stable release. I went to System > Administration > Synaptic Package Manager, there on the secondary computer, and searched for Audacious. It was there. I installed it. It appeared under Ubuntu's Applications > Sound & Video. I ran it. It was like other audio players I had seen: nice, streamlined interface with no explanation of controls or options, to be learned on your own time. It didn't appear to be able to recurse subdirectories (i.e., to look for MP3s, not only in the immediately indicated folder, but also in subfolders under that one). So I would have to manually load my 15 or 20 different music folders if I wanted all those songs to be available, which I did. Oh, well. At least I now had a music player. I belatedly remembered that I had already had one, in the form of RhythmBox Music Player. But I couldn't get it to work like I wanted either. This was another one of those things I would have to try to look into later. Next, I had a problem on the secondary computer. For some reason, Second Life kept causing it to completely shut down. I suspected this was a hardware rather than software issue, because it happened when I ran Second Life in Ubuntu too. I moved Second Life to the primary computer and ran it in Ubuntu without a problem. But then the secondary computer also started crashing when I was trying to create MPG files in Adobe Premiere Elements in Windows XP. It didn't crash when I tried to save video as AVI; it just happened when I tried to save video as MPG. Odd, but it happened enough to be predictable. It also happened when I reinstalled an older image of my programs drive and installed and ran Premiere Elements on that. I didn't want to have to reinstall Windows if it wasn't necessary. But then I recalled that WinXP had been kind of flaky on that machine anyway, and I thought I might as well take the opportunity to do a new installation, installing only the essential programs, to see if I couldn't make that system more stable. For starters, I used Acronis True Image to make a backup of my Ubuntu installation. I did that because I had installed Ubuntu on the same drive as Windows, and now, when I tried to reinstall Windows using the Windows XP CD, the installer basically indicated that it couldn't create a WinXP partition on that drive. So I wanted to move everything off that drive, if I could, and start over. I then remembered that it was an older drive, and wondered whether that might possibly be part of the problem. I connected another drive to that machine, but when I tried to boot it, I got "Error 17" from GRUB. From what I gathered, this error would happen when GRUB would perceive your drives in the wrong order, and the best solution (other than just unplugging that additional drive) was to edit the GRUB configuration to make it right. I found what looked like a really good explanation along those lines. But rather than spend the time to work through it, I booted with the Ubuntu CD and took a look at what was on that third (newly added) drive. There were several partitions, but it looked like I had emptied them out (as I would have expected) before setting that drive aside. So I ran the Ubuntu CD's System > Administration > Partition Editor and deleted those empty partitions. Then I rebooted. This time, I got GRUB Error 22. This evidently resulted from GRUB looking for a partition that had been there and wasn't anymore. I found a thread that recommended either (a) running Recovery Console (i.e., boot with the WinXP CD and choose repair) and selecting FIXBOOT and FIXMBR, or (b) using Super Grub Disk to edit GRUB. I was thinking I still had another step to go anyway before worrying about that, and that was to restore the Ubuntu backup to a now-empty partition on the second hard drive, so that I'd have my Windows program files on one drive and my Ubuntu files on another, and I could then reinstall Windows from scratch on that first drive. After wiping out the Ubuntu partitions on that Windows drive, though, I was still getting the same error message from the WinXP installation CD:

Windows XP Professional Setup To install Windows XP on the partition you selected, Setup must write some startup files to the following disk: 305243 MB Disk 0 at Id 1 on bus 0 on atapi [MBR] However, this disk does not contain a Windows XP-compatible partition.

It occurred to me that maybe this was a problem that could be fixed by FIXBOOT or FIXMBR. I rebooted the WinXP CD again and ran Recovery Console. There, I ran FIXBOOT, but it said, "FIXBOOT cannot find the system drive, or the drive specified is not valid." That made sense, since I had deleted the Windows program partition in preparation for reinstallation. I ran FIXMBR, though, and then rebooted with the WinXP CD and tried again to do a new WinXP installation. But I still got the same message: "does not contain a Windows XP-compatible partition." It was described as "New (Raw)" in the setup program, but for some reason WinXP could not install into it. So, OK, I rebooted with the Ubuntu CD and moved all of the data from the other partitions to the other drive, so I could wipe the drive clean and start over. But even then, I still got the no-compatible-partition error. I finally unplugged the other drives and rebooted with the WinXP CD. I was then able to install Windows XP, and when that was done, I shut down, plugged in the other drives, and rebooted to continue configuring the system. The remaining question was whether Adobe Premiere Elements (APE) or Second Life would cause this new Windows installation to crash. I installed APE and tried again to make an MPG from an AVI. Previously, its problem, which occurred every time I tried, was that it would crash during the second pass while creating a two-pass VBR MPEG. This time, it sailed right through, successfully creating several different MPGs with exactly the same settings. But then it crashed on the second pass of the fourth MPG. Shut the computer completely off, just like before. Surely, this was a hardware problem. But which hardware? Nothing seemed to be overheating, and there did not seem to be anything else unusual going on on the computer. My guess was the power supply, but I didn't have time or inclination at this moment to shut down both computers, swap out, and fool around until maybe I could make it crash again. APE tried again to make that same MPG, and on its second pass it ran just fine. Hmm. This made me think that the problem may have been a program conflict between APE and something else. I'd had a number of programs open previously, but could not remember specifically which ones. It wasn't the screen saver -- I had set that to "None." Power settings? Maybe. It was set to shut down the monitor after 20 minutes; but when that kicked on, the computer now kept running. Anyway, now that I was on the lookout for the possibility of a program conflict, maybe I would be able to home in on it in the future. Meanwhile, with nothing else running, it completed the MPGs just fine. I ran the same five two-pass MPGs to test it, and there were no further problems. So that part of my dual-boot scheme still seemed to be working. Once I had WinXP reinstalled on the secondary computer, I went into Computer Management. I think Computer Management may not show up as an option in Windows. I forget how I discovered it. It was available by double-clicking on compmgmt.msc in C:\WINDOWS\system32. For this and other enduring items, I kept a copy of my Start menu (right-click Start and select Explore All Users) on a separate drive, so that I would not lose the links if I reinstalled Windows. Anyway, in Computer Management, I selected Disk Management and selected the first drive on the list. I right-clicked on it and chose Properties > Tools > Error-checking > Check Now > Automatically fix file system errors > Start. I did not check "Scan for and attempt recovery of bad sectors," because doing that would make it run immediately. After checking Start, I okayed through the rest of the messages. The basic idea was that it would run its check when I rebooted. I did this for each drive and then rebooted. This version of CHKDSK ran on each drive. When it got to the external USB hard drive, however, it started giving me this message: "Deleting an index entry from index $0 of file 25." It kept giving me those messages. I went to bed and, when I got up the next morning, it was still doing it. It went on for hours. I Googled this and saw a thread in which someone concluded that Acronis Disk Director or Acronis True Image had caused the problem. In their case, the series of "Deleting an index entry" messages concluded with some messages that prompted them to think the drive was corrupt. In my case, for some bizarre reason my battery backup suddenly shut off -- no reason in the world, out of the blue, just up and died -- so the experiment was terminated. I switched the external backup hard drive to be connected to the primary computer. When I turned the computers back on, though, Ubuntu was not able to mount that drive. I rebooted the primary computer into WinXP and set up each drive there to be checked via Computer Management. On reboot, it started giving me that same "Deleting an index entry from index $0 of file 25" message, but this time it seemed to be doing it during the testing of an internal drive, not the external drive (though maybe it would have done it there too, later; maybe it had only gotten as far as one of the internal drives). I couldn't tell for sure because I wasn't paying attention when it indicated which drive it would be checking next. It kept giving me that message for an hour, at which point I put in the Windows XP CD, hit the computer's Reset button, went into Recovery Console, and ran chkdsk /r on each partition that Windows could see. I wasn't sure what happens to a Linux partition when your computer is rudely shut down by a failure in your battery backup. After I finished running chkdsk /r (repeatedly, in each case where errors were reported), I ran a manufacturer's diagnostic CD to check all the drives that way too. Or at least I tried to. The Seagate drive passed the SeaTools diagnostic, but the Samsung drives weren't even recognized by the Samsung diagnostic CD that I had from the previous year, and SeaTools crashed when I tried to use it to test the Samsung drives. I downloaded and burned a CD with what appeared to be the latest Samsung diagnostic program, but the program did not run. It gave me an "Abort from unhandled exception" error message. So anyway, I booted back into Windows and set Computer Management to check all the drives again, and rebooted. This time, everything went fine. Whew! End of encounter no. 32335 with Windows XP. I rebooted the primary machine back into Ubuntu and tried to pretend that nothing had happened. By this time, I had turned my attention to the secondary computer. I had reinstalled Windows XP, and that had wiped out my GRUB boot manager. So now I didn't get a choice of whether to boot into Windows or Ubuntu; it just went straight to Windows. This launched me onto a whole extended problemsolving adventure of its own. When it was done, though, I had GRUB and Ubuntu working again on the secondary computer. In case I had not previously said so, it seemed appropriate at this point to take a moment and acknowledge how much I appreciated Ubuntu and VMware for letting me freeze a project, or a state of the computer, and return to it later. In Ubuntu, I mean, the Hibernate option actually worked. I had had multiple problems with it in Windows, over the years, and had finally given up on it. And VMware, with its option of suspending a virtual machine, meant that I could get a VM set up to work on a project, and then suspend and resume it whenever I wanted to stop or start working on it again. Next project: Skype. I had used it in Windows, but never in Ubuntu. Now I looked around for a way to install it on the secondary computer. They said it wasn't available in any repository and, sure enough, I couldn't find it in System > Administration > Synaptic Package Manager. I went to the Skype download page. The Ubuntu download for Skype 2.0 for Linux was actually a Debian .deb file, so I retrieved my notes on how to install a Debian package. It seemed that all I had to do was to right-click on the .deb file and select "Open with GDebi Package Installer." But I couldn't figure out where the download had gone. It wasn't on my desktop or anywhere. Finally found it in the /tmp folder. When I opened it with GDebi, though, I got "Error: Wrong architecture 'i386'." Apparently Skype 2.0 for Linux was not going to run on my 64-bit Ubuntu. So, OK, forget that. I didn't want to have to reboot into Windows just to make calls, so I tried installing Skype for Windows in a VMware virtual machine. It seemed to install fine, but then it turned out my headset was not getting a response from the microphone port. I had had problems like that before. I put a USB headset on my Newegg wishlist and put this issue on hold. Next: backup. I had wrestled with this repeatedly during the Ubuntu transition process, and had decided to try to get rsnapshot to work. It was a hassle to configure rsnapshot, but I could also see that the person who had created it had potentially saved users from a lot of work in trying to configure rsync itself. Rsync just had too many options. Rsnapshot took care of most of the options that would be irrelevant for a user like me, and the comments were generally clear enough. Anyway, following the advice in rsnapshot.conf, I also edited /etc/cron.d/rsnapshot so that rsnapshot would run at regular intervals. I had seen a lot of references to cron as the program that would let you schedule things, but this was my first involvement with it. But now, by the time I resumed writing at this point in this post, my working situation had changed, and I had no time or patience to learn how to write scripts or edit .conf files to make rsnapshot work. So I looked at the several webpages I had opened for rsnapshot advice -- a Debian page, a BackupCentral Wiki page, an rsnapshot page, and a Google search for good measure -- and I shut 'em down. I looked at an sBackup instruction page and decided to give sBackup another try. I searched for it in Synaptic and noticed there was also a program called backuppc, and that one (unlike sbackup) had an Ubuntu logo next to it. I did a Google search for backuppc and found the Ubuntu documentation, but that didn't have quite what I was looking for. I did another search and still didn't quickly see a comparison of the two. Having already fiddled with sbackup a couple of times, I installed backuppc. Someone had said it was better for backing up multiple drives. When I marked backuppc for installation, Synaptic said I also had to install apache2 and a boatload of other programs. I said OK. As it was installing, it gave me some apache options, but the Help said that only plain old apache (not apache2 etc.) supported automatic something-or-other. So I went with that. Next, it said I could manage backuppc through its web-based interface at http://P4-VM/backuppc, where P4-VM was the name of my computer. It gave me a web user account named backuppc and a password to access that interface. Then the installation was done. But where was backuppc? I was searching through the Ubuntu menu, doing "whereis backuppc," rooting around in folders, and then I said, Duh. So I went to the web-based interface page just mentioned and there it was. Not. I got "Not Found." Screw it. Back to Synaptic to install sbackup. I installed it, set it up to Include or Exclude nothing except the one data drive I wanted to back up, and ran it. It said a backup was being initiated in the background. A minute or two later, the light on the external (target) drive started to flicker, and we were off and running (I hoped). Later, I checked out the results. Sbackup had created a folder called 2008-09-29_19.40.12.438074.P4-VM.ful. That appeared to be the date, the time, and the name of the computer, with a .ful extension. In that folder, there were five files. The main one was files.tgz, which (at 31.5GB) apparently contained old copies of the files that had changed. I was not able to tell quickly what files were in that backup; the flist file was empty. Otherwise, sbackup appeared to have preserved the copies of my data folders that I had previously copied to the external backup drive. But not entirely: a comparison of properties for the data drive and these backup folders indicated that the backup was missing almost 600 files that were on the original data drive. Were these files somehow represented in that .ful backup folder, or were they just missing? I did not know of any way to tell for sure. I realized, at this point, that I did not wish to go back to a black-box kind of backup system, where I would just assume that the backup program was doing it right. I had been burned by those in the past. What I wanted was exactly what I had gotten with Second Copy 2000 in Windows: a simple mirroring system that would back up my data drive periodically. I didn't want a RAID arrangement where I would have instant mirroring; I wanted to have the luxury to retrieve a file as it had existed a half-hour or an hour before. I had set up Second Copy to do the backup every several hours, so chances were that it would not yet have replaced the previous version with a newer one. The backup had been a lifesaver on a number of occasions that way. With Second Copy 2000, I could do a simple comparison of properties to verify that what was on the backup matched what was on the original data drive. I posted a question about this in an Ubuntu community forum. What I got back was just some advice to use rsync and cron. By this point, I was running out of energy and time for fixing Ubuntu and VMware. I had a working system with a number of things to tackle at some point in the future. I decided it was time to wrap up this 10-week effort. The summary appears in a follow-up post.