Sorting and Manipulating a Long Text List to Eliminate Some Files

In Windows XP, I made a listing of all of the files on a hard drive.  For that, I could have typed DIR *.* > OUTPUTFILE.TXT, but instead I used PrintFolders.  I selected the option for full pathnames, so each line in the file list was like this:  D:\FOLDER\SUBFOLDER\FILENAME.EXT, along with date and file size information.

I wanted to sort the lines in this file list alphabetically.  They already were sorted that way, but DIR and PrintFolders tended to insert blank lines and other lines (e.g., "=======" divider lines) that I didn't want in my final list.  The question was, how could I do that sort?  I tried the SORT command built into WinXP, but it seemed my list was too long.  I tried importing OUTPUTFILE.TXT into Excel, but it had more than 65,536 lines, so Excel couldn't handle it.  It gave me a "File not loaded completely" message.  I tried importing it into Microsoft Access, but it ended with this:

Import Text Wizard

Finished importing file 'D:\FOLDER\SUBFOLDER\OUTPUTFILE.TXT to table 'OUTPUTTXT'.  Not all of your data was successfully imported.  Error descriptions with associated row numbers of bad records can be found in the Microsoft Office Access table 'OUTPUTFILE.TXT'.

And then it turned out that it hadn't actually imported anything.  At this point, I didn't check the error log.  I looked for freeware file sorting utilities, but everything was shareware.  I was only planning to do this once, and didn't want to spend $30 for the privilege.  I did download and try one shareware program called Sort Text Lists Alphabetically Software (price $29.99), but it hung, probably because my text file had too many lines.  After 45 minutes or so, I killed it.

Eventually, I found I was able to do the sort very quickly using the SORT command in Ubuntu.  (I was running WinXP inside a VMware virtual machine on Ubuntu 9.04, so switching back and forth between the operating systems was just a matter of a click.)  The sort command I used was like this:

sort -b -d -f -i -o SORTEDFILE.TXT INPUTFILE.TXT

That worked.  I edited SORTEDFILE.TXT using Ubuntu's GEDIT program (like WinXP's Notepad).  For some reason, PrintFolders (or something) had inserted a lot of lines that did not match the expected pattern of D:\FOLDER\SUBFOLDER\FILENAME.EXT.  These may have been shortcuts or something.  Anyway, I removed them, so everything in SORTEDFILE.TXT matched the pattern.

Now I wanted to parse the lines.  My purpose in doing the file list and analysis was to see if I had any files that had the same names but different extensions.  I suspected, in particular, that I had converted some .doc and .jpg files to .pdf and had forgotten to zip or delete the original .doc and .jpg files.  So I wanted to get just the file names, without extensions, and line them up.  But how?  Access and Excel still couldn't handle the list.

This time around, I took a look at the Access error log mentioned in its error message (above).  The error, in every case, was "Field Truncation."  According to a Microsoft troubleshooting page, truncation was occurring because some of the lines in my text file contained more than 255 characters, which was the maximum Access could handle.  I tried importing into Access again, but this time I chose the Fixed Width option rather than Delimited.  It only went as far as 111 characters, so I just removed all delimiting lines in the Import Text Wizard and clicked Finish.  That didn't give me any errors, but it still truncated the lines.  Instead of File > Get External Data > Import, I tried Access's File > Open command.  Same result.

I probably could have worked through that problem in Access, but I had not planned to invest so much time in this project, and anyway I still wasn't sure how I was going to use Access to remove file extensions and folder paths so that I would just have filenames to compare.  I generally used Excel rather than Access for that kind of manipulation.  So I considered dividing up my text list into several smaller text files, each of which would be small enough for Excel to handle.  I'd probably have done that manually, by cutting and pasting, since I assumed that a file splitter program would give me files that Excel wouldn't recognize.  Also, to compare the file names in one subfile against the file names in another subfile would probably require some kind of lookup function.

That sounded like a mess, so instead I tried going at the problem from the other end.  I did another directory listing, this time looking only for PDFs.  I set the file filter to *.pdf in PrintFolders.  I still couldn't fit the result into Excel, so I did the Ubuntu SORT again, this time using a slightly more economical format:

sort -bdfio OUTPUTFILE.TXT INPUTFILE.TXT

This time, I belatedly noticed that PrintFolders and/or I had somehow introduced lots of duplicate lines, which would do much to explain why I had so many more files than I would have expected.  As advised, I used another Ubuntu command:

sort OUTPUTFILE.TXT | uniq -u

to remove duplicate lines.  But this did not seem to make any difference.  Regardless, after I had cleaned out the junk lines from OUTPUTFILE.TXT, it did all fit into Excel, with room to spare.  My import was giving me lots of #NAME? errors, because Excel was splitting rows in such a way that characters like "-" (which is supposed to be a mathematical operator) were the first characters in some rows, but were followed by letters rather than numbers, which did not compute.  (This would happen if e.g., the split came at the wrong place in a file named "TUESDAY--10AM.PDF."  So when running the Text Import Wizard, I had to designate each column as a Text column, not General.

I then used Excel text functions (e.g., MID and FIND) on each line, to isolate the filenames without pathnames or extensions.  I used Excel's text concatenations functions to work up a separate DIR command for each file I wanted to find.  In other words, I began with something like this:

D:\FOLDER\SUBFOLDER\FILENAME.EXT

and I ended with something like this:

DIR "FILE NAME."* /b/s/w >> OUTPUT.TXT

The quotes were necessary because some file names have spaces in them, which confuses the DIR command.  I forget what the /b and other options were about, but basically they made the output look the way I wanted.  The >> told the command to put the results in a file called OUTPUT.TXT.  If I had used just one > sign then that would have meant I wanted OUTPUT.TXT to be recreated every time a match was found.  Using two >> signs was an indication that OUTPUT.TXT should be created if it does not yet exist, but otherwise the results of the command should just be appended to whatever is already in OUTPUT.TXT.

In cooking up the final batch commands, I would have been helped by the MCONCAT function in the Morefunc add-in, but I didn't know about it yet.  I did use Morefunc's TEXTREVERSE function in this process, but I found that it would crash Excel when the string it was reversing was longer than 128 characters.  Following other advice, I used Excel's SUBSTITUTE command instead.

I took the thousands of resulting commands (such as the DIR FILENAME.* >> OUTPUT.TXT shown above), one for each file type (e.g., FILE NAME.*) that I was looking for, into a DOS batch file (i.e., a text file created in Notepad, with a .bat extension, saved in ANSI format) and ran it.  It began finding files (e.g., FILENAME.DOC, FILENAME.JPG) and listing them in OUTPUT.TXT.  Unfortunately, this thing was running very slowly.  Part of the slowness, I thought, was due to the generally slower performance of programs running inside a virtual machine.  So I thought I'd try my hand at creating an equivalent shell script in Ubuntu.  After several false starts, I settled on the FIND command.  I got some help from the Find page in O'Reilly's Linux Command Directory, but also found some useful tips in Pollock's Find tutorial.  It looked like I could recreate the DOS batch commands, like the example shown above, in this format:

find -name "FILE NAME.*" 2>/dev/null | tee -a found.txt

The "-name" part instructed FIND to find the name of the file.  There were three options for the > command, called a redirect:  1> would have sent the desired output to the null device (i.e., to nowhere, so that it would not be visible or saved anywhere), which was not what I wanted; 2> sent error messages (which I would get because the lost+found folder was producing them every time the FIND command tried to search that folder) to the null device instead; and &> would have sent both the standard output and the error messages to the same place, whatever I designated.  Then the pipe ("|") said to send everything else (i.e., the standard output) to TEE.  TEE would T the standard output; that is, it would send it to two places, namely, to the screen (so that I could see what was happening) and also to a file called found.txt.  The -a option served the same function as the DOS >> redirection, which is also available in Ubuntu's BASH script language, which is what I was using here:  that is, -a appended the desired output to the already existing found.txt, or created it if it was not yet existing.  I generated all of these commands in Excel - one for each FILE NAME.* - and saved them to a Notepad file, as before.  Then, in Ubuntu, I made the script executable by typing "chmod +x " at the BASH prompt and then ran it by typing "./" and it ran.  And the trouble proved to be worthwhile:  instead of needing a month to complete the job (which was what I had calculated for the snail's-pace search underway in the WinXP virtual machine), it looked like it could be done in a few hours.

And so it was.  I actually ran it on two different partitions, to be sure I had caught all duplicates.  Being cautious, I had the two partitions' results output to two separate .txt files, and then I merged them with the concatenation command:  "cat *.txt > full.lst."  (I used a different extension because I wasn't sure whether cat would try to combine the output file back into itself.  I think I've had problems with that in DOS.)  Then I renamed full.lst to be Found.txt, and made a backup copy of it.

I wanted to save the commands and text files I had accumulated so far, until I knew I wouldn't need them anymore, so I zipped them using the right-click context menu in Nautilus.  It didn't give me an option to simultaneously combine and delete the originals.


Next, I needed to remove duplicate lines from Found.txt.  I now understood that the command I had used earlier (above) had failed to specify where the output should go.  So I tried again:

sort Found.txt | uniq -u >> Sorted.txt

This produced a dramatically shorter file - 1/6 the size of Found.txt.  Had there really been that many duplicates?  I wanted to try sorting again, this time by filename.  But, of course, the entries in Sorted.txt included both folder and file names, like this:

./FOLDER1/File1.pdf
./SomeotherFolder/AAAA.doc

Sorting normally would put them in the order shown, but sorting by their ending letters would put the .doc files before the .pdfs, and would also alphabetize the .doc files by filename before foldername.  Sorting them in this way would show me how many copies of a given file there had been, so that I could eyeball the possibility that the ultimate list of unique files would really be so much shorter than reported in Found.txt.  I didn't know how to do that in bash, so I posted a question on it.


Meanwhile, I found that the output of the whole Found.txt file would fit into Excel.  When I sorted it, I found that each line was duplicated - but only once.  So plainly I had done something wrong in arriving at Sorted.txt.  From this point, I basically did the rest of the project in Excel, though it belatedly appeared that there were some workable answers in response to my post.

Ubuntu: Schedule Items with Cron

I wanted to schedule regular backups in Ubuntu 9.10.  I had already worked out the rsync commands I wanted to use; now it was a matter of running them automatically at certain times or on certain days.  I began by seeing what was already scheduled in my crontab (i.e., my chronological table).  Actually, I had two of them:  one for me, and one for the root (i.e., administrator).  I checked them with "crontab -l" (that's a small L) and "sudo crontab -l" and both say "no crontab."  This supposedly meant that there were no crontab files in /var/spool/cron/crontabs.  I verified that via "sudo nautilus."  It seemed like that would apply to the root's cron, but I wasn't able to find any different location where the user's cron should be, so I just moved on to the next step.

The next step was to edit the crontab by using "crontab -e."  This seemed to be creating a crontab for me, as distinct from root:  it said "no crontab for ray - using an empty one."  To confirm that, I tried in a separate Terminal session with "sudo crontab -e."  It seemed to flash the same choice as had appeared for me, showing a choice of editors; but then it went directly into nano, which the other Terminal session was describing as the "easiest" of the three available editors.  So, OK, since I ordinarily ran my rsync backups as me, user, not as root, I figured I would want to set up my own crontab, not a root crontab.  (Later, I found some statements that it was a bad idea to edit root's crontab.)  So I killed that nano session and went back to the first Terminal session.  There, I chose no. 3, nano, as my editor.

The top of the nano screen was showing me "# m h  dom mon dow   command."  This was my cue for the things that I needed to enter on a line, in order to schedule a cron job:  minute, hour, day of month, month, day of week, and command.  (The leading # was to indicate that this sample line was just a comment and should not be executed.)

According to About.com, up through the week level, permissible values began at zero:  that is, minutes of the hour ran from 0 to 59, hours of the day ran from 0 to 23, and days of the week ran from 0 to 7 (where Sunday was both 0 and 7, as you prefer).  Beyond that, days of the month ran from 1 to 31, and months of the year ran from 1 to 12.  If they had names, you could use their first three letters (e.g., "Mon" and "Jul" but not "minute 23"). Cron uses the union (not the intersection) of the two day commands.  That is, if you specify a day of the week (e.g., Fri) and also a day of the month (e.g., 15), the command will run on both days (e.g., every Friday, and also the 15th of every month).  You could use an asterisk to indicate "every"; for example, * * * * * would indicate that you want to run the command every minute of every hour of every day of every month.

There were additional options for numbers below the date level; that is, these wouldn't work on days of the week or of the month.  One of these options was to use fractions:  for instance, */4 would mean "every fourth" minute or hour or whatever.  You could also use a range:  40-45 would mean it should run every minute of the hour from 40 to 45 minutes (i.e., 12:40 AM, 12:41 AM, 12:42 AM . . . 12:45 AM, 1:40 AM . . . 11:45 PM).  You could use lists, separating items with commas, so that 40,41 would mean that it should run only on the 40th and 41st minutes of the hour. So designating the hour as 0-11/2 would mean that it should run every other hour, in the morning only.  You could use a list of ranges; for example, an hour designation of 0-1,10-11 would indicate that the command should run in the first two and also in the last two hours of the morning.  Range and list commands start on the first number; for instance, 2-6/2 runs at 2 AM, 4 AM, and 6 AM (at whatever minute you specify).

There was one other category of entry:  special words.  These words would replace all five numbers.  In other words, if you wanted the precise control offered by the numbers for minute, hour, etc., use the numbers; but if you want the convenience of just entering one word without having to think much about what it means, use the word.  These special words were @reboot (run at reboot), @yearly or @annually (once a year), @monthly, @weekly, @daily or @midnight (run once a day), and @hourly.  These all run as soon as the time period starts (e.g., January 1, 12:00 midnight).  There was more to know about these commands, in the official documentation.

I decided the first thing to schedule was a backup of my CURRENT partition to a backup internal hard drive.  I wanted this backup to run several times a day.  I hadn't set it up as a RAID array because I didn't want it to happen immediately; I wanted to allow some time in case I accidentally deleted something, or made some other stupid mistake.  The more frequently it ran, the more likely it would contain the most recent version of the relevant folder - which could mean it would be more likely to have the version that existed *after* my stupid mistake.  My compromise was to set it up to run every two hours.  The cron line I used, then, was this:

0   */2   *   *   *   [rsync command]

In crontab, I put several spaces between the numbers for readability.  Here, in this blog posting, I had to use nonbreaking spaces for the display shown above, because plain old spaces tend to get ignored in HTML.  I haven't reproduced, here, the long command that I want executed, because I want to focus on the cron parts of the line.  (The rsync command of choice is shown in the previous post.)

Then it occurred to me that, instead of putting that long command in cron, where I would have to do some minor translation every time I wondered what it meant, I could probably write a basic Ubuntu shell script that would do the same thing and would allow me to add explanatory notes and other commands.  So I took a brief detour into the land of scripts.  By the time I returned and finished my look at rsync, I had two scripts.  One was called backup-hour.sh, to be run every few hours.  The other was called backup-day.sh.  I put them into /home/ray/bin and wrote the following cron lines for them:

0   */2   *   *   *   ./home/ray/bin/backup-hour.sh

0   2   *   *   *   ./home/ray/bin/backup-day.sh

The first one would hopefully run backup-hour.sh every two hours, all day and all night.  The second one was intended to run backup-day.sh every day at 2 AM.

To put these lines into crontab, I typed crontab -e.  It all looked good.  But nothing was happening.  A couple of days went by, and cron didn't run.  The problem, I suspected, was with those periods I had put at the start of my path names.  I had thought that was part of a command, but nobody else was using them in their cron files.  So I deleted those and waited until the next even-numbered hour, to see if backup-hour.sh would run.

Then I wondered whether I was saving crontab in the right place.  I noticed that nano, my default crontab editor, was saving it to /tmp/crontab.zfItNF/crontab.  Somehow, that didn't look right.  I did a quick search and found a variety of theories on where crontab should be, and none of them involved the /tmp folder.  Someone suggested typing "which crontab" at the prompt.  That came back with /usr/bin/crontab, which wasn't one of the options those other people had suggested.  I tried to save this crontab to /usr/bin/crontab and got a message that the file already existed.  I tried "gedit /usr/bin/crontab," but even with sudo I got a message that the file could not be opened.  I decided to pass on that one for the time being and, selecting a seemingly knowledgeable opinion, I thought about saving it in /var/spool/cron.  It turned out that there was a subdirectory there, and more specifically we had a file called /var/spool/cron/crontabs/ray.  When I looked in that, I saw a copy of my crontab file, the one that nano had been trying to save in a /tmp folder, except that it began with the line, "DO NOT EDIT THIS FILE - edit the master and reinstall."  So it seemed that maybe nano knew what it was doing after all.  So I let nano save crontab in that /tmp folder after all.  Then, following some advice, I decided to output the error messages, if any, to a crontab.log file.  So the first complete line in my crontab looked like this:

0 */2 * * * /home/ray/bin/backup-hour.sh 2>> /Folder1/crontab-errors.log

Unfortunately, at 4:00 PM, nothing happened.  I decided to follow the suggestion that it is much easier to use the gnome-schedule package (though there were problems for those who upgraded from Ubuntu 9.04 to 9.10 rather than doing a fresh install), so I installed that in Synaptic.  This gave me a new option at Applications > System Tools > Scheduled Tasks.  (Scheduled Tasks was apparently a simple front end for cron.)  But before Scheduled Tasks would work, I had to make sure that crontab and a program called At were installed.  Both were marked as installed in Synaptic.  Cron was for recurrent tasks, and At was for one-time jobs (e.g., run this "At" startup).

The basic idea, it developed, was that crontab -e would add scheduling files to the /var/spool/cron/crontabs folder, but it was apparently advisable to just let crontab -e do that, and not try to find and edit those files directly.  I tried typing "sudo /etc/init.d/cron restart," but that gave me a suggestion:  "Rather than invoking init scripts through /etc/init.d, use the service(8) utility, e.g. service cron restart."  So, OK, I tried that.  This gave me a long message that started with "restart: Rejected send message."  That didn't sound good, so I tried "ps -eaf | grep cron" and that gave me these three lines:

ray       8381  8343  0 19:46 pts/0    00:00:00 man 5 crontab
root      8409     1  0 19:50 ?        00:00:00 cron
ray       8416  8343  0 19:52 pts/0    00:00:00 grep --color=auto cron

When I had tried running that command previously, I had gotten only the second and third lines, not the first.  So perhaps the problem had been that my cron had not been running when I had tried to run it previously, and now it was running.  On that assumption, I could have just gone back to the command-line approach at this point, but I liked the added option of using At to schedule one-time events. But as I checked further, "crontab -l" said "no crontab for ray," so apparently crontab was *still* not running.  But no, one source said, "The output: no crontab for [username] means crontab is installed."

I found a long thread that told me I could use gedit instead of nano, to edit crontab, by typing this:

export EDITOR=gedit && crontab -e

(The "&&" part combined separate commands; apparently I could have achieved the same thing by typing these two on separate lines.)  Someone said I could make this permanent by putting the command in my .bashrc file, which they said I would find in /home/ray, which was true.  But if .bashrc hadn't been there, apparently I could have used gedit to create it, with something like this:

# .bashrc - bash config file #
# export variables
export EDITOR=gedit

Since .bashrc was already there, I just added those last two lines to the end of it.  But anyway, back at the cron issue, someone in that long thread said I could change my cron line to look like this:

* * * * * export DISPLAY:=0 && xterm [command]

if I wanted to see something on the screen when the command was executing.  But this got me back to the realization that, for all of the flexibility I was seeing as I worked my way through page 9 of that very long thread, I would probably prefer, right now, to just get something working.  So I guessed that Scheduled Tasks would work just fine if plain old crontab was working.  So, as others had done, I wrote up a simple command to test crontab.  The entry looked like this:

* * * * * export DISPLAY:=0 && xterm dir

Sadly, this did nothing.  I opened Scheduled Tasks, thinking I would try something similar there, and instead I saw that my crontab line was already there, albeit in ugly form: 

Recurrent    At every minute     export DISPLAY:=0 &amp&amp dir

Anyway, that didn't seem to be running, so I deleted it, in Scheduled Tasks, and tried doing similar as a one-time task.  Here's what I ran:

dir > /home/ray/DIRDIRDIR

and it worked!  I got a text file named DIRDIRDIR that contained a directory listing.  So it seemed the one-time part of Scheduled Tasks was working properly.  So I returned to the question of recurrent tasks.  I remembered that, in Ubuntu, we use "ls" rather than "dir."  So in Scheduled Tasks, I used more or less the same command to append updated directory listings each minute, showing the time when DIRDIRDIR had been last updated:

ls -l >> /home/ray/DIRDIRDIR

and that worked too.  So now I felt I should try again with the lines I had attempted earlier, as revised.  This time, I entered them into Scheduled Tasks rather than into crontab, so I didn't need the * * * * * parts of the entries.  So here are the commands I entered in Scheduled Tasks:

/home/ray/bin/backup-hour.sh 2>> /Folder1/crontab-errors.log
/home/ray/bin/backup-day.sh 2>> /Folder1/crontab-errors.log

I didn't actually enter them both right away; I started with the first one, and made it run just once, at nine minutes past the hour (which was about two minutes ahead of when I was working on it).  When the time came, it ran, or it seemed to:  there was now a file named "crontab-errors.log" with 0 bytes in Folder1.  That was good enough for now.  I tried another line, this time telling Scheduled Tasks to make it an "X application" rather than "Default behavior."  That didn't seem to do anything, but whatever.  It looked I had what I needed, and I could add more knowledge later.

Basic Ubuntu (Bash) Shell Scripts

In Ubuntu 9.04, I wanted to write a script that would execute an rsync command, so that I could put a brief reference to the script into my crontab file, instead of putting the whole long rsync command there.

For a brief, tiny moment, I was almost tempted to consider learning the GAMBAS (Gambas Almost Means BASIC) programming language, just because (pre-Visual) BASIC was the only programming language I ever learned.  Instead, I moved toward basic instructions on writing a shell script.  Here's what I wrote:

#!/bin/bash
# This is backup-hour.sh
# It backs up CURRENT to CURRBACKUP every few hours
rsync -qhlEtrip --progress --delete-after --ignore-errors --force --exclude=/.Trash-1000/ --exclude=/lost+found/ /media/CURRENT/ /media/CURRBACKUP

As the instructions said, the first line was essential to tell the computer to use BASH to interpret the following lines.  The next two lines were comments, and the final line (wrapping over onto multiple lines here) was exactly the rsync line I'd been using to do the backup.  In other words, learning how to write the command was almost all I needed to write the script.

The next step was to save it somewhere.  I had previously heard, and the instructions said, that the common place to put it is in your bin folder.  I went with that, but made a note that I would need to be sure to back up my bin folder, because I didn't want to go to the trouble of writing all these scripts and then see them vanish.

The usual location for the user's bin folder is at /home/[username]/bin.  In my case, that's /home/ray/bin.  Getting there in Nautilus (i.e., Ubuntu's File Browser, also started by typing "nautilus" in Terminal) can be confusing:  you can get there via the Home Folder option (assuming you're showing Tree rather than Places or something else (or nothing) at the left side of the file browser), or you can go to File System/home/[username]/bin.  Same thing, either way.  So I saved the script (above) in my bin folder as backup-hour.sh.  That turned the comment lines (beginning with #) blue in gedit.

Next, the instructions said, I needed to set permissions.  This was a confusing aspect of Ubuntu.  The documentation seemed to say that there were ten permissions that a person could set.  These were represented by ten hyphens or minus signs:  ----------.  I couldn't tell what the first one was for, but the remaining nine were divided into three sets of three.  The first three belonged to the owner, the second three to the group, and the last three to "other."  Within each set of three, the first one was for read, the second was for write, and the third was for execute (i.e., run it as a program).  So if you set the owner's permissions to read (r), write (w), and execute (x), your ten hyphens would now change to this:  -rwx------.  If you set all three parties (i.e., owner, group, and other) the same, they would look like this:  -rwxrwxrwx.

You could set the permissions using the chmod command.  I found an Ubuntu manual page on chmod.  It was not really that complicated, but it looked like it was going to require a time investment to make sure I had it right, and at this point I was getting impatient.  The basic idea seemed to be that you could use chmod to enter values of 4 (for read permission), 2 (for write permission), and/or 1 (for execute permission).  So, for example, you could type "chmod 755" and that would give a value of 7 to the first of the three users mentioned above (i.e., the owner), a value of 5 to the second of the three (i.e., the group), and a value of five to the third of the three (i.e., other).  The 7 would mean that you gave read + write + execute (4 + 2 + 1) permissions to the owner, whereas the 5 would mean that you gave only read + execute (4 + 1) permissions to the rest.  Since that's what the instructions suggested, I went with that.  To set the script with those permissions, I typed "chmod 755 backup-hour.sh."

I wasn't too sure of who the owner was (i.e., me or root), not to mention the group or other.  I mean, this was for my home computer.  Not a lot of people milling around, waiting to take my hard drive for a spin.  These kinds of options seemed to be set up for networked computers, where the "accounting" department might be a group that would own a file.  I found what looked like a good tutorial on file owners, and another interesting (yawn!) page about permissions, but fortunately I did not have time to work through them.

When I typed "chmod 755 backup-hour.sh," I got "cannot access 'backup-hour.sh': No such file or directory."  One solution was to use a change directory (cd) command to get the Terminal prompt into the bin subfolder, so it would see what I was looking for.  But since I planned to put more scripts into that folder, and anyway since I wanted cron or other programs to know right away what I was talking about when I referred to something like backup-hour.sh, I decided to figure out how to put the bin folder in my "path."  The path is the list of folders where the operating system looks for guidance on what a command means.  To change my path so that the system would always know to look in bin, they said I needed to find and edit my .bash_profile file.  Unfortunately, they didn't say where it was.  It wasn't easy to find.  I ran searches in Nautilus (both user and root), but while those were grinding away, I found that I could just type "locate .bash_profile."  That turned up nothing, but very quickly.  Then I got some advice that, if it didn't exist, I could create it by using "touch ~/.bash_profile."  So I did that, and then tried again with "chmod 755 backup-hour.sh."  Still no joy.  Ah, but maybe that was because I hadn't rebooted; backup-hour.sh would run only on startup.  OK, so I used the other approach after all:  I changed directory to the bin folder and tried again.  Now I got "Permission denied."  What if I gave everybody full permissions with chmod 777?  I tried that instead of chmod 755.  That seemed to do it.  The hard drive was doing its thing now.

I wanted to see what was going on, so I decided to create a log file.  I wanted it to store only the error messages, not to list the thousands of files that backup-hour.sh was backing up successfully, so I put this on the end of (that is, on the same command line as) my rsync command in backup-hour.sh (above):

2> /media/CURRENT/backup-hour.log

The log filename thus matched the script filename.  I put "backup" first so that I could see all of my backup scripts in the same part of the folder's directory listing, and then I set up a backup-day.sh script along the same lines.  New problem:  these backup scripts would generate empty log files if there were no errors, and I didn't want to have to delete them manually.  So I found a forum post with advice on how to delete them automatically, using the "find" command.  In my version, it looked like this:

find /media/CURRENT/ -name "*.log" -size 0c -exec rm -f {} \;

I put that at the end of the backup-day.sh script, and it seemed to work.  It said, basically, look in the CURRENT folder for files whose name ends with .log and have zero bytes; and if you find any files like that, execute the "remove" command without asking for permission.  I didn't know what that ending punctuation is about, but that's what the advisor suggested.

In my backup-day.sh (not backup-hour.sh) script, I included the instructions for updating my USB jump drive (above).  I also included a set of commands to save my e-mail (I was using Thunderbird in Ubuntu) as a .tar compressed file. Actually, as seven .tar files, one for each day of the week.  That part of backup-day.sh looked like this:

# Assign Thunderbird mail & profile to be backed up
backup_files="/home/ray/.mozilla-thunderbird"
dest="/media/BACKROOM/Backups/Tbird"
# Create archive filename
day=$(date +%A)
hostname=$(hostname -s)
archive_file="$hostname-$day.tgz"
# Back up to a tgz file
tar zcvf $dest/$archive_file $backup_files 2>> /media/CURRENT/A-INCOMING/T-bird-Backup.log

So these seemed to be the basic kinds of tools I needed to set up rsync scripts and crontab entries.