Windows 7: Finding a DIR Alternative

I needed a DIR-type listing that would provide extended information about a file:  its name, date, and size, and also its path (i.e., the folder and subfolder where it was located), all on a single line of output.  DIR didn't seem to be capable of this, and neither did the utilities I found with a search (e.g., Karen's Directory Printer).

Another search raised the possibility that certain Linux utilities brought over to Windows might have this kind of capability.  I didn't want to run a Linux virtual machine on Win7; I just wanted to be able to run Linux commands that might add functionality I wasn't getting in Windows 7.

Linux commands were probably not the only alternative.  For instance, I could have learned how to use Windows PowerShell scripts.  My general impression of the Microsoft approach (as in the contrast between original BASIC and VB) was that, unfortunately, something that could be done with one relatively simple command in another tool would require three or four lines of code, which I would be able to write only after mastering a handful of relatively abstruse programming concepts, in the Microsoft product.  This impression seemed borne out when a search led to an indication that the DIR equivalent in PowerShell would require a multiline FOREACH loop.

Preliminary inquiries gave me the impression that Cygwin sought to provide a subsystem that would emulate a Linux machine within Windows.  There were indications that other projects (e.g., MSYS) sought to provide a somewhat comparable (e.g., 110MB) environment.  These seemed a tad heavy for my purposes; I was looking for something more like GnuWin, which was described as relying "only on libraries provided with any standard 32-bits MS-Windows operating system" and as not needing any Unix emulation.  Ideally, I would have some cool, relatively simple Linux-like commands available at the Windows command prompt.

By this point in my investigation, several people had mentioned CoreUtils.  This turned out to be a package within GnuWin.  The CoreUtils homepage described it as "the basic file, shell, and text manipulation utilities of the GNU operating system."  GNU was "a Unix-like operating system," in development since 1983, that apparently provided most of the materials used by Linux (which was, in turn, the source of Debian Linux, from which Ubuntu was built).

To clarify, it appeared that the CoreUtils existed in GNU, and there was an offshoot called CoreUtils for Windows.  Apparently this was what I would be getting through GnuWin.  There were other approaches to this sort of thing (e.g., Gow, UTools, UnxUtils), but my sense at this point was that GnuWin was dominant in this category.

I looked at the list of tools included in CoreUtils (for Windows).  I didn't count them, but I thought I remembered seeing an indication that there were more than 100 of them.  They were grouped into three main categories:  file utilities, text utilities, and shell utilities.  In the file utilities group, the description of the ls command was simply "lists directory contents,"; and vdir would apparently provide a "long directory listing."  These sounded like what I needed.  Examples in the text utilities category included comm ("compares two sorted files line by line") and uniq ("remove duplicate lines from a sorted file").  Examples in the shell utilities category included sleep ("suspends execution for a specified time") and uname ("print system information").

Although I could have just clicked on a download link, I went into the folder for the latest version and saw that it had not been updated since 2005.  This made me wonder whether I should have opted instead for Gow (short for GNU on Windows), which had apparently been updated as recently as November 2011.  I found a spate of (1 2 3 4 5) brief summaries of Gow published about that time.  Their similarities raised the thought that they may have been written from similar press releases.  Not that that would necessarily be bad.  Any product being promoted in 2011 could count as fresh air against a 2005 alternative.  But it was not reassuring that none of these explained clearly whether Gow was genuinely different, or just a borrowing, from the seemingly better-documented and more widely used GnuWin.  I found a page stating that Gow had been developed by a corporation in 2010 and used for some years before being released as open source.  This appeared to be an authoritative page.  It puzzlingly characterized GnuWin as being appropriate "if you want just one or two utilities."  A list of Gow utilities seemed similar, at a glance, to the GnuWin list (above), though I noticed that it did not have vdir.  The seeming mischaracterization of GnuWin, combined with the sense of evasion in the press-release writeups, persuaded me to stick with Plan A.

So now I did download and install the executable (exe; not src.exe) version of CoreUtils (6MB).  But, weird thing, they didn't give me a way to run the program.  My Start Menu had links to several PDFs.  Actually, it was rather messed up: they gave me four shortcuts to a total of two PDFs, and some of those links were buried about five layers deep in superfluous subdirectories. They also gave me two links to CoreUtils Help files that, when I clicked on them, gave me the familiar "Why can't I get Help from this program?" message that Windows 7 kindly provided when I would try to run Help files written for Windows XP.

Obviously, I ignored the manuals' actual contents and went looking for a way to run the program.  Weird thing:  I had all these redundant and dysfunctional help materials, and a link to an Uninstall routine, but no actual "Run CoreUtils" shortcut. I was half-tempted to uninstall them as defective, when it occurred to me that, well, they're supposed to be run from the command line, not the Start Menu.  So, OK, I went to the command line and typed "ls."  Windows said, "'ls' is not recognized as an internal or external command, operable program or batch file."  Hmm.  The manual, then, if I must.  Or manuals, I should say:  a regular-looking manual and also what appeared to be the set of Linux MAN (i.e., manual) pages, both in PDF format.  Neither had installation instructions.  I went to the ls MAN page.  It seemed to say that "ls -a" would be a working command.  Well, not on my machine, it wasn't.

I rooted around and found an article on how to use CoreUtils.  It said that I would have to adjust the PATH environment variable to tell the system where to look for the CoreUtils command instructions.  My way of applying those instructions was as follows:  first, in Windows Explorer, find where the CoreUtils executables (e.g., ls.exe) were installed.  On a 32-bit Windows 7 system, the location would probably be C:\Program Files\GnuWin32\bin; on a 64-bit system, C:\Program Files (x86)\GnuWin32\bin.  With that folder selected, click on the address bar at the top of Windows Explorer, make sure the whole address was highlighted (Ctrl-A if necessary), and copy the address (Ctrl-C).  Now I went to Start > Run > SystemPropertiesAdvanced.exe (could have used sysdm.cpl and then the Advanced tab) > Environmental Variables > System Variables > highlight Path > Edit > hit the End key.  There, I typed a semicolon (";") and then pasted in what I had copied from the Windows Explorer address bar.  (Could have typed it manually, using the 32-bit or 64-bit address just shown, but this was more accurate and it also forced me to verify the actual location.)  I OKed out of there and tried ls -a again on the command line.  Did I have to reboot to make the Path take hold?  Yes.  That was it.  I had ls, and it listed files.

So now, how about getting all that information mentioned at the outset -- path, date, etc., all on one line?  First question:  how could I get command-line command help?  In Windows, it was DIR /?.  But the /? option gave me an error with ls.  "man ls" didn't work either.  Page 9 of the manual PDF said the MAN pages were no longer being maintained.  I wasn't sure if that applied to what looked like the MAN pages included with GnuWin.  There wasn't a MAN MAN page in that PDF.  Page 10 said --help might work.  I tried "ls --help" and experienced satisfaction.  What I was seeing there looked like what appeared on pages 50-52 of the man PDF, pages 60-70 of 176 (text pages 52-62) in the more explanatory help PDF.  I wasn't inclined to read 11 pages to figure out how to get my directory listing.  Skimming down through the ls --help output, I tried "ls -l -N -R."  Good, but no cigar:  the path wasn't on the same line as the filename; no improvement over DIR.

The user's guide PDF didn't seem to think that there actually was a way to print the file's path on the same line as its date, filename, etc.  And so there I was.  I had come all this way with faith in my heart for the infinite possibilities of Linux.  I fervently believed that, with GNU, anything was possible.  But now, with my limited knowledge of Linux and such, cruel reality was saying Bismillah, no! we will not let you have all that stuff on one line of output.  There actually probably was a way to do it with some other tool, like the awe-inspiring grep, available in a different GnuWin package.  But I wasn't quite ready to go there.  In this project, grep looked, for me, like a bridge too far.

I thought about posting a question in the GnuWin Help forum.  But there had only been a handful of posts there in the last couple of months.  I also thought about going down the list of other utilities contained in CoreUtilities, so as to demonstrate to myself that this hadn't been a wild goose chase.  I thought about trying Gow after all, just in case its version of ls had different capabilities.  I thought about working up a kludge in which I would do a listing of all directories first (with e.g., "dir /ad /s /b") and then try to invent a way to append the pathname to each file line.

But before pursuing those rather lame possibilities, I noticed TCC/LE, advertised as a complete, powerful replacement for Windows CMD.  (TCC was short for "Take Command Console.")  It got 3.5 stars from 43 voters at Softpedia, only a solitary vote (five stars) at CNET -- but it had apparently been updated there just a few days earlier.  At MajorGeeks, it averaged 4.07 from 38 voters.  The description said it had enhanced commands (specifically including DIR) with new options.  A search didn't encourage the sense that there was a regular category of this sort of thing, with lots of competitors.  I downloaded and installed it.  The installation process seemed pretty slick, ending with a direct ride to their forums.  The installation left me with an open CMD window with a funky prompt, though apparently it was actually their own version of a command window.  (I did have another Win7 command window open throughout the installation.  It remained functional; I was able to close and open a new one after installation.)  I typed Help at their command prompt and went straight into their GUI help dialog, which actually made me say "Wow."  It wasn't spectacular; it was just good, and helpful, which I guess counts as spectacular after a long slog.  I replaced their ugly prompt with the ordinary Windows one by typing "prompt $P$g" at the prompt, though not without first amusing myself with variants (e.g., "Now what?").

Eventually I discovered that their help dialog was more or less the same as their online help page.  The manual had a large number of further instructions on how to tinker with the prompt and, it seemed, everything else.  Typing "option" at the prompt brought up settings, but not an obvious way to preserve prompt settings between sessions; it appeared the answer to that might lie somewhere within their SET command.  Anyway, I found information on their DIR command almost instantly, and also got a cursory version of it by typing dir /? at their prompt.  It led me to PDIR, and there I found the answer I was looking for.  What I had to type in a TCC/LE command window was this:

pdir D:\ /s /(dy-m-d zc fpn) > dirlist.txt

That gave me all of the information I was looking for, on a line-by-line basis, for every file on drive D, output into dirlist.txt.  Specifically, with the options in that sequence, I got the date (y-m-d), size (with commas), and the file path and name.

I took a quick look at their list of Commands by Category.  I also saw that they had a number of video and textual tutorials.  An impressive program.  But in any case, this investigation was done.

Basic Ubuntu (Bash) Shell Scripts

In Ubuntu 9.04, I wanted to write a script that would execute an rsync command, so that I could put a brief reference to the script into my crontab file, instead of putting the whole long rsync command there.

For a brief, tiny moment, I was almost tempted to consider learning the GAMBAS (Gambas Almost Means BASIC) programming language, just because (pre-Visual) BASIC was the only programming language I ever learned.  Instead, I moved toward basic instructions on writing a shell script.  Here's what I wrote:

#!/bin/bash
# This is backup-hour.sh
# It backs up CURRENT to CURRBACKUP every few hours
rsync -qhlEtrip --progress --delete-after --ignore-errors --force --exclude=/.Trash-1000/ --exclude=/lost+found/ /media/CURRENT/ /media/CURRBACKUP

As the instructions said, the first line was essential to tell the computer to use BASH to interpret the following lines.  The next two lines were comments, and the final line (wrapping over onto multiple lines here) was exactly the rsync line I'd been using to do the backup.  In other words, learning how to write the command was almost all I needed to write the script.

The next step was to save it somewhere.  I had previously heard, and the instructions said, that the common place to put it is in your bin folder.  I went with that, but made a note that I would need to be sure to back up my bin folder, because I didn't want to go to the trouble of writing all these scripts and then see them vanish.

The usual location for the user's bin folder is at /home/[username]/bin.  In my case, that's /home/ray/bin.  Getting there in Nautilus (i.e., Ubuntu's File Browser, also started by typing "nautilus" in Terminal) can be confusing:  you can get there via the Home Folder option (assuming you're showing Tree rather than Places or something else (or nothing) at the left side of the file browser), or you can go to File System/home/[username]/bin.  Same thing, either way.  So I saved the script (above) in my bin folder as backup-hour.sh.  That turned the comment lines (beginning with #) blue in gedit.

Next, the instructions said, I needed to set permissions.  This was a confusing aspect of Ubuntu.  The documentation seemed to say that there were ten permissions that a person could set.  These were represented by ten hyphens or minus signs:  ----------.  I couldn't tell what the first one was for, but the remaining nine were divided into three sets of three.  The first three belonged to the owner, the second three to the group, and the last three to "other."  Within each set of three, the first one was for read, the second was for write, and the third was for execute (i.e., run it as a program).  So if you set the owner's permissions to read (r), write (w), and execute (x), your ten hyphens would now change to this:  -rwx------.  If you set all three parties (i.e., owner, group, and other) the same, they would look like this:  -rwxrwxrwx.

You could set the permissions using the chmod command.  I found an Ubuntu manual page on chmod.  It was not really that complicated, but it looked like it was going to require a time investment to make sure I had it right, and at this point I was getting impatient.  The basic idea seemed to be that you could use chmod to enter values of 4 (for read permission), 2 (for write permission), and/or 1 (for execute permission).  So, for example, you could type "chmod 755" and that would give a value of 7 to the first of the three users mentioned above (i.e., the owner), a value of 5 to the second of the three (i.e., the group), and a value of five to the third of the three (i.e., other).  The 7 would mean that you gave read + write + execute (4 + 2 + 1) permissions to the owner, whereas the 5 would mean that you gave only read + execute (4 + 1) permissions to the rest.  Since that's what the instructions suggested, I went with that.  To set the script with those permissions, I typed "chmod 755 backup-hour.sh."

I wasn't too sure of who the owner was (i.e., me or root), not to mention the group or other.  I mean, this was for my home computer.  Not a lot of people milling around, waiting to take my hard drive for a spin.  These kinds of options seemed to be set up for networked computers, where the "accounting" department might be a group that would own a file.  I found what looked like a good tutorial on file owners, and another interesting (yawn!) page about permissions, but fortunately I did not have time to work through them.

When I typed "chmod 755 backup-hour.sh," I got "cannot access 'backup-hour.sh': No such file or directory."  One solution was to use a change directory (cd) command to get the Terminal prompt into the bin subfolder, so it would see what I was looking for.  But since I planned to put more scripts into that folder, and anyway since I wanted cron or other programs to know right away what I was talking about when I referred to something like backup-hour.sh, I decided to figure out how to put the bin folder in my "path."  The path is the list of folders where the operating system looks for guidance on what a command means.  To change my path so that the system would always know to look in bin, they said I needed to find and edit my .bash_profile file.  Unfortunately, they didn't say where it was.  It wasn't easy to find.  I ran searches in Nautilus (both user and root), but while those were grinding away, I found that I could just type "locate .bash_profile."  That turned up nothing, but very quickly.  Then I got some advice that, if it didn't exist, I could create it by using "touch ~/.bash_profile."  So I did that, and then tried again with "chmod 755 backup-hour.sh."  Still no joy.  Ah, but maybe that was because I hadn't rebooted; backup-hour.sh would run only on startup.  OK, so I used the other approach after all:  I changed directory to the bin folder and tried again.  Now I got "Permission denied."  What if I gave everybody full permissions with chmod 777?  I tried that instead of chmod 755.  That seemed to do it.  The hard drive was doing its thing now.

I wanted to see what was going on, so I decided to create a log file.  I wanted it to store only the error messages, not to list the thousands of files that backup-hour.sh was backing up successfully, so I put this on the end of (that is, on the same command line as) my rsync command in backup-hour.sh (above):

2> /media/CURRENT/backup-hour.log

The log filename thus matched the script filename.  I put "backup" first so that I could see all of my backup scripts in the same part of the folder's directory listing, and then I set up a backup-day.sh script along the same lines.  New problem:  these backup scripts would generate empty log files if there were no errors, and I didn't want to have to delete them manually.  So I found a forum post with advice on how to delete them automatically, using the "find" command.  In my version, it looked like this:

find /media/CURRENT/ -name "*.log" -size 0c -exec rm -f {} \;

I put that at the end of the backup-day.sh script, and it seemed to work.  It said, basically, look in the CURRENT folder for files whose name ends with .log and have zero bytes; and if you find any files like that, execute the "remove" command without asking for permission.  I didn't know what that ending punctuation is about, but that's what the advisor suggested.

In my backup-day.sh (not backup-hour.sh) script, I included the instructions for updating my USB jump drive (above).  I also included a set of commands to save my e-mail (I was using Thunderbird in Ubuntu) as a .tar compressed file. Actually, as seven .tar files, one for each day of the week.  That part of backup-day.sh looked like this:

# Assign Thunderbird mail & profile to be backed up
backup_files="/home/ray/.mozilla-thunderbird"
dest="/media/BACKROOM/Backups/Tbird"
# Create archive filename
day=$(date +%A)
hostname=$(hostname -s)
archive_file="$hostname-$day.tgz"
# Back up to a tgz file
tar zcvf $dest/$archive_file $backup_files 2>> /media/CURRENT/A-INCOMING/T-bird-Backup.log

So these seemed to be the basic kinds of tools I needed to set up rsync scripts and crontab entries.