Exporting Thunderbird Emails to PDF - Another Cut

I was using Thunderbird 11.0.1 in Windows 7.  I had accumulated some emails that I wanted to export as individual EML files.  An EML would still be readable in Thunderbird, and it would carry any attachments along with it.  I had attacked this problem on several previous occasions.  As before, I was not sure I would get all the way through from Thunderbird to EML to PDF.  This post provides another contribution in the slog toward that outcome.

First Step: From Thunderbird to EML Format

Some of my previous efforts to export to EML and then convert to PDF had produced something of a mess.  Exporting, itself, was easy enough.  I was using ImportExportTools.  It would give me EMLs with names containing some, but not all, of the information that I wanted in file names.  Specifically, it would provide the date and time, the sender, and the subject; but it did not include the recipient.  I could get it to produce a separate Index.csv file that would contain the full information, but that would just be a spreadsheet file.  I could use that spreadsheet file to give me nice names for files; but which file was supposed to get which name?  Matching them up had required a surprising amount of manual effort, last time around.  I was hoping to make the process smoother, if I could.

It wouldn't help to print a PDF directly from Thunderbird.  As far as I knew, that would require me to enter PDF filenames manually.  I was looking for a mass-production kind of solution.  About.com recommended mbx2eml, but it seemed to have some disadvantages, notably a very limited set of options for the resulting EML filenames -- which was the main problem.  Generally, it did not seem that any solution had broken through into prominence, in either the T-bird to EML or T-bird to PDF category.

In my first try at this problem, I had tried Total Thunderbird Converter and Birdie EML to PDF Converter, but for various reasons had not been impressed with either.  I did like Attachment Extractor, for when I got to that part of the project.  My notes seemed to favor Universal Document Converter (UDC) ($69), if I wanted a direct T-bird-to PDF-solution.  As I reviewed the struggles I'd had in that first try at this problem, and also in the second and third tries, I wondered if I should have focused more seriously on UDC.  But it did not seem to have command-line capability or other automation features.  It was basically a glorified PDF printer.  Moreover, its default filenames did not include all the information I wanted.

My previous notes did not seem to mention that Thunderbird messages were apparently already in EML format, stored in Thunderbird subfolders.  For instance, I had moved the messages that I was now seeking to export to a Local Folders subfolder called Export, and I could see that folder in Windows Explorer as Mail\Local Folders\Export.mozmsgs.  But this was confusing:  the number of EML files in that folder was not very close to the number of messages in the Export subfolder in Thunderbird.  Anyway, the EMLs in Export.mozmsgs had seemingly random names that would be useless for my purposes.

So I went ahead with ImportExportTools.  My first step was to eliminate duplicates.  For this, I used Remove Duplicate Messages (Alternate).  Then, in Thunderbird, I went to Tools > ImportExportTools > Export all messages in the folder > EML format.  The first time around, this produced undesirable results (see below).  But I didn't know that until I was partway through the second step.

Second Step:  Adding Recipient to the EML File Name

I had my EMLs.  But as noted above, I wanted to add the name of the Recipient to the filename, in the format Date-From-To-Subject.  As a first step, I thought I would just try to append the Recipient's name to the end of the filename.  Then I would figure out how to shuffle the words around to the desired order.

Given my limited knowledge of programming and such, I decided to try to achieve this with a Windows batch file.  I struggled to figure out how to write a suitable one, and finally posted a question on it.  One of the early answers to that question led to a separate pursuit -- a one-line batch file that would convert Word and WordPerfect documents to PDF.

The answers that I had received, at the point when I was writing up these notes, fell into two categories.  One, which I found easier to understand (and, predictably, seemed less popular among the knowledgeable respondents), involved a simple loop that would call an external process.  Basically, in plain English, it went like this:

FOR each EML file, run Process.
Repeat loop.
When list of files is exhausted, quit.

Process starts here.
Do various things.
End of process

By contrast, the approach preferred by most of the answering individuals would put all the steps inside the loop, instead of having a separate process afterwards.  It seemed to be a matter of style.  A second difference was that, in discussing the specific steps, they seemed divided between two general possibilities:  with, or without, delayed expansion.  Delayed expansion was apparently a response to a complication in how the FOR command worked.  As I understood it, the computer would read the entire contents of a FOR command as soon as it hit the word FOR.  So assigning a value to a variable inside a FOR loop would be too late; the computer would already have decided what value that variable had.  The variable would have been immediately expanded to its value.  Delayed expansion would postpone definition of the variable's value until later in the game.  A variable would be marked for delayed expansion by surrounding it with exclamation marks (e.g., !VAR!).  I wasn't familiar with delayed expansion, so I was in accord with some advisors' feeling that it would be better to proceed without it if possible.  What they (especially Aacini) suggested was:

@ECHO OFF

IF EXIST fullnames.txt DEL fullnames.txt

FOR %%f IN (*.eml) DO (

SET firstfind=

FOR /F "delims=" %%l IN ('findstr /B /C:"To: " "%%f"') DO (

IF NOT DEFINED firstfind SET firstfind=now & ECHO %%f%%l >> fullnames.txt

)

)

I have double-spaced the lines for clarity, anticipating that Blogger will wrap some long lines.  I haven't indented the way a programmer would, because of apparent limitations in the formatting options here in Blogger.  Basically, this batch file said, give me a fresh output file called Fullnames.txt; and on each line in Fullnames.txt, type the contents of two variables.  The first variable, %%f, was the name of the EML file under consideration, in all its Date-Sender-Subject glory.  There would be one such filename assignment for each EML file in the folder; hence a FOR loop.  The batch file would loop through all EML files in the folder.

Inside that FOR loop, there would be an examination of the contents of each individual EML.  This examination would use FINDSTR to locate the first line beginning with "To:  ."  The contents of that line would be assigned to the %%l variable.  (That's an L, not a one.)  I wasn't sure why this had to be done inside a second, inner loop, and I also didn't know how the "now" part worked.  But I was an openminded individual.  I was interested in new ideas.  The point is, I was willing to plow ahead and give it a try. 

So I copied the foregoing lines of script, beginning with @ECHO OFF and continuing to the last closed parenthesis (")"), and pasted them into a file in Notepad.  I saved that file as EMLNamer.bat, and put it into the folder containing the EMLs that I had exported from Thunderbird (above).  There, I ran it (either double-click it or highlight and hit Enter).  The command window displayed nothing, which was a bit disconcerting; but, viewing the folder in Windows Explorer, I could see Fullnames.txt spring into existence and grow larger.

When it was done, the command window disappeared, and Fullnames.txt stopped getting bigger. I put EMLNamer.bat into a folder where I could find it later.  I opened Fullnames.txt file and pasted its contents into Excel.  Some lines seemed to be missing.  Not many, but less than the total number of files shown in the Windows Explorer folder minus two (for EMLNamer.bat and Fullnames.txt).  I guessed that the names of a few EMLs had presented complications for the script.  I would have to process the rest and see what remained.

Third Step: Improving the EML File Name

I looked at the new Excel spreadsheet.  Spot checks, supplemented by previous experience with ImportExportTools, yielded the following observations:

• The first 13 characters in each filename seemed match the date and time (in 24-hour format) shown in Thunderbird for the email in question -- the time, that is, when the email was sent or received.
• The next characters indicated the sender.  This string ended, in some cases, with three characters (namely, "_-_") and in other cases with just one (namely, "-").  It seemed that ImportExportTools would surround some senders' names with underscores ("_") but would not do so for others.  The reason seemed to be that those senders' names appeared within brackets.  For instance, I had emails from "[Wordpress.com]" that now appeared as "_WordPress_com_."  So at least in these situations, the underscore seemed to be something that I could replace with a space, which would then be removed by an Excel TRIM command if it appeared at the start or end of a string.
• Some senders' names ended with "_com."  Ordinarily, the preceding note would suggest replacing that with ".com," and likewise for ".org," ".edu," and so forth.  But I decided that step would come later, if at all:  instead, I would start by identifying full names (e.g., "Yahoo_com") that I might want to replace with simpler names (e.g., "Yahoo").
• Hyphens were not always a reliable indicator of the end of a sender's name.  For example, an email from some "Pan-European" organization came through the ImportExportTools process unchanged.
• ImportExportTools seemed to replace apostrophes with underscores.  So instead of "Miller's" I would get "Miller_s_."  Likewise for other uses of the apostrophe (e.g., "Don't" became "Don_t_").  It seemed that, before doing any sweeping replacement of underscores, I might want to look for those sorts of special cases.
• Sometimes a hyphen would not be a reliable indicator of the end of a sender's name.  An example appeared in an email from a "Pan-European" organization:  it came through the ImportExportTools process unchanged.
• Due to the EMLNamer process, the end of the Subject field and the beginning of the Recipient field were marked by ".emlTo:" -- which was certainly recognizable.
• Subject fields often began with things like "Fwd_" and "Re_" -- which, I had decided in a previous use of ImportExportTools, would best be deleted.

In short, the default results from ImportExportTools (possibly altered during my previous tinkering) were creating some confusion.  I deleted the existing EMLs from the output folder, so as to start over.  Then I went into Thunderbird > Tools > ImportExportTools > Options and made several changes.  In the Misc. tab, I set each item to a maximum of 100 (rather than 50) characters.  (This wasn't exactly a mistake, but I would later realize that, as a result of this change, I needed to be more aggressive in keeping the total filename length relatively short; otherwise, it would cause problems in some other Windows operations.)  In the Filenames tab, I unchecked the option to "Use just alphanumeric characters in export"; I left the format to be Date - Sender - Subject; I left "Add time to date" checked; and I unchecked the "Cut subject" and "Cut complete file path" options.  In the Export directories" tab, I chose the "create a new directory and the index of messages" option.

When I ran that, I got an index.html file listing relevant information about each file:  its subject, from, to, date, and an indication of whether it had attachments.  This did not appear likely to be helpful, given its HTML format.  In the output folder, there was the right number of files.  I ran EMLNamer.bat again.  This time, the command window gave me some error messages.  Preliminarily, it seemed they were produced by the length of the filenames.  I could not save them before the command window closed.  There was probably a way to modify EMLNamer.bat to save those messages to a file, but I did not tinker with that at this point.  These messages appeared to be in addition to the unknown problems that had prevented Fullnames.txt from containing a complete list of all EMLs:  there were now about 20 filenames missing from the output that I pasted into Excel.  So, again, those would have to be dealt with manually.

This time around, when I pasted the results from Fullnames.txt into Excel, I saw that the output filenames had characteristics largely similar to, but in some regards different from, those noted above.  There were fewer underscores, which meant that it would probably be simpler to develop rules to translate them into more useful characters.  Hyphens were still not reliable field-end indicators.

Manipulating the File Information in a Spreadsheet

In Excel, after a couple of false starts not detailed here, I took the following steps:

• Insert row 1 for column headings.  Label column A as "Combined."  These entries contained the combined original filename plus the "To:" information added by EMLNamer.bat.
• In column B (heading:  "Original"), use =LEFT(A2,FIND(".emlTo: ",A2)-1) to obtain the original filename as exported from Thunderbird.  I would need this to remain unchanged:  my ultimate goal, a batch command indicating how the original filename should be changed, would need this information to tell the command processor what file was being renamed.  As with all other columns discussed below, I copied the formula down the column to all rows in use.
• In column C (heading:  "Find & Replace"), use =A2.  Fix the values in this column -- that is, make them permanent by highlighting them all and using the Edit - Copy, Edit - Paste Special -Values sequence.  The shortcut key sequence for Excel 2003 -- which I believed would work in ribbon versions like Excel 2007 and 2010 --was Alt-E C, Alt-E S V Enter Enter.  Now column C contained values rather than formulas.
• Move the values from column C to a new worksheet.  Don't rearrange them.  I needed a new worksheet because I was going to be using global find-and-replace (Ctrl-H) commands, and I didn't want to have to try to protect columns A and B from being affected by these commands.
• In that new worksheet, I made changes to the list that I had just brought over from column C in the first worksheet.  The first thing I did was to search for an unusual character, one I searched, in Excel, to find a character that did not already appear in the list.  The caret ("^") was one such character.  I would use this as my field delimiter.  I didn't want any of my Subject field entries to begin with "Re" or "Fwd," so I started by replacing "-Re_" and "-Fw_" and "-Fwd_" with carets, gambling (on the basis of previous experience) that there would be few instances where this would prove inadvisable.
• I also replaced the "-_" and "_-" and "-[" combinations with carets.  To reduce the number of underscores potentially requiring manual attention, I did one or two additional find-and-replace operations in obvious cases; for example, "Woodcock_s " (ending with a space) became "Woodcock's ."  It could have been counterproductive to go too far with this, though.  For example, I did not try to remove underscores from every version of my name and email address, because that could have created additional variations on my name, somewhere down in the list, potentially complicating the number of things I would have to look for later.  It was better to leave the underscore as a flag for some purposes.  Then I cut and pasted that modified list back into column C in the main worksheet.
• Back in the main worksheet, in column D, I set up a Date and Time column B, using =LEFT(C2,13).  I didn't parse that column for the various year, month, day, hour, and minute components at this point; that could wait until I needed that information.
• In column E, I created my first Remainder column.  The purpose of the Remainder columns was to show what was left from the modified values appearing in column C, after removing whatever I had just separated out (in this case, the date and time).  The formula was =TRIM(MID(C2,15,LEN(C2))).
• I used column F for the Recipient (i.e., "To") value, from the end of the string appearing in the Remainder column (E).  The reason was that this was a fairly obvious entry, and its removal would simplify the next steps.  The formula in column F was =TRIM(MID(E2,FIND(".emlTo: ",E2)+7,LEN(E2))).
• Column G could be another Remainder column:  =TRIM(LEFT(E2,FIND(".emlTo: ",E2)-1)).
• In column H (heading:  "Left 1"), I entered =LEFT(G2,1).  The reason was that ImportExportTools had failed to export the names of some senders, notably those appearing in angle brackets ("< >"), and I couldn't identify them by just sorting on the Remainder column because Excel would irritatingly overlook those characters when doing a sort.  But now I could sort on column H and make manual entries of those senders' names in the appropriate column.  I had not yet created that column, nor made those manual entries, because there was something else I needed to do first:
• In column I, under a "Hyphen" heading, I entered =IF(ISERROR(FIND("-",G2)),"",FIND("-",G2)).  In column J (heading:  "Caret"), I entered =IF(ISERROR(FIND("^",G2)),"",FIND("^",G2)).  Finally, in column K (heading:  "Best"), I entered =IF(J2="",I2,J2).  Column I would look for the first occurrence of a hyphen in the Remainder (column G).  Column J would do likewise for a caret.  It was necessary to use both because, at this point, either one might have been the delimiter indicating the end of the Sender field.  Column K would favor carets over hyphens, so as to reduce the number of problems with senders with hyphenated names.
• In column L ("Sender), I used =TRIM(LEFT(G2,K2-1)).  This produced good Sender names in most cases.  It was not yet time to deal with the exceptions.
• In column M ("Subject"), I used =TRIM(MID(G2,LEN(L2)+1,LEN(G2))). This produced good Subject names in most cases. Now it was time to deal with the exceptions.
• I went back and sorted on column H to identify those rows where I would have to make manual entries of Sender names because none was provided by ImportExportTools.  I put those entries in column L as needed, replacing whatever the automatic calculation had put there.  To assist in my process of looking up those that I didn't recognize, I sorted the From column in Thunderbird, for the Export folder, to gather all those senders at the top of the list for easier reference; I moved these items into a separate subfolder, sorted by Subject; I maximized the viewable space for that list; and once I had dealt with them, I moved them to another subfolder, so as to reduce the size of the list that I would have to page through.  The objective here was just to make sure I had a coherent division of information between the Sender and Subject columns -- to prevent some Sender data from appearing in the Subject column, or vice-versa. Cleaning them up or otherwise improving them at this point would have been premature.  Changing Sender names worked best if I made the changes back in column G, or if I altered or removed numbers in columns I and J.  Just making a change in the Sender column would leave a problem in column M.  It helped, for this purpose, to fix the values in column G (that is, to replace formulas with values; see the procedure described in connection with column C, above).
• I sorted on column M ("Subject") and cleaned up the entries there.  I found that I wanted to do find-and-replace operations on multiple entries.  I decided at this point that I could safely fix the values of the entire spreadsheet.  It seemed that I would want to sort and re-sort these Subject values to get similar ones together.  To preserve the original order, I added an Index column, indicating the original numerical order of entries.  (Enter 1 and 2 in the first two rows; highlight all rows to be numbered; then hit Alt-E I S Enter.)  Then I moved the Subject and Index columns to a separate temporary worksheet, where I could do these sweeping changes without affecting other columns.  There, I reversed these two columns, putting Index on the left, to keep it out of harm's way.  My changes here included LEFT and RIGHT commands to sort by first and last characters of Subjects (supplemented, on the left, with CODE comparisons, to identify unwanted lowercasing), as well as FIND and Ctrl-H searches and replacements for underscores (doing many replaces to eliminate most instances) and other text that I wanted to change across multiple Subjects.  To identify undesirable characters (e.g., exclamation marks and others whose presence in filenames might mess up batch commands and other applications), I used SUBSTITUTELIST.  SUBSTITUTELIST would remove the characters listed in a separate worksheet (generated with a series of numbers 1-255 in column A and a corresponding =CHAR(A1) in column B).  I could have had it remove characters that looked unwanted, but to be cautious I decided instead to have it remove everything that I knew was normal (i.e., 0-9 and a-z and A-Z, plus a few others) and show me what was left.
• I deleted columns that were unnecessary, now that I had fixed the values.  I also moved some columns, and inserted a few ones.  My arrangement was now as follows:  Index (column A), Original (B), Date & Time (C), Sender (D), NewSender (E), Recipient (F), NewRecipient (G), and Subject (H).
• I copied values from columns D (Sender) and Recpient (F) to a separate worksheet.  There, I did a unique filter.  This gave me a list of names that I might want to change or simplify.  I put the original (unique) name in column A in that separate worksheet, sorted it, and entered the desired replacement names in column B.  I sorted this Names worksheet on column A (Original).  I named this worksheet Names; I planned to keep it for future Thunderbird EML exports.  I named the main worksheet Data.  I sped up the process of developing replacement names by using various functions (e.g., FIND, MID) to distinguish first and last names of individuals.  When I had my completed list of preferred names for Senders and Recipients, I went back to the main (Data) worksheet.  In column E (NewSender), I entered =VLOOKUP(D2,Names!$A$2:$B$869,2,FALSE).  (There were 869 rows in the Names spreadsheet.)  I copied that formula over to column G (NewRecipient); it provided a similar replacement for the Recipient values.
• I inserted columns to figure out the date and time.  In column D ("Y"), I used =LEFT(C2,4).  In colunn E ("M"), I used =MID(C2,5,2). In column F ("D"), I used =MID(C2,7,2).  In column G ("H"), I used =MID(C2,10,2).  In column H ("M"), I used =MID(C2,12,2).  Finally, in column I ("NewDate"), I used =D2&"-"&E2&"-"&F2&" "&G2&"."&H2.
• I added column O ("New Name").  There, I used =CHAR(34)&I2&" Email from "&K2&" to "&M2&" re "&N2&".eml"&CHAR(34).  This produced a new name for the EML file.  I sorted on this column to identify instances where my formulas had failed, and made corrections as needed.
• I added column P ("Batch").  There, I used ="ren "&CHAR(34)&B2&".eml"&CHAR(34)&" "&O2.  This produced a batch command to rename the EML file to my preferred new name.  I copied the command down the column and then copied all those commands, one from each row, to Notepad.  I saved the Notepad file as Renamer.bat and put it into the folder where the EMLs were located.  I ran Renamer.bat.  The renamed files sorted conspicuously in Windows Explorer, so I didn't need to work up a modification of these REN commands in a new column Q, using MOVE instead of REN, to move the newly renamed files to another folder.  Instead, I could just cut them from the folder in Windows Explorer and put them aside.
• Now I had a couple dozen EMLs remaining.  They had not renamed properly.  I probably should have added something like " > errorlist.txt" at the end of each batch command, to show me whether I was trying to give the same name to two different files.  I did a DIR of the files remaining, saved its output to dirlist.txt, copied the contents of dirlist.txt into Excel, and compared them against my main spreadsheet.  To my surprise, none of these files appeared in the original list of files shown there.  I'd had some problems not described in this post; had I somehow dropped some EMLs somewhere along the line?  Was I not doing this comparison properly?  I did not have a clear answer.  I worked up another set of new file names for these EMLs, substantially following the steps presented above, and renamed them.  It looked like, somehow, at least some of them were duplicates after all.  So I was not understanding something there.  Others were apparently not renaming properly because the original filenames contained characters like ®.

In the end, I wound up with all but 10 of the exported emails.  But which 10 did I lose?  I probably didn't lose any.  There was a point when I deleted a handful of what I thought were duplicates.  Now it seemed they probably weren't.

As these final notes suggest, while this process went much more smoothly than on my previous exports of emails from Thunderbird, there were still some parts of the process where I was making mistakes or where things were not going smoothly.

Fourth Step:  Converting the Appropriately Renamed EML to PDF

Some of the previous posts cited at the top of this post grappled with the problem of converting EML to PDF.  It would seem that it should have been a straightforward matter of selecting EMLs in Windows Explorer -- indeed, within Thunderbird itself -- and clicking a Print command.  Alas, it was not, not if the goal was to have PDFs whose filenames would be recognizable.  I hoped there would be a Thunderbird add-in that would solve all the many steps shown above.  I hadn't found one yet.  In my most recent effort, I had proceeded only as far as a truly cumbersome solution that divided EMLs with and without attachments, used Emacs to edit EMLs with attachments so that they would print, extracted attachments to separate files that could also be PDFed, and then manually matched the PDFed attachments up with their PDFed parent emails.  Truly a mess, very time-consuming, and for that reason I hadn't gone very far with it.  Most of my emails were still in EML rather than PDF format.

I had decided, generally, that PDF was the superior long-term archival format.  I didn't want lots of formats rattling around, lest the day come (as had happened for previous formats) when it was a struggle to find software that would read it.  That said, EMLs were presently displaying nicely in Thunderbird, with easy access to attachments.  Having devoted the time to the foregoing effort, I was presently out of time for further development of this project.  So the stack of EMLs grew higher, and the day for conversion to PDF still lay somewhere in the future.

Windows 7: Thunderbird: Add Security Exception

I was using Mozilla Thunderbird 10.0 for email.  I suddenly got this message:

Add Security Exception

You are about to override how Thunderbird identifies this site.

Legitimate banks, stores, and other public sites will not ask you to do this.

Server Location:  imap.exchange.iu.edu:993

Certificate Status

This site attempts to identify itself with invalid information.

Wrong Site

Certificate belongs to a different site, which could indicate an identity theft.

Below that, there was an option to "Permanently store this exception" and buttons to Confirm Security Exception or else Cancel.  Canceling didn't achieve anything:  the same dialog came right back.  I was afraid to click anything else.  The site in question was linked to Indiana University, which seemed legit.  I had gotten something like this before, using an earlier version of Thunderbird, but there the problematic email account had been Hotmail.

I wasn't sure why I was getting this.  A Mozilla page said, "The problem usually arises when the mail server's certificate is invalid for some reason. . . . Often this problem takes care of itself, in that the mail server provider will realize that they have made an error with their certificate and will replace it with a corrected version."  I had started getting this program maybe a year earlier.  I didn't know if this meant that I was the only person at Indiana University using Thunderbird, or what the explanation might have been.

I didn't find anything on point in Indiana's knowledgebase.  A search led to advice to make a change in Server Settings.  To get there, I had to click Cancel with the "Permanently store this exception" box checked; otherwise the dialog wouldn't budge.  Even so, I had to click Cancel a bunch of times to get out of there.  I went into Thunderbird > Tools > Account Settings.  I went to the Server Settings heading under the listing for the Indiana University account on the left side.  The advice seemed to be that, in that area, I should go to the Security Settings area and set Connection Security to None, instead of its present setting of "SSL/TLS."  Doing that changed the Authentication Method from "Normal password" to "Password, transmitted insecurely."

I wasn't sure about that.  I looked further down that same thread.  Someone else seemed to be saying that the address should be imap.exchange.iu.edu.:993, with a period before the colon.  Staying in the Account Settings dialog, and looking specifically at the list of options on the left side, I moved from the Server Settings option down to "Security," the last option for the Indiana University email account.  There, I went into Certificates > View Certificates > Servers tab.  I saw that I had certificates here for Mozilla, Google, Yahoo, etc.  It seemed like a legitimate list.  I decided to go with the advice, which was to go into Add Exception and type https://imap.exchange.iu.edu.:993.

Before I finished that, I took another look at the Mozilla page.  They said that the mail server provider (i.e., IU) should provide the necessary connection information.  They said that the Add Security Exception option that I was just about to finish would make my email through that account nonencrypted and visible to third parties.  Writers in another recent thread indicated that they, too, were having this problem.  The list of certificates (for e.g., Google, Yahoo) seemed to indicate that they had done what was necessary to provide email security at some level, but Indiana University had not.  I tried another search, but it led to surprisingly few hits, among which the main relevant reaction was puzzlement.

The Mozilla page seemed to be saying that you have three options in this kind of case.  You can ask the mail server people to get it together.  You can add a security exception -- or, as it appeared in my case at least, you would have to add a security exception if you wanted the program to be usable.  Or you could switch to a different account.  I wasn't sure whether switching to a different email program would provide another possible solution.

Thunderbird: The file Mailbox [filename] Cannot Be Found

I was using Thunderbird 3.1.16 in Windows 7.  I had a Hotmail account set up in T-bird, with the usual folders (Inbox, Drafts, etc.), and I also had a folder called E-mail Archive set up under Local Folders.  I was in the habit of removing messages from Hotmail's Inbox, when I was done with them, and storing them in the E-mail Archive folder.

Then I ran into a problem.  I clicked on a message in the E-mail Archive local folder and saw that there was no message body.  I could still see the From, To, Date, and all that, but the text of the message was gone.  At about the same time, I got an error message.  It said something like, "The file mailbox:///C|/Users[filename] cannot be found.  Please check the location and try again."

I ran a search.  It appeared that this was a pretty rare problem.  A different search led to a MozillaZine page that led me to think the problem was that I had not been compacting the E-mail Archive folder often enough.  The page said that compacting was a way to keep folders in good shape.  If I saw an email message with a weird date (e.g., sometime in 1969), that was a sign that the folder had become corrupted and should have been compacted.  I had indeed seen a message or two like that.  I had already set automatic compacting (Tools > Options > Advanced > Network & Disk Space), but apparently my value of 50MB (i.e., "Compact folders when it will save over 50000 KB") was too high.  So now I set that to 5MB instead.  It sounded like I would now be getting compaction prompts more frequently.  That webpage had advice on how to respond to this error message if the problem was with the Inbox, but that wasn't my problem.

Another MozillaZine page offered tips on how to set up and maintain Thunderbird.  I had stayed with version 3 instead of updating to a newer version of Thunderbird (currently 8.0 was the lastest) because I was using some add-ons that weren't compatible with the newer versions.  But this webpage seemed to offer a way around that.  So one possibility at this point was to upgrade and see if that would somehow solve the E-mail Archive folder problem as well.

Before doing that, it looked like there might be something I could do to retrieve the E-mail Archive folder within my existing setup.  One thread gave me the impression that an email folder might consist of a pair of files, working together.  One would have a name like REMC#1.msf, and the other would be REMC#1 (without the .msf extension).  I used Everything to search for *.msf files.  My system had 34 of them, all in the Thunderbird folder.  Originally, it seemed, the default installation location for Thunderbird in Windows 7 was at C:\Users\<account>\AppData\Roaming\Thunderbird\Profiles\[8 random characters].default, but I had moved that folder to drive D.  Specifically, I saw an E-mail Archive.msf file in that folder; but when I searched again for that filename, there wasn't an accompanying E-mail Archive file (without the extension).  I did have a backup of one, though.  It was big -- 500MB, apparently containing my tons of emails that should have been compacted.  So now I closed Thunderbird, copied that backup file (i.e., E-mail Archive) to the Thunderbird folder alongside E-mail Archive.msf, moved (instead of deleting) the E-mail Archive.msf as advised, and started T-bird again.

While doing the necessary screwing around in vague efforts to find a possible solution, I had already closed T-bird once before, after discovering the problem with the E-mail Archive folder, and when I started T-bird again, that folder was completely gone.  My guess was that the *. file (that is, E-mail Archive, without the extension) was responsible for telling T-bird that it should display a folder called E-mail Archive.  So when that non-extension file disappeared, so did Thunderbird's indication that I was supposed to have a Local Folders folder called E-mail Archive.

But now that I had copied the E-mail Archive *. file from backup to the Thunderbird folder, I did once again have an E-mail Archive folder visible under Local Folders.  I clicked on it.  The status bar said "Loading message . . ." for a minute, and then it was ready to show me the message bodies (i.e., the text of my email messages) once again.  So, wow, problem solved.  I dragged a boatload of messages from the E-mail Archive folder to another local folder.  I did that because it seemed that the E-mail Archive folder had gotten too big to work properly, so I wanted to reduce its contents before doing anything else.

Some months earlier, I had undertaken a project to remove old emails from Thunderbird and put them into individual PDF files.  It was convenient to have recent emails in T-bird, where I could quickly search for and reply to them.  But as messages got older, it was less likely that I would be using them for that sort of search and reply.  I decided to create a Thunderbird local folder called Ready to Archive, and I moved all messages older than six months into that folder.  I was a little superstitious about the E-mail Archive folder, so I decided to replace it with a new folder called Email Archive (without the hyphen). So I moved the messages that were less than six months old into Email Archive.  Now the E-mail Archive folder was empty.  Searches in Everything now showed that my new folders -- Ready to Archive and Email Archive -- were a couple hundred megabytes each.  Together, their sizes came close to the total size of the old E-mail Archive folder, and the combined numbers of emails in those folders (as displayed in the status bar at the bottom of the Thunderbird window) added up to approximately the correct total.  (I didn't think of recording the exact number before moving those items out of the old E-mail Archive folder.)  So it looked like things had worked out.  (Everything showed that the old E-mail Archive folder was still huge, but I guessed that compaction would probably shrink it down to almost nothing, if I had bothered to compact it.)

Deleting the E-mail Archive folder was not as easy as I thought it should be.  This folder had a different icon than the ones I had just created.  Maybe that explained why there was no right-click Delete option for this folder, and the Delete key didn't work.  I went into this folder's right-click > Properties and saw that there was a Repair Folder button that, according to the information provided there, might repair the .msf index file.  Nice to know, but I didn't need it anymore.  I closed Thunderbird, went into Everything, deleted the E-mail Archive and E-mail Archive.msf files, and restarted Thunderbird.  Now the E-mail Archive folder was gone, and the new Email Archive and Ready to Archive folders seemed to be working properly.

I closed Thunderbird, backed up the Thunderbird folder, and decided that my next projects in this area would be, first, to see if the advice cited above would work -- if I could upgrade T-bird while keeping my add-ons -- and, second, to take another shot at a hopefully streamlined process for that project (above) of converting old emails into PDFs.

Windows 7: Adding Hotmail as Default Email in Firefox 8.0.1

I was using Firefox 8.0.1 in Windows 7.  (Apparently these instructions would also work for Firefox 3.0.)  I was looking at a webpage containing an email address.  I clicked on the email link and got a dialog saying this:

Launch Application

This link needs to be opened with an application.
Send to:
Thunderbird
Yahoo! Mail
Gmail

Choose an Application

Hotmail was not on the list.  But I wanted to use Hotmail.  To find a solution, I did a search and wound up with some advice on the process.  I decided to save a writeup of the advice here because it took me a while to find it.

The advice was to type "about:config" in the Firefox address bar, hit Enter, type "gecko" into the Filter box, and double-click on the first item on the list.  This was the item called "gecko.handlerService.allowRegisterFromDifferentHost."  Double-clicking on it put it into boldface and changed its value from false to true.  Then I closed the about:config tab.  I went back to the address bar and typed this command, all on a single line:

Javascript:window.navigator.registerProtocolHandler("mailto","http://hotmail.msn.com/secure/start?action=compose&to=%s","Hotmail")

and then I hit Enter.  This gave me a bar at the top with the question, "Add Hotmail (hotmail.msn.com) as an application for mailto links?"  I clicked on "Add Application."  That gave me a blank screen.  I went to Tools > Options > Applications.  There were a bunch of items listed there.  I went down the Content Type column until I found "mailto."  I selected it.  This gave me a drop-down menu on the right side.  I selected the "Use Hotmail" option and clicked OK.

I went back to the webpage where I saw that email link.  I clicked on that link.  This opened up the Hotmail sign-in webpage.  I signed in.  Hotmail immediately started an email addressed to that email address.  I composed and sent an email.  It seemed to go OK.  Problem solved.

Windows 7: Archiving Emails (with Attachments) from Thunderbird to Individual PDFs - First Try

I had been collecting email messages in Thunderbird for a long time.  I wanted to export them to PDF-format files, one message per file, with filenames in this format:

2011-03-20 14.23 Email to John Doe re Tomorrow.pdf

reflecting an email I sent to John Doe at 2:23 PM on March 20, 2011 with a subject line of "Tomorrow."  This type of filename would sort correctly in Windows Explorer, chronologically speaking; hence, I could quickly see the order of messages.  There were a lot of messages, so I didn't want this to be a manual process.  This post describes the steps I took to make it semi-automated.

The Screen Capture Approach

The first thing I did was to combine all of the email messages that I wanted to export into one folder in Thunderbird.  Then I deleted duplicates from that folder.  Next, I decided that, actually, I was just interested in exporting messages prior to the current year, since recent messages might have information I would want to search for in Thunderbird.  So I moved the older messages into a separate folder.  I maximized the view of that folder in T-bird and shut down unnecessary layout features (i.e., message pane, status bar), so that I could see as many emails as possible on the screen, and as much as possible of the relevant data (i.e., date, time, sender, recipient, subject) for each email.  I did that because I wanted to capture the text information about the individual email messages.  The concept here was that I would do a screenshot for each screenful of emails, and would save the data from that screenshot into a text file that I could then massage to produce the desired filenames.  For this purpose, I tried a couple of searches; I downloaded and ran JOCR; but after a bit of screwing around I decided to revert to the Aqua Deskperience screen text capture shareware that I had purchased years earlier.

Index.csv

Then it occurred to me that perhaps I could just export all that information from T-bird at once.  I ran a search and downloaded and installed the ImportExportTools add-on for T-bird.  (Alternatives to ImportExportTools included IMAPSize and mbx2eml, the latter explained by About.com.)  It took Thunderbird a while to shut down and restart after the installation.  I assumed it was getting acquainted with the fact that I had relocated so many messages to a new folder.  When it did restart, I ran the add-on (Tools > ImportExportTools > Export all messages in the folder > Just index (CSV)).  I opened the CSV file (named index.csv) in Excel and saw that this was perfect:  I had a list of exactly the fields mentioned above (date, time, etc.).  I verified that Excel was showing a number of rows equal to the number of messages reported on the status bar back in Thunderbird.

I noticed that some of the data in the Excel file included characters (i.e., \ / : * ? " < > | ) that were not permitted in Windows filenames.  The Mbx2eml option (above) would have removed these characters automatically, but for this first time I wanted to do everything manually, so as to see how it was all working.  I thought this might also be better for purposes of making things the way I wanted them.  I was also not sure that Mbx2eml would produce a CSV file, or that it would output the emails in the same order.  There seemed to be some other potential limitations.  It looked like a worthy alternative, but I didn't explore it.

Somewhere around this point, I went ahead prematurely with a time-consuming effort to revise the entries in the spreadsheet, so as to remove the unacceptable characters and otherwise make them look the way I wanted.  Eventually, I realized that this was a mistake, because now I would have a hard time matching spreadsheet entries automatically with the actual emails that I would be exporting from Thunderbird.  So I dropped that attempt and went back to the point of trying to plan in advance for how this was all going to work.

Attachments

I had assumed that I wanted to export emails to individual .eml files because EML format would bring along any attachments that happened to be included with a particular email message.  But I didn't plan to just leave the individual emails in in EML format; I wanted to save them all as PDFs.  In other words, I wanted to have the email and its attachment within a single PDF.

A quick test notified me that printing EMLs would be no more successful at including the attachments than if I just printed directly from Thunderbird, without all this time-consuming exporting and renaming.  There were other solutions that would have worked for that purpose as well.  A search led me to InboxRULES, which for $40 would do something or other with attachments in Outlook.  (Alternate:  Automatic Print Email for $69.)  There didn't seem to be a solution for Thunderbird, and I wasn't keen on spending $40 and having to install Outlook and move all these emails there in order to print their attachments.  I thought about handling the attachments manually -- print the email first, then print the attchment, and append it to the email -- but a quick sort in Thunderbird told me there were hundreds of messages with attachments.  Funny thing about that, though:  as I arrow-keyed down through them in Thunderbird, allowing them to become visible one at a time, I saw that Thunderbird would change its mind with many of them:  it thought they had attachments, but then it realized they didn't.  That trimmed out maybe 5% of the ones that had originally been marked as having attachments.  But there were still a lot of them.

Another search led to some T-bird options, but it still looked like there was going to be a lot of manual effort before I'd have a single PDF containing both the email and its attachment.  Total Thunderbird Converter looked like it might come close, at a hefty price ($50).  It wasn't reviewed on CNET.com or anywhere else, as far as I could tell, so there was a risk that (as I'd experienced in other cases) the program simply wouldn't work properly.  But then I saw that they were offering a 30-day free trial, so I downloaded and installed it.  It turned out to be useless for my purposes:  it had almost no options, and therefore could not find my Thunderbird folders, which I was saving on drive D rather than C so as to avoid losing them in the event of a Windows update or reinstallation.  I looked at Email Open View Pro (or its alternate, emlOpenView Free), which also offered a free trial.  It didn't look like it (or Universal Converter, or MSG Viewer Pro, or E-mail Examiner, or Convert EML to PDF) would bring the attachments into the same PDF as the email, so I moved on.  I tried Birdie EML to PDF Converter.  Their free demo version allowed me to convert one EML file at a time.  I liked its interface:  it gave me eight different naming options for the resulting file (e.g., "date + subject + from," in several different date formats).  I didn't like the output, though:  they formatted the PDF for the EML file oddly, with added colors that I didn't want, and all they did with the attachment was to put it into a subfolder bearing the same name as the resulting PDF.  I'd still have to PDF it -- the example I used was an EML with a .DOC file attachment -- and merge it with the PDF of the EML.  But now they had led me to see that perhaps I could at least automate the extraction of attachments, getting me partway to the goal.

At about this point, Thunderbird inconveniently lost a folder containing several thousand email messages.  It just vanished.  The program was struggling there for a few minutes before that, and part of me was instinctively thinking that I should shut down the program and do a backup, but this would have been a very deeply subconscious part of me that was basically unresponsive under real-life conditions.  In other words, I didn't.  So now I had to go rooting around in backups to see what I could rescue from the wreckage.  I found that Backup Maker had been happily making backups, as God intended.  Amazing what can happen, when you have about five different backup systems running; in this case I had just wiped a drive, moved a drive, etc., and so of course Backup Maker was the *only* backup system that was actually in a position to restore real data when I seriously needed it.  What Backup Maker had saved was some files with an .MSF extension.  These were supposedly backups of Thunderbird.  But then, no, on closer inspection I realized these were much too small, so I had to do some more digging.  Eventually I did patch together something resembling the way things had been before the crash, so I could go back and pick up where I had left off.  A couple of days passed for other interruptions here, so the following information just reports where I went from this point foward.

I had the option of just saving the Thunderbird file, or the exported emails, for some future date when there would perhaps be improved software for printing attachments to PDF in a single operation with the emails to which they were attached.  There had been times when software developments had saved (or would have saved) a great amount of time in what would have been (or actually was) a tedious manual effort.  On the other hand, I had also seen situations where letting something sit meant letting it become confused or corrupted, or where previous learning (especially on my part) had been lost.  I decided to go ahead with converting the emails to PDF to the extent possible without a tremendous time investment.

My searching led to Attachment Extractor, a Thunderbird extension.  I installed it, highlighted two emails with attachments, right-clicked on them, and selected "Extract to Suggested File-Folder."  It worked -- it did extract the attachments without removing them from the emails.  I assumed it would do this with hundreds of emails if I so ordered.  Then, to get them matched up with PDFs of the emails to which they were attached, I would apparently have to page down through those emails one by one, looking at what attachments they had, and setting them up for more or less manual combination.  Attachment Extractor did have one potentially useful feature for this purpose:  a right-click option to "Extract with a Custom Filename Pattern."  I found that I could configure the names given to the extracted attachments, so that they would correspond at least roughly with the names of emails printed to PDF.  To configure the naming in Attachment Extractor, I went into Thunderbird > Tools > Add-ons > Extensions Tab > AttachmentExtractor > Options > Filenames tab.  There, I used this pattern:

#date# - Email from #fromemail# re #subject# - #namepart# #count##extpart#

and, per instructions, in the Edit Date Pattern option I used this pattern:

Y-m-d H.i

That gave me extracted attachments with names that were at least roughly similar to the format I wanted (see above).

Batch Exporting Emails with Helpful Filenames

Now if I could print the corresponding email to PDF with a fairly similar name, the manual matching might not be so difficult.  A search led to inquiries about renaming PDF print output.  For $60, I could get Zan Image Printer, which sounded like it would have some capability for automating PDF printout filenames.  Print Helper, for $125 to $175, was another option.  A Gizmo's Freeware article did not say much about this kind of feature, though several people asked about it.  A list of free PDF printers led me to think that UltraPDF Printer was free and would do this; its actual price was $30. 

The pdf995 Experiment

At this point, I was reminded of how much time I could waste on uncooperative software.  No doubt many people have used pdf995 successfully.  I was not among them.

I tried Pdf995's Free Converter.  The instructions on bypassing the Save As dialog were in the Pdf995 Developer's FAQs page.  They seemed to require me to open C:\Program Files\PDF995\res\pdf995.ini in Notepad.  But that .ini file seemed to be configured for printing one specific file that I had just printed.  They didn't say how to adjust it.  Eventually I figured out that I needed to download and install pdfEdit995, and make the changes there.  So I tried that.  But I got an error message:

PdfEdit995 requires that Pdf995 v9.1 or later and the free converter v1.2 or later are already installed on your system.

But they were!  I had just installed them.  Was I supposed to reboot first?  No, a reboot didn't fix it.  I tried again to install basic pdf995 and the Free Converter, which I had downloaded together.  Once again, I got the webpage saying I had installed it successfully.  Was I supposed to install the printer driver too?  I understood the webpage to be saying that was included in the 9MB download.  But I tried that.  Got the congratulatory webpage, so apparently it installed correctly.  Now I noticed I had another error, which had not come up on top, so I was not sure how long it had been there:

Setup has detected you have an old version of pdfEdit995 that is incompatible with the latest version of pdf995.

But I had just downloaded it from their website!  Not an altogether auspicious beginning here.  But I downloaded and installed the very latest and tried again, and now it seemed to work, or at least I got a different congratulatory webpage than before.  A cursory read-through still did not give me a clear approach to automated naming of PDFs.  Instead, they said that maybe I wanted their OmniFormat program for batch PDF creation.  Who knew?  I downloaded and installed OmniFormat.  Got a new congratulatory webpage, but still no straightforward explanation of batch naming.  Instead, it said that pdfEdit995 was what I wanted to create batch print jobs.  So, OK. a bridge too far.  Though at this point they specified "batch print jobs from Microsoft Office applications," raising the prospect that this wasn't going to work from Thunderbird.  Went back to their incredibly tiny-print pdfEdit instructions page.  It said I would have to set pdf995 as the default printer to do the batch thing.  That was OK.  But it still sounded like it was intended primarily for batch printing from Microsoft Word.  I decided to just try making pdf995 the default printer.  That required me to go to the Windows Start button > Settings > Printers > right-click on PDF995 > set as default printer.  While I was there, I right-clicked on PDF995 and looked at its Properties, but there didn't seem to be anything to set for purposes of automating printing.  Now I went to Thunderbird, selected several messages, and did a right-click > Print.  Funny, it defaulted to Bullzip, which was my usual default printer.  Checked again:  yeah, pdf995 was definitely set as my default printer.  Tried again, and this time manually set it to pdf995 when it was printing.  It asked for the filename, so that wasn't working.  Back in Printers, I looked at the Properties for Bullzip, but it didn't seem to have any options for automatic naming either.  It seemed pdf995 was not the solution for me.  I came away from this little exploration with less time and ambition for the larger project.  Certainly I wasn't in the mood to buy software and then discover that I couldn't make it work.

Further Exploration

I ran across an Addictive Tips article that said PrintConductor was a good batch printing option, though I might need to have Adobe Acrobat installed first.  I did, so I took a look.  There was an option to download Universal Document Converter (UDC) as well.  I wasn't sure, but I thought I might need that for Print Conductor, so I installed both.  PrintConductor didn't seem to have a way of printing EML files.  Meanwhile, UDC's installer gave me the option of making it the default printer, so I tried that.  But as before, Thunderbird defaulted to Bullzip, so I had to select UDC as my printer manually.  (Print Conductor did not appear in the list of available printers.)  When I selected UDC as the printer, before printing, I clicked on the print dialog's Properties button and went into the Output Location tab.  There, I chose the "predefined location and filename option."  I left the default filename alone and printed.  And it worked.  Sure enough, I had a couple of PDFs whose names were the same as the Subject fields shown in Thunderbird for those printed emails.  So I would be able to match them with the attachments produced by Attachment Extractor (above).  All I had to do now was to pay $69 for a copy of UDC, so that each PDF would not have a big black "Demo Version" sticker on it.

Recap

So to review the situation at this point, I had a way of extracting email attachments with highly specific date, time, and subject filenames.  I also had a way of extracting emails themselves whose filenames would show date and subject, using ImportExportTools (above):  Tools > ImportExportTools > Export all messages in the folder > EML format.  Unfortunately, there could be a number of messages in a single day on the same subject.  Without the time data in the filename, I would have duplicates.  More to the point, it would be difficult to match emails and attachments automatically, and I didn't want to go through that matching process for large numbers of emails.  I would also prefer a result in which emails converted to PDFs would appear in the right order in Windows Explorer, and that would require the time data.  As I was writing this recap, several days after writing the foregoing material, I was not entirely sure that I had verified that the output filename in UDC would include the time data.  But if that proved to be the case on revisit, at this point one option would be to buy UDC (or perhaps one of the other programs just mentioned) for purposes of producing properly named emails.  Another would be to export the list of emails to Index.csv (above) and to hope that this list would match the order in which ImportExportTools would export individual emails.  There would still be the possibility that such a program would sometimes fail to do what it was supposedly doing, perhaps without me noticing until long after the data from which I had exported and renamed various files would be long gone.

The Interim Solution

I decided that, at this point, I could not justify the enormous time investment that would be required to complete this project -- in particular, to manually print to PDF each attachment to each email, to combine those PDFs, and to match and merge them them with a PDF of the email message to which they had been attached.  This seemed like the kind of project that really had to await some further development in application software.  For all I knew, the kind of solution I was seeking already existed, and I was just waiting until the day when I would become aware of it.  It was not at all an urgent project -- I rarely consulted attachments for old emails, and almost never consulted them for prior years, where I was focusing my present attention.

I wanted to get those old emails out of Thunderbird.  I didn't like the idea of having all that data at the mercy of a relatively inaccessible program (i.e., I couldn't see those emails in Windows Explorer), and anyway I didn't want T-bird to be so cluttered.  It seemed that a good solution would be to focus on the emails themselves for now.  I would export them to EML format.  EMLs would continue to contain the attachments.  I would then zip the EMLs into a small number of files, each no more than a few gigabytes in size, perhaps on a year-by-year basis, and I would store them until further notice.  Before zipping, I would make sure the EMLs were named the way I wanted, and would print each of them to separate PDFs.  So then I would have at least the contents of the emails themselves in readily accessible format, and could go digging into the zip file if I needed an attachment.  If I did someday find a way to automate the task of combining the emails and their attachments into a single PDF, I would give those PDFs the same name as the current email-only PDFs, so that the more complete versions would simply overwrite the email-only versions in the folders where I would store them.

Export and PDF via Index.csv

I decided to try and see if the Index.csv approach would work for purposes of producing EMLs whose names contained all of the elements identified above (i.e., date, from, to, subject).  I had sorted the old emails in Thunderbird into separate folders by year.  I went to one of those folders in T-bird and sorted it in ascending date order.  Then I went into Tools > ImportExportTools > Export all messages in the folder > Just index (CSV).  This gave me what appeared to be a matching list of those messages, in that same order.  The number of lines in the CSV spreadsheet (viewed in Excel) matched the number of messages in that folder as stated in T-bird's status bar.

I wondered what would happen if I exported another Index.csv after sorting the emails in that T-bird folder in descending chronological order.  Good news:  the resulting index.csv produced in that experiment seemed to be reversed from the one I had produced in ascending order.  At least the first and last emails were in reversed positions.  So it did appear that index.csv matched the order that I saw in T-bird.

On that basis, I added an Index number column at the left end of the index.csv file I was working with, the one with emails sorted in ascending date order.  This index column just contained ascending numbers (1, 2, 3 ...), so that I could revert to the original sort order if needed.  I assumed that the list would continue to sort in proper date order, but I planned to revise the date field (presently in "7/4/1997 18.34" format) so that it could function for file sorting purposes (e.g., 1997-07-04 18.34).  I wasn't sure that the present and future date fields would always sort exactly the same.  I could have retained the existing date field, but I wasn't sure that it, itself, was reliable for sorting purposes:  would two messages arriving in the same minute always sort in the same order?

Now I exported the emails themselves:  Tools > ImportExportTools > Export all messages in the folder > EML format.  As partially noted above, these were named in Date - Subject - Number format.  I now did a search to try to figure out what that number signified.  It wasn't clear, but it seemed to be just randomly generated.  Too bad.  It would have been better if they had included the time at the start of that random number, and had put it immediately after the date, so that the EMLs would sort in nearly true time order.  (There could still be multiple emails on the same subject within the same minute, and T-bird didn't seem to save time data down to the second or fraction of a second.)  It seemed I would have to manually sort files bearing the same subject line and arriving or being sent on the same day.  There would surely be large numbers of files like that.  I now realized they would not at all be sorted correctly in Windows Explorer:  with only date (not time) data in the filename, a file arriving in the morning with a subject of Zebras would be sorted, in Windows Explorer, after a file arriving in the afternoon on the subject of Aardvarks, and if there were three on the subject of Aardvarks they would all be sorted together even if they had arrived at several different times of day.

Ah, but now I discovered that ImportExportTools had file naming options.  Silly me.  I had just overlooked that.  But there they were:   Tools > ImportExportTools > Options > Filenames tab.  I selected "Add time to date" and I chose Date - Name (Sender) - Subject format.  Now I tried another export of EMLs.  The messages now had names like this:

19970730-0836-Microsoft-Welcome!

I checked and, sure enough, that was a message from Microsoft on that date at 8:36 AM.  Suddenly the remainder of my job got a lot easier.  I went back to the Index.csv spreadsheet (now renamed as an .xls) and worked toward perfecting its match with the filenames produced by ImportExportTools.  There were two parts to this mission.  First, I had to rework the Index.csv data exported from T-bird so that it would match the filenames given to the EMLs by ImportExportTools.  Second, I would then use the spreadsheet to produce a batch file that would rename those files to the format I wanted.  This called for some spreadsheet manipulation described in another post.

Converting EMLs to PDF

Now I faced the problem of converting the exported EMLs to PDF, as distinct from the problem (above) of exporting PDFs from Thunderbird. 

I found that EMLs could be converted into TXT files just by changing their extensions to .txt, which was easy enough to do en masse with a program like Bulk Rename Utility.  That would permit them to be converted to PDFs without the rich text, if necessary, since it was a lot easier to find a freeware program that would do that (or, in my case, to use Acrobat) than to find one that would PDF an EML.  This appeared to be a viable solution for some of the older emails, which had apparently been through the wringer and were not showing much sign of having glorified colors or other rich text or HTML features.

Before proceeding with this, I decided to export all of the remaining EMLs from Thunderbird.  I knew I could read the EMLs and the TXTs (if I renamed them as that); I also knew I could reimport them into T-bird.  This seemed like a separate step.  I also decided that going back through the exporting process would give me an opportunity to write a cleaner post that would summarize some of the foregoing information.

Windows 7: Archiving Thunderbird Emails to Individual PDFs - Retry

I had a large number of emails in Thunderbird (an email program like Outlook, but open source freeware).  I wanted to export each of those emails to its own distinct PDF file with a filename containing Date, Time, Sender, Recipient, and Subject information in this format:

2011-03-20 14.23 Email from Me to John Doe re Tomorrow.pdf

In that example, I might ultimately eliminated the "from Me" part as understood, but of course other emails would be from John back to me, so for starting purposes I wanted all five of the fields just listed.  The steps I went through are described below.  There is a summary at the end of this post.

Recap and Development:  Converting Emails into EML Format Files with Preferred Filenames

So far, I had already worked through the process of exporting those emails to distinct EML files.  I had also used a spreadsheet to rename those EML files so that they would provide clearer and more complete information about the file's contents.  (I was using Excel 2003 for spreadsheeting.  OpenOffice Calc was now able to handle a million rows (i.e., to rename a million files), but it had not been stable for me.  One option, for those who had more than 65,000 EMLs and therefore couldn't work within Excel's 65,000-row limit, was to do part of the list at a time.)  This post picks up from there, summarizing a more streamlined approach to the steps described at greater length in the two previous posts linked to in this paragraph.

I had previously tried to begin with the Index.csv file exported from Thunderbird via ImportExportTools, but that had been a very convoluted and unsatisfactory process.  I did continue to use Index.csv, but my main effort was to work up a spreadsheet that would use and alter the filenames created when I exported EMLs from T-bird, also using ImportExportTools.  As described previously, I had developed some rules for automated cleaning of various debris from filenames, such as the underscores that ImportExportTools inserted in place of quotation marks and other characters.

To summarize the approach described in more detail in the previous post, I got the filenames from the folder where ImportExportTools had put them by using this command at the CMD prompt (Start > Run > cmd):  "DIR /b > dirlist.txt" and then I copied and pasted the contents of dirlist.txt into an Excel spreadsheet.  There I extracted the Date, Sender, and Subject fields from those filenames using Excel functions, including FIND, MID, TRIM, and LEN, all described in Excel's Help feature and in the previous post.  I also used Excel in a separate worksheet to massage the data on the individual emails as provided in Index.csv.

The two worksheets did not produce the same information, and I needed them both.  The one contained actual filenames, which I wanted to revise en masse to be more readable and to include the "To" field, which was contained in Index.csv.  Many of the things that ImportExportTools screwed up about the subject fields of emails, for purposes of CMD-compatible filenames (and going well beyond that) involved the underscore character.  Hence, the chief sections in the main worksheet (where I revised the data from dirlist.txt), going across the columns, were as follows:

Dirlist (raw EML filenames)
Date & time conversion (from 19980102-0132 to 1998-01-02 01.32
Subject: Clean up starting & ending underscores of From names
Subject: Replace "Re_ " with "Reply re" in Subject field
Subject: Replace "n_t" and "_s" (as in "don_t" and "Mike_s") with apostrophes
Subject: Replace serial underscores: "_ _" becomes " - "
Subject: Replace "I_m" with "I'm" and "you_re" with "you're"
Subject: Replace underscore and space ("_ ") with hyphen (" - ")
Subject: Remove starting and ending hyphens

That accounted for the bulk of the needed changes in the Subject field, in the files I was working with.  I set these rules up to eliminate the first one, or in some instances two, occurrences of the underscore string in question.  Few emails contained more than that; for those few, leaving the additional underscores in place was acceptable.  There would be some predictable misfires of these rules, but they would generally improve the situation, and when dealing with a large number of EMLs that I didn't intend to rename manually, this was the best that I could hope for under the circumstances.

Then I used VLOOKUP to search for a match with the Index.csv-style Date and Time (e.g., 19980102-0132) data in the Index.csv worksheet, and also for a match with the Index.csv Date+Time+From combination.  (Sometimes the From field was necessary to distinguish two or more emails sent at the same time.  Because of the underscores and other oddities about the EML filenames, subjects were too different to compare in most cases.)  This identified precise matches between the two worksheets for about 80% of EMLs.

So now I was going to try using that same spreadsheet with another batch of emails exported from Thunderbird.  I exported the Index.csv and the EMLs, and set to work on the spreadsheeting process of reconciling their names and producing MOVE commands for a CMD batch file that would automatically rename large numbers of EMLs to be readable and to include data from the To field.

This time around, I did a first pass to bulk-recognize and batch-rename that first 80% of the EMLs.  The CMD command format was this:

MOVE /-y "Old Filename.eml" "Renamed\New Filename.eml" 2> errlog.txt

This renamed the old EMLs to the desired new EML filenames, put them into the Renamed subfolder, and gave me an error log to say what went wrong with any of the renames.  The error log wasn't very useful, so I stopped creating it in these commands.  What I had to do instead, to find out which EMLs had been successfully renamed, was to do a dirlist.txt for the Renamed folder, feed that back into the spreadsheet, and delete those lines that had executed successfully.  For about 15% of the emails, I could not automatically detect matches between data from Index.csv and actual files, so I wound up naming those files according to date, time, and sender only.  Finally, I got down to less than 1% of emails that I had to rename in a more manual fashion, mostly due to non-ASCII characters in their filenames.  For that, I used Bulk Rename Utility.

I was not sure whether this route wound up being better than the approach of using one of the shareware programs discussed in the previous post.  I was not aware of the potential difficulties when I was looking at those programs, so for example I didn't try them out on emails with Chinese characters in their Subject fields.  The other way always looks easier after a project like this.  The approach I had taken had surely been more time-consuming than if I had known of a killer app that would do exactly what I wanted without unanticipated complications or failures.  Absent a reliable, obvious solution at an affordable price, the main thing I could say at this point was that at least the conversion to EML was done.

Final Step:  Converting EMLs into PDF

With EMLs thus exported from Thunderbird and mostly renamed to indicate date, time, sender, recipient, and subject, the remaining task was to convert the EMLs to PDF.  This, it developed, might not be as simple as I had hoped.  There was, first, the problem of finding a program that would do that.  Some of the emails were simple text and could have been easily converted to TXT format just by changing their extensions from .eml to .txt.  Acrobat and other PDF programs would readily print large numbers of text files, unlike EMLs.  Other EMLs, however, contained HTML (e.g., different fonts, different colors of print, images).  I wasn't sure what would happen if I changed their extensions and then printed.  I noticed that the change to .txt caused the HTML codes to become visible in one message that I experimented with.  When I converted that file to PDF using Acrobat, its header appeared in a relatively ugly form, but the colors and fonts seemed to be at least somewhat preserved.  In another case, though, the PDF was largely a printout of code -- a truly undesirable replacement for what had been a pretty email with photos included.  My version of Acrobat (ver. 8.2) did not provide any editable settings for conversion from text or HTML to PDF. 

Thunderbird was my default program for displaying EMLs.  I wondered if a different program could view them and would have better PDF printing capabilities, or if I should try converting them into another interim format in order to then convert them to PDF.  A search led to the claim that Microsoft Word (or other programs) could display EMLs.  I tried and found that this was essentially untrue:  in Word, there was almost nothing left of that pretty email I had just tested.  Converting EML to MSG seemed to be one option, but this looked like a dead end; that is, it didn't look like it would be any easier to PDF an MSG file than to PDF an EML.  Getting the EMLs into Outlook wasn't likely to be a solution; as I recalled, my version of Outlook (2003) had been unable to batch print emails as individual PDFs.  Marina Martin said that MBOX was the standard interoperable email file type.  I could have exported from Thunderbird directly to MBOX using ImportExportTools, but I had not investigated that; I had assumed that MBOX meant one large file containing many emails, like PST, and I had wanted to rename my emails individually.  Martin gave advice on using eml2mbox to convert EML to MBOX; hopefully I would not have lost anything by taking the route through EML format.  But if MBOX was such a common format, there was surprisingly little interest in converting it to PDF.  My search led to essentially nothing along those lines.  Well, but couldn't Firefox or any other web browser read HTML emails?  I tried; neither Firefox nor Internet Explorer were willing to open an EML.  I renamed it to be an .html file.  Both opened that, but here again the problem was that the header was so ugly and hard to read:  it was just a paragraph-length jumble of text mixing up the generally important stuff (e.g., from, to) with technical information about the transmission.  Even assuming I could work out a batch-PDF process for HTMLs, this was not the solution.  There were other possibilities, but in the end it did appear that I simply needed to buy an EML-to-PDF converter.

It tentatively appeared that MSGViewer Pro ($70) might be the most frequently downloaded program in this area, ahead of its own sister program PSTViewer Pro as well as Total Mail Converter.  A search for reviews led to very little.  It didn't appear that MSGViewer Pro had the ability to include image attachments within the PDF of an EML, as Total Mail Converter Pro ($100) supposedly did.  On the other hand, MSGViewer Pro supposedly provided a free five-day trial.  I decided that I did not have time to mess with endless numbers of attachments right now, and was therefore willing to just zip the EMLs into a single file for possible future processing, if I decided that there was sufficient need and time for that.  Given my unlikelihood of using these programs very often, I also hoped that their prices would drop.  I figured that if the MSGViewer Pro trial was fully functional, I might be able to take care of my need for it now, converting EMLs into PDFs without attachments, and otherwise let the matter sit for another year or more.

On that basis, I downloaded and installed MSGViewer Pro.  It was apparently designed for an older version of Windows.  When I installed it, I got one of those Win7 messages indicating that it might not have installed properly, and inviting me to reinstall using "recommended settings," whatever that meant.  I accepted the offer.  Once properly installed, I ran the program.  A dialog came up saying, "Trial is not licensed for commercial use."  I clicked "Run Trial."  Right away, I found that its Refresh feature did not work:  I copied some EMLs into a separate folder to experiment with, and could not get the program to find that folder.  I killed the program and started over.  Now it found the folder.  I selected those messages, clicked the Export button, and told it to give the resulting PDF (one of the available output options; the others were TXT, JPG, BMP, PNG, TIFF, and GIF) the same names as the input files.  It had a nice option, which I accepted, to copy failed messages to a separate folder.  A dialog came up saying, "You can only export 50 emails in trial version of MsgViewer Pro."  So that popped that fantasy.  It ran pretty quickly and reported that all of the files had been successfully exported.  Sadly, the results were no better-looking than I had been able to achieve on my own, with other measures described above.  HTML codes were visible in some PDFs -- or perhaps I should say, not visible, but overwhelming:  it looked like a piece of ordinary HTML coding.  The typeface was tiny.  Some lines were actually split down the middle horizontally, with the top half of a line of text appearing at the bottom of one page and the bottom half appearing at the top of the next page.  In a word, the results were junk.  I uninstalled MSGViewer Pro.

I decided to try Total Mail Converter Pro.  No installation problems.  When installation ended, the program started right up without giving me a choice.  Then it decided I needed to log onto Gmail.  This was not my plan, so I canceled that.  I liked its interface better than MSGViewer Pro:  smaller but still readable font, seemingly more options.  I selected my test files and clicked the PDF button.  It gave me options to combine the files into one PDF or produce separate files.  It also provided a file name template, with choices of subject, sender, recipient, date, and source filename.  I tried these.  There were other options:  which fields to export, whether to include attachments in the doc or put them in separate folders, header, footer, document properties.  It did the conversion almost instantly.  The date format was month-day-year.  The subject data weren't cleaned up, so I would still have had to go through something like my spreadsheet process to get the filenames the way I wanted them.  Moment of truth:  the file contents included a colored top part, as I had encountered with Birdie (see previous post). HTML codes were still visible in some messages, but in others the HTML seemed to have been better converted into rich text.  Typefaces were still tiny.  Definitely a better program.  But worth $100 for my needs?

Ideally, I would have been converting my emails to PDF as I went along, without converting them around and around, from Outlook to Thunderbird to EML and wherever else they might have gone over the past several years.  This might have better preserved what I recalled as the colorful, more engaging look of some of them, and perhaps I would have come up with better ways of capturing those characteristics as I continued to become more experienced with the process.  In the present circumstances, where I really just wanted to get the job done and move on, it seemed that playing with that sort of thing was not a short-term option.

Since I was planning to keep the EMLs anyway, and since I did not plan to view these emails frequently, I decided that I really didn't lose much in informational terms by going with the free option identified above.  I took a larger sample of EMLs and, using Bulk Rename Utility, renamed them to be .txt files (though later I realized I could have just said "ren D:\Documents\*.eml *.txt").  Since I had installed Adobe Acrobat, I had a right-click option to convert to Adobe PDF.  No doubt some freeware PDF programs provided similar functionality.  The Acrobat conversion of these files into PDF was not nearly as fast as that performed by Total Mail Converter Pro.  Acrobat put each of those newly created PDFs onscreen and obliged me to manually confirm that I wished to save them.  I had converted 40 files, and wasn't interested in manually closing all 40; ultimately I had to use Task Manager to shut them down.  That problem turned out to be just a result of the settings I was using for my default Bullzip PDF printer; changing those defaults and using Acrobat's Advanced > Document Processing > Batch Processing option made the process completely automatic.  In terms of appearance, it seemed the fonts, HTML handling, and other features were more or less the same as I had gotten from those other programs (above).  I probably could have made the average resulting email more readable (except where HTML formatting made clear who was responding to whom) by looking for a program that would strip the HTML codes out of those TXT files, but I didn't feel like investing the time at this point and wasn't sure the effort would yield a net improvement.

Briefly, then, the PDFing part of this process involved using a bulk renamer to replace the .eml extension with a .txt extension, and then using a bulk PDF printer or converter to convert those TXT files into PDF.  This approach still preserved the look of some emails, while allowing others to be overrun with HTML codes.

I ran that batch process on a full year's set of EMLs.  I converted 1,422 EMLs into TXT files by changing their extensions with Bulk Rename Utility.  Somehow, though, Acrobat produced only 689 PDFs from that set.  Which ones, and what had happened to the rest?  Acrobat didn't seem to be offering a log file.  My guess was that Acrobat went too fast for Bullzip.  There was no real reason why I shouldn't have been using Acrobat's own PDF printer for this particular project -- in fact, I did not remember precisely what Acrobat snafu had prompted me to switch to Bullzip as my default PDF printer in the first place -- so I went into Start > Settings > Printers and made that change now.  I also right-clicked and changed some of the Printing Preferences, for that printer, so that it would run automatically.  I deleted the first set of PDFs and tried again.  I noticed, this time, that Acrobat was not even trying to convert more than 689 files -- it was saying, "1 of 689," "2 of 689," etc.  What was causing it to overlook these other files, I was not sure.  It seemed I would have to do a "DIR /b > Printed.txt" command in CMD, and then convert Printed.txt into a Deleter.bat file that would delete the text files that were successfully printed, so as to highlight the ones that remained.  (See previous post for details on these sorts of commands.)

(Incidentally, I had also noticed, now that I was working with the Acrobat batch options, that it had a "Remove File Attachments" option.  While it did not seem to work with EMLs, possibly it would have been useful if these emails had been in MSG or PST format.)

The automated process got as far as file no. 2 in the list before it stalled.  Why it stalled, I had no idea.  I clicked on the X at the top right-hand corner of the dialog to kill it -- I even said "Close the Program" when Windows gave me that option -- and then Acrobat took off and printed a couple hundred more PDFs before stalling in that same way again.  Possibly I had the Acrobat PDF printer's properties set to stop on encountering an error.  I ran through most of that first set before spacing out and killing Acrobat (the whole program) at a stall, instead of just killing the stalled task.  I deleted those that had printed successfully, creating a Deleter.bat file for the purpose as just mentioned, and ran another batch.  This time, Acrobat was printing a total of 667 files.  So I figured the situation was as follows:  Acrobat would print PDFs through a glorified command-line kind of process, and that command line would accommodate only so many characters.  If I'd had shorter file names, maybe it would have been willing to print thousands of TXT files at one go.  If I had wanted to add complexity to the process, I could have renamed my files with names like 0001.txt, reserving a spreadsheet to change their names back to original form after conversion to PDF.  But with my filenames as they were, it was only going to process 600 or 700 at a time.  That was my theory.

When Acrobat was done with the second set -- the first one that had run through to completion -- it showed me a list of warnings and errors.  These were errors pertaining to maybe a dozen files.  The errors included "File Not Found" (typically referring to GIFs that were apparently in the original email), "General Error" (hard to decipher, but in some cases apparently referring to ads that didn't get properly captured in the email), and several "Bad Image" errors (seemingly related to the absence of an image that was supposed to appear in the email).  A spot check suggested that the messages with these errors tended to be commercial (e.g., advertising) messages, as distinct from personal or professional messages that I might actually care about.  In a couple of cases a single commercial email would have several errors.  But anyway, it looked like they were being converted, with or without errors.

I decided to try printing the next batch with Bullzip instead of Acrobat printer.  I had to set it as the default printer in Settings > Printers.  I also had to adjust its settings (by going to its Options shortcut in the Start Menu > General and Dialogs tabs) so that it would run without opening dialogs or completed PDFs.  Would it now process significantly more than 600 input files?  The short answer:  no.  So for the next round, I tried selecting all the TXT files in a folder and right-clicking > Convert to Adobe PDF.  This was a bad idea.  Now Acrobat wanted to open a couple thousand documents onscreen.  I had to force-reboot the system to stop this one.

So now I thought maybe I'd look for some other text-to-PDF converter.  It sounded like ActivePDF was a leading solution for IT professionals, but I didn't care to spend $700+.  Shivaranjan recommended Zilla TXT To PDF Converter ($30).  Softpedia listed a dozen freeware converters, of which by far the most popular was Free EasyPDF. But I couldn't quite figure out what was going on there.  There was no help file, and the program wasn't even listed on its supposed creator's webpage.  CNET called it fatally crippled.  I didn't know why 30,000 people would have downloaded it.  Back to Softpedia's list:  Free Text to PDF Converter was another possibility with a Good rating.  Its webpage said it could batch-convert text to PDF files.  I went into its Open option, selected a boatload of TXT files, and saw no sign that it had any intention of doing anything with them.  Looking more closely at its starting screen, I saw it said this:

Command Line usage:
TXT2PDF <inputfile> <output.pdf> [parameter table]

The documentation webpage said I was supposed to drag the TXT files into the window on the main screen to convert them.  It also said this program would convert only plain text, not HTML.  I wasn't sure what that meant for the EMLs that contained HTML code as plain text.  The optional parameters had to do with font, paper size, etc.  In the folder where I had my TXT files to be converted, I tried this command:

"C:\Program Files\Text2PDF v1.5\txt2pdf.exe" "Text File to Be Converted.txt"

with quotation marks as shown, on the command line.  It worked.  It produced a PDF.  There was no word wrap, so words would just break in the middle at the end of the line, like this:

We can't pledge that we've entirely emerged from th
at episode, but this
past summer I sat down and rewrote the entire man
ual in a way that makes
more sense. The guy just didn't know how to phrase

The print size was very large, though there were parameters to change that, but nothing, apparently, to persuade lines to break at the ends of words rather than in the middle.  This could defeat Copernic text searching, rendering some PDF file contents unfindable, so it wasn't going to be a good solution for me.  But it really seemed like the command line approach, which would let me name each file to be converted, was the answer to the problem of being able to process only ~600 text files at a time.  Another possibility:  AcroPad.  The following command worked:

Acropad "File to Convert.txt" "File Converted.pdf" Courier 11

I could have named other typefaces and font sizes.  Output was double-spaced.  Lines were broken at the ends of words, not in the middle.  HTML code in the file was just treated as text and printed out as-is.  I kept searching.  A post by Adam Brand said I could use a command to automate printing if I had Acrobat Reader installed.  That prompted another search that led to several insights.  First, it turned out I could print a file from the command line using a Notepad command in the form of "notepad.exe /p filename."  Since my default printer was a PDF printer, it printed a PDF -- a nice one, too, for basic purposes, nicer than some of the output I was getting from the programs tested above.  It put the output on the desktop.  I changed the location for the output by going into the Desktop folder for my username.  Since I was running as Administrator, the location was C:\Users\Administrator\Desktop.  There, I right-clicked on the Desktop folder, went to Properties > Location tab and changed it.  (Another Notepad option, which I didn't need, was to specify which printer I wanted to use:  /pt.)

The Notepad approach did nothing with HTML codes in these plain text files.  An alternative that would work with rich text, which might or might not help in my case, was supposedly to try the same switch with Wordpad:  "wordpad.exe /p filename."  But when I did that, I got an error message:

'wordpad.exe' is not recognized as an internal or external command, operable program or batch file.

This was odd.  To fix it, I ran regedit (Start > Run) and went to
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\App Paths\.  There, following instructions, I right-clicked on App Paths and selected New > Key.  I named the new key "Wordpad."  I right-clicked on that Wordpad key and selected New > Expandable String Value.  It apparently didn't matter what I called it.  I called it ProgramPath.  I right-clicked on ProgramPath and pasted in the path where Wordpad was, which I had obtained by going into the Properties of the Wordpad shortcut on my Start Menu.  In other words, what I entered here included quotation marks and the name of the executable wordpad.exe, with extension.  The instuctions said that, to run Wordpad from the command line (as distinct from in Start > Run), the command would have to begin with the Start command.  For present purposes, what I would type at the C prompt would be "start wordpad /p filename."  This worked (and I exported the new registry key and added it to my Win7RegEdit.reg file for future installations), but it did not produce a superior PDF compared to that which Notepad had produced, and for some reason it truncated the filename of the resulting PDF.

Revised Final Step:  Converting TXT to HTML to PDF

Searching onward, there was a possibility of treating them as HTML rather than TXT files.  I had flirted with this earlier but had not grasped that, of course, these actually were HTML files in the first place; they had become EMLs and TXTs only later.  I typed "ren *.txt *.htm" to rename them all as HTML files.  To print them, there were some complicated approaches, but I hoped that PrintHTML.exe would do the trick.  The syntax, for my purposes, was this:

printhtml.exe file="filename.htm"

with optional leftmargin=1, rightmargin=1, topmargin=1, and bottommargin=1 parameters, among others that I didn't need.  The printhtml.exe file would of course have to be in the folder with the files being printed unless I wanted to add it to the registry as just described for Wordpad.  PrintHTML wouldn't work until I installed the DHTML Editing Control.  I did all that, and got no error messages, but also did not seem to get any output.  I decided to put that on hold to look at another possibility:  automated PDF printing using Foxit Reader on the command line.  Pretty much the same command syntax as above:

"FoxitReader.exe" /p filename

Here, again, there was a need for a registry edit, unless I wanted to park a copy of Foxit in every folder where I would use it from the command line.  But the instructions were only for using Foxit to print PDFs, so I got an error:  "Could not parse [filename]."  There was also an option of using Acrobat Reader to print a PDF silently or with a dialog box, but there again it wasn't what I needed:  I was printing HTMLs.  I returned to that printhtml.exe program mentioned above.  The command ran, with no indication any errors, but there did not seem to be any output.  Another possibility was:

RUNDLL32.EXE MSHTML.DLL,PrintHTML "Filename.htm"

But for me, unfortunately, that produced an empty PDF.  Turning again to freeware possibilities, I found an Xmarks list of top-ranked HTML to PDF programs.  Most of the top-ranked items were online, one-file-at-a-time tools.  Others required PHP knowledge that I didn't have (e.g., HTML_ToPDF, PDF-o-Matic).  HTMLDOC looked promising for command-line usage; I found its manual; but when I downloaded and unzipped it, I couldn't find anything that looked like a setup or installation file.  Apparently the version that's free is the source code, and I didn't know how to compile it.   DomPDF and html2pdf (and, I suspect, some of these others) were apparently for Linux, not for Windows.  I tried wkhtmltopdf.  When I ran it, I got an error:

wkhtmltopdf.exe - System Error
The program can't start becuase libgcc_s_dw2-1.dll is missing from your computer.  Try reinstalling the program to fix this problem.

Possibly the reason I got that error is that I was trying the same trick of running the program in a folder where my PDF files were.  I had copied the executable (wkhtmltopdf.exe) to that folder, but had not brought along its libraries or whatever else it might need.  I tried running it again -- I was just trying to use the help command, "wkhtmltopdf -- help" -- but this time pointing to the place where the program files were installed:

"C:\Program Files\wkhtmltopdf\wkhtmltopdf.exe" -- help

and that worked.  I got a long list of command options.  What I understood from it was that I wanted, in part, a command like this:

"C:\Program Files\wkhtmltopdf\wkhtmltopdf.exe" -s Letter "File to be converted.htm"

I tried that.  It gave me an error:

Error: Failed loading page http: (sometimes it will work just to ignore this error with --load-error-handling ignore)

So I tried adding that long parameter to the command.  It seemed like it worked:  it gave the error message but then proceeded through the rest of its steps and announced, "Done."  But I didn't see any output anywhere.  Then I realized there was an error in what I had actually typed.  I tried again.  This time, it gave me a different error message:  "You need to specify at least one input file, and exactly one output file."  So the format I was supposed to use, aside from that additional "--load-error-handling ignore" parameter, was this:

"C:\Program Files\wkhtmltopdf\wkhtmltopdf.exe" -s Letter "HTML file to be converted.htm" "New PDF File.pdf"

And that worked.  At last, I had a mass-production way of converting EMLs (by changing their extension to .htm, not .txt) to PDFs.  It was too early to break out the champagne, but at least the computer and I were back on speaking terms.  Now I just needed to run "DIR /s /b > dirlist.txt" in the top-level folder under which I had sorted my emails, convert that dirlist.txt file into a .bat file that would convert the file listings into batch commands, and run it.  I was afraid the whole command, with the introductory reference to C:\Program Files, would be too long for Windows in some cases, so I edited the registry as described above, so that I would only have to type wkhtmltopdf.exe at the start of each command line.  But now that registry edit wasn't working -- it certainly seemed to previously -- so I copied all of the wkhtmltopdf program files to the folder where I would be running this batch file.  I didn't want the computer to crash itself by opening hundreds of simultaneous wkhtmltopdf processes, and I wanted to move the PDFs, so the format I used for these commands was:

start /wait wkhtmltopdf -s Letter "D:\Former Directory\HTML file to be converted.htm" "D:\New Folder\New PDF File.pdf"

That worked.  Now I investigated the longer list of wkhtmltopdf command-line options, by typing "wkhtmltopdf -H" (with a capital H).  Whew!  The list was so long, I couldn't view it in the cmd window -- it scrolled past the point of recall.  I tried again:  "wkhtmltopdf -H > wkhtmltopdf_manual.txt."  I couldn't add too much to the command line -- I was already afraid the long filenames would make some commands too long for CMD to process.  But having viewed some output of these various PDFing programs, a few sets of commands seemed essential, including these:

-T 25 -B 25 -L 25 -R 25
--minimum-font-size 10

The first set would give me one-inch margins all around.  Putting these on the already long command line increased my interest in another option:  --read-args-from-stdin.  This one, according to the manual, would also have the advantage of speeding up the process, since I would be starting wkhtmltopdf just once, and then re-running it with different arguments.  The concept seemed to be that my conversion batch file (or, really, just a typed command) would contain this:

start wkhtmltopdf --read-args-from-stdin < do-this.txt

and then do-this.txt would contain line after line of instructions like this one:

-T 25 -B 25 -L 25 -R 25 --minimum-font-size 10 -s Letter "D:\Former Directory\HTML file to be converted.htm" "D:\New Folder\New PDF File.pdf"

Or perhaps they could be rearranged so that some of the contents of the second could be in the first, and therefore would not have to be repeated on every line in do-this.txt.  In which case the main conversion command would look like this:

start wkhtmltopdf --read-args-from-stdin -T 25 -B 25 -L 25 -R 25 --minimum-font-size 10 -s Letter < do-this.txt

and do-this.txt would contain only the "before" and "after" filenames.  I decided to try this approach.  Unfortunately, it didn't work.  It froze.   So then I tried just the minimal one shown a moment ago, putting all options except --read-args-from-stdin in the do-this.txt file.  Sadly, that froze too.  I tried the minimal command plus just filenames, leaving out the several additional commands about margins and font size.  Still no joy.  So, plainly, I did not understand the manual.  I decided to go back to the approach of just putting it all on one line and repeating all commands, in a batch file, for each HTM file that I was converting to PDF.  Each line would begin with "start /wait," not just "start," for reasons stated above.  This worked, but now I noticed a new problem that I really hadn't wanted to notice before, because I just wanted this project to be done already.


Separating EMLs With and Without HTML Code

The new problem was that emails that were originally in HTML format turned out best when they were now renamed with an .htm extension, and processed that way, but the ones that didn't have HTML codes in them were now reduced to a mess.  Specifically, line and paragraph breaks were gone; everything was just jumbled together in one continuous stream of text.  Every non-HTML email was now being represented by a single long paragraph.  To get decent output, it seemed that I needed to separate the emails that contained HTML code from those that did not.  I would then use wkhtmltopdf with the former, but not with the latter.  But how could I tell whether a file contained HTML code?  I decided that an occurrence of "</" would be good enough in most cases.  But then it occurred to me that there might be programs that would sort this out for me.  A search led to the FileID utility.  Their read-me file led me to think that this command, entered in the top-level folder where the files to be checked, might do the job:

"D:\FileID Folder\fileid" /s /e /k /n

This would run FileID from the folder where its program files were stored, and would instruct FileID to check all files in all subdirectories, to automatically change file extensions to match contents, to delete null files, and not to prompt me for input.  But it did not seem to be working.  Regardless of whether I entered these options as upper- or lower-case (e.g., /S or /s), FileID paused after every screenful of information, and did not seem to be renaming anything.  I decided to try again with another command-line program of similar purpose, TrID.  TrID had an online version and a GUI.  On second thought, I decided to give the GUI version a whirl.  I downloaded the program and its XML definitions.  (I already had the necessary .NET Framework installed.)  As advised by Billy, I moved everything from the XML definition folder (after unzipping them with WinRAR) into the folder containing the TrIDNet.exe file.  I doubleclicked on that executable and saw that it would process only one file at a time.

I moved on to the command-line version.  This called for a download of a different set of program files and definitions.  I wasn't sure whether TrID would actually change incorrect extensions, or just detect them.  Again, rather than plow into the support forums, I just tried it out.  But in this case, that strategy didn't work:  there was no manual or other use instructions in the download.  The forum contained a tip on using PowerShell to fix extensions, but I didn't know enough about PowerShell to be able to interpret and adapt that tip to my situation.  But, silly me, I forgot about just getting online help.  In the folder where I had unzipped TrID.exe, I opened a cmd window and typed "trid -?" and got the idea that I could type "trid -ce" or perhaps "trid *.* -ce" to have the program change file extensions as needed, for all files in the current directory.  It didn't appear to have a subdirectory option, so I would have to do some file moving.

A different approach was to use a CHK recovery program to detect the proper extension for anything with a CHK extension.  While FileCHK looked like the better program for recovering real CHK files, it looked like UnCHK would have more flexibility for my situation, provided I first ran "ren *.htm *.chk" to change the file extensions to .chk.  When I tried to run unchk.exe, I got an error message:

The program can't start because MSVBVM50.DLL is missing from  your computer.  Try reinstalling the program to fix this problem.

Eric had already warned me, in the read-me file, that this meant I needed to download and install the Visual Basic 5 runtime.  I did, and tried again.  Now it ran.  I couldn't find documentation or a /help option to explain its settings.  It took me a while to realize it wasn't a command-line program, though it could run from the command line.  It was very bare-bones.  I started it, navigated to the first of the folders I wanted to repair, and (having renamed files to have .chk extensions), gave it a try.  It gave me a dialog asking about Scan Depth.  I knew from the read-me that I wanted the Whole Files option.  It ran for a while and then disappeared.  It didn't seem to have done anything.  After some more searching around, I concluded that this CHK approach wasn't what I wanted.

So I looked elsewhere.  If I wanted to spend a day or so refreshing my aging knowledge of BASIC programming, or invest some time in learning more about batch scripting or Microsoft Access or some other program, I was pretty sure I could work up a way to examine file contents.  But I wanted a solution faster than that, if possible.  The CMD batch FIND command looked like it might do the job.  But the command that I thought should work,

FOR %G IN (*.txt) do (find /i "</" "%G")

didn't.  It wasn't because "</" were weird characters; it wasn't finding files containing ordinary text either.  I tried again with the FINDSTR command:

findstr /m /s "</" *.* > dirlist.txt

This looked promising.  But when I examined dirlist.txt, I saw that many of the files listed in it were better presented as TXT than as HTM.  Apparently I should have been looking for files with more substantial HTML content.  A spot check of several emails suggested that the existence of an upper- or lower-case "<html" might be a good guide.  So apparently I would have to run FINDSTR twice:

findstr /m /s "<HTML" *.* > dirlist.txt
findstr /m /s "<html" *.* >> dirlist.txt 

with two ">" symbols in the second one, so as to avoid overwriting the results of the first search with the results of the second.  I tried that.  There were some error messages, "Cannot open [filename]," apparently attributable to weird characters in the file's name; somehow it seemed I had still not entirely succeeded in cleaning those up.  I assumed FINDSTR's failure in this regard would leave those files being treated as TXT by default, which would probably be OK since the majority of files overall appeared to be non-html.  Ultimately, dirlist.txt contained a list of maybe 40% of all of the emails I was working on.  That seemed like it might be about right.  In other words, it seemed that about 60% of the emails were best treated as plain text, and I would be getting to those shortly.  I put dirlist.txt into a spreadsheet to produce commands that would run wkhtmltopdf on the files that those two commands listed in dirlist.txt.  The key formula from that spreadsheet:

="start /wait /min wkhtmltopdf -T 25 -B 25 -L 25 -R 25 --minimum-font-size 12 -s Letter "&CHAR(34)&B1&"\"&C1&".htm"&CHAR(34)&" "&CHAR(34)&"..\Converted\"&C1&".pdf"&CHAR(34)

That formula, applied to each file identified as containing "<html," produced PDFs that looked relatively good.  I found that I needed a way of testing them, though, because in a number of cases wkhtmltopdf had produced PDFs that would not open.  I also noticed that the batch file running these commands kept acting like it had died. Windows would say, "wkhtmltopdf.exe has stopped working," and I would click the option to "Close the program." And then, after a while, it would come roaring back to life.  This may have happened especially when wkhtmltopdf was converting simple email messages into PDFs of a thousand pages or more.  A thousand pages of gibberish.  In a number of cases, too, the resulting PDF was a failure.  When I tried to open those PDFs, Acrobat said this:

There was an error opening this document.  The file is damaged and could not be repaired.

I was not sure what triggered these problems.  I wondered if possibly the simpleminded conversion from EML to HTM by merely changing the extension caused problems in the case of EMLs that contained attachments.  If that was the case, then what I should have done might have been to export from Thunderbird in HTML format in the first place -- to do two exports, in other words:  one for EMLs, which would include attachments, to be zipped up into an archive and shelved until the future day when there would be a simple, cheap solution for the PDFing of emails plus their attachments; and another export in HTML, for purposes of PDFing here and now, without attachments.  I tested this with one of the gibberished emails and found that, when exported from T-bird as HTML using ImportExportTools, it did print to a good-looking PDF.  In that approach, the naming procedures used to rename the exported emails in the desired way -- containing date, time, sender, recipient, and subject information -- would apparently have to be preserved and reapplied, so that both exported sets -- the EMLs and the HTMLs -- would be named as desired.

To investigate these questions, I traced back one PDF that did not open -- that produced the error message quoted above -- and one that opened but that was filled with gibberish.  The one that was damaged did not come from an email that originally contained attachments.  I was able to print that email directly from Thunderbird without problems.  So I wasn't sure what the problem was there.  For a sample of one filled with gibberish, I chose the largest of them all.  This was a 3,229-page PDF that was produced from a little two-page email that did originally have an attachment.  I sampled three other PDFs containing gibberish.  All three had come from emails that originally had attachments.  So it did appear that attachments were foiling my simplistic approach of just changing file extensions from EML to HTM.  I wondered if it was too late to just change the extensions back to .eml, for the ones that had not produced good PDFs, and maybe PDF them manually.  I tried with one, and it worked.  So that would have been a possibility, assuming I had time for printing emails one by one.

It seemed the gibberish might not be gibberish after all.  It might be a digital representation of the photograph or whatever else was attached to the email.  I didn't know of a way to test text for gibberish, so this didn't seem to be a problem that I could deal with very effectively at this point.  I could name some files as HTM, as I had done, and just accept a certain amount of gibberish -- perhaps after screening out the really large PDFs (or, earlier in the process, the large EMLs, TXTs, or HTMs), which seemed most likely to have had attachments -- or I could rename them all as TXTs and print them that way, looking solely for the text content without regard to their appearance (and still probably getting gibberish).  If I needed to know how they looked originally, I would have to go back to the archived EML version of the PDFd text.  A third option was to go back to T-bird and re-export everything as HTML, thereby skimming off the attachments, and then use my saved renaming spreadsheets to rename the newly produced, roughly named HTMs, and then do my PDFing from those new HTMs.  Presumably, that is, the new HTMs would print correctly, since they would not have attachments.

Back to the Drawing Board:  T-Bird to HTML to PDF

I decided to try that third option.  I went back to Thunderbird and used ImportExportTools to export the emails as HTML rather than as EML.  It would have been more logical to start by PDFing these HTMLs, to make sure that would work; but at this point I had such a clutter of emails in various formats that I decided to proceed, as before, with the renaming process first, so as to be able to delete those that I wasn't going to need.  Having already worked through the process of renaming to the point of achieving final names, I used directory listings and spreadsheets to try to match up the "before" names (i.e., the names of the raw HTML exports) and the "after" names (i.e., the final names I had developed previously). 

Once I had the emails in individual HTML files with workable filenames, I ran wkhtmltopdf again.  I started by taking a directory listing of the files to be converted; I put those into a spreadsheet, as before; and in the spreadsheet I used more or less the same wkhtmltopdf formula shown above in order to produce working commands.  These pretty much succeeded.  I was now getting good PDFs from the emails.  It seemed that wkhtmltopdf had a habit of wrapping lines severely or perhaps indenting them too much.  That is, if I wrote an email in reply to someone else, the text of my email would look fine,

but the text of the message

to which I was replying,

typically shown below the

reply text, would be

indented and then broken

like this.

Wkhtmltopdf converted HTML files to PDF at a rate of somewhat more than one email per second.  Of course, these were small files, as email messages tend to be.  There was a problem with them taking up a lot of disk space; it seemed I might have been well-advised to format the drive to have smaller than the default cluster size.  The program slowed down considerably at times.  I assume it was running into complexities with some HTML files.

The batch file ran and finished, but it had converted only about half of the HTMLs into PDFs.  I decided to test the PDFs before deleting the corresponding HTMLs.  I opened a half-dozen of them without a problem.  Then, for a more thorough test, as described in a separate post, I ran an IrfanView batch conversion from PDF into RAW format.  I chose RAW because it would result in just one file.  TIF might have been another possibility.  It did appear that this process was all working well.  Ultimately, these steps converted all of the HTMLs into PDFs. 

Summary

The first part of what I was able to achieve, at this point, was to export my emails from Thunderbird to EML format, using the ImportExportTools add-on for Thunderbird.  Once I had exported all those EMLs, I used a zipping program (either WinRAR or 7zip) to bundle them together into a single file containing all of a year's emails.  I took these steps because EML files, unlike HTML, PDF, JPG, TXT, or other formats, were able to contain email attachments along with the text of the email messages.  I planned to keep these year-by-year ZIPs of EMLs until some point when I could find a cheap and broadly accepted program for printing both the email message and its attachment into a single PDF.

The other main achievement was to work out a process for converting HTMLs (also exported from Thunderbird via ImportExportTools) into PDFs.  I used wkHTMLtoPDF for this purpose.  I ran it in a batch file, produced by a spreadsheet, so that there was one command per file.  I used DIR folder comparisons and other means to test that all files were being converted and that they were being converted into valid PDFs.