Showing posts with label file. Show all posts
Showing posts with label file. Show all posts

Saturday, March 17, 2012

Microsoft Word: "Attempting to open a file created in earlier version"

I was working in Microsoft Word 2003.  I tried to open a .doc file.  It may have been originally written in WordStar, XyWrite, WordPerfect for DOS, or some other word processing program.  Word gave me an error message:

You are attempting to open a file that was created in an earlier version of Microsoft Office. This file type is blocked from opening in this version by your registry policy setting.
Of course, this was not what I wanted to hear.  I did a search and found a Microsoft webpage that, for my purposes, led to another webpage that led me to four REG file downloads whose contents I combined in a single REG file that I named MicrosoftOfficeUnblock.reg, whose contents were as follows:
Windows Registry Editor Version 5.00

; Unblock Word

[HKEY_CURRENT_USER\Software\Microsoft\Office\11.0\Word\Security\FileOpenBlock]
"FilesBeforeVersion"=dword:00000000


; Unblock Excel

[HKEY_CURRENT_USER\Software\Microsoft\Office\11.0\Excel\Security\FileOpenBlock]
"LotusandQuattroFiles"=dword:00000000
"LegacyBinaryFiles"=dword:00000000
"LegacyDatabaseAndDatasourceFiles"=dword:00000000

[HKEY_CURRENT_USER\Software\Microsoft\Office\11.0\Excel\Security\FileSaveBlock]
"LotusandQuattroFiles"=dword:00000000
"LegacyBinaryFiles"=dword:00000000
"LegacyDatabaseAndDatasourceFiles"=dword:00000000


; Unblock PowerPoint

[HKEY_CURRENT_USER\Software\Microsoft\Office\11.0\PowerPoint\Security
\FileOpenBlock]
"FilesBeforePowerPoint97"=dword:00000000

[HKEY_CURRENT_USER\Software\Microsoft\Office\11.0\PowerPoint\Security
\FileSaveBlock]
"FilesBeforePowerPoint97"=dword:00000000


; Unblock Corel Draw

[HKEY_LOCAL_MACHINE\Software\Microsoft\Shared Tools\Graphics Filters\Import\CDR]
"Enabled"=dword:00000001
I ran that REG file and also added its contents to my Win7RegEdit.reg file, along with a reminder to run that file both early and, again, late in the process of installing Windows 7.  The REG fix appeared to work without a need for a reboot; I was immediately able to open files in Word that were blocked just a few minutes earlier.

Thursday, March 15, 2012

Batch Converting Many Text Files to PDF

I had a bunch of .TXT files that I wanted to convert to PDF.  I had solved this problem previously, but it looked like I hadn't written it out clearly, so that's the purpose of this post.  This explanation includes solutions to several other sub-problems.  All together, the things presented here were useful for solving a variety of problems.

First, I made a list of the files to convert.  My preferred way of doing this was to use DIR.  First, I would open a command window.  My preferred way of doing *that* was to use the "Open command window here" context menu (i.e., right-click in Windows Explorer) option.  An alternative was to use Start > Run > cmd, but then I would have to navigate to the desired folder using commands like CD.

The DIR command I usually used, to make a list of files, was DIR /s /a-d /b > filelist.txt.  (Information on DIR and other DOS-style commands was available in the command window by typing the command followed by /?.  For example, DIR /? told me that that the /s option would tell DIR to search subdirectories.  A variation on the DIR command:  DIR *.txt /s /a-d /b.  The addition of *.txt, in that example, would tell DIR that I wanted a list of only the *.txt files in the folder in question (and its subfolders).  If I wanted to search a whole drive, I'd make it DIR D:\*.txt /s /a-d /b > filelist.txt.  If I wanted to search multiple drives, I'd use >> rather than > in the command for the second drive, so that the results would add to rather than overwrite the filelist.txt created by the preceding command.

Using DIR that way could gather files from all over the drive.  Sometimes it was better to gather the files into one folder first, and then run my DIR command just on that folder.  An easy way of finding certain kinds of files was to use the Everything file finding utility, and then just cut and paste all those files from Everything to the desired folder.  For instance, a search in Everything for this:

"see you tomorrow" *.txt
would find all text files whose names contained that phrase.  Cutting and pasting that specialized list into a separate folder would quickly give me a manageable set of files on which I could focus my DIR command.  (There were other directory listing or printing programs that would also do this work; I just found them more convoluted than the simple DIR command.)

Once I had dirlist.txt, I copied its contents into Excel (or I could have used Excel to open dirlist.txt) and used various formulas to create the commands that would convert my text files into PDF.  The form of the command was like this:
notepad /p textfile.txt
I wasn't sure in the case of Notepad specifically, but I was able to run some programs (e.g., Word) from the command line by just typing one word (instead of e.g., "notepad.exe," or a longer statement of the path to the folder where e.g., winword.exe was located) because I had put the necessary shortcuts in C:\Windows.

Those Notepad commands would send the text files to my default printer.  My default printer was Bullzip.  When I installed it, it gave me a separate shortcut leading to its options.  For this purpose, I set its options so that it did not open the document after creation (General tab), specified an output folder (General tab), and indicated that no dialogs or questions should be asked (Dialogs tab).

I copied the desired commands from Excel to a Notepad text file and saved it with a .bat extension.  The rest of the file name didn't matter, but the .bat extension was important to make it an executable program.  In other words, if I double-clicked on PrintThoseFiles.bat (or if I selected PrintThoseFiles.bat and hit Enter) in Windows Explorer, the batch file would run and those commands would execute.  (I could also run the batch file from the command line, just by typing its name and hitting Enter -- which meant that I could have a batch file running other batch files.)

So that pretty much did it for me.  I ran the batch file, running lots of Notepad commands, and it produced lots of good-looking PDFs.

Please feel free to post questions or comments.

Friday, February 17, 2012

Windows 7: Finding a DIR Alternative

I needed a DIR-type listing that would provide extended information about a file:  its name, date, and size, and also its path (i.e., the folder and subfolder where it was located), all on a single line of output.  DIR didn't seem to be capable of this, and neither did the utilities I found with a search (e.g., Karen's Directory Printer).

Another search raised the possibility that certain Linux utilities brought over to Windows might have this kind of capability.  I didn't want to run a Linux virtual machine on Win7; I just wanted to be able to run Linux commands that might add functionality I wasn't getting in Windows 7.

Linux commands were probably not the only alternative.  For instance, I could have learned how to use Windows PowerShell scripts.  My general impression of the Microsoft approach (as in the contrast between original BASIC and VB) was that, unfortunately, something that could be done with one relatively simple command in another tool would require three or four lines of code, which I would be able to write only after mastering a handful of relatively abstruse programming concepts, in the Microsoft product.  This impression seemed borne out when a search led to an indication that the DIR equivalent in PowerShell would require a multiline FOREACH loop.

Preliminary inquiries gave me the impression that Cygwin sought to provide a subsystem that would emulate a Linux machine within Windows.  There were indications that other projects (e.g., MSYS) sought to provide a somewhat comparable (e.g., 110MB) environment.  These seemed a tad heavy for my purposes; I was looking for something more like GnuWin, which was described as relying "only on libraries provided with any standard 32-bits MS-Windows operating system" and as not needing any Unix emulation.  Ideally, I would have some cool, relatively simple Linux-like commands available at the Windows command prompt.

By this point in my investigation, several people had mentioned CoreUtils.  This turned out to be a package within GnuWinThe CoreUtils homepage described it as "the basic file, shell, and text manipulation utilities of the GNU operating system."  GNU was "a Unix-like operating system," in development since 1983, that apparently provided most of the materials used by Linux (which was, in turn, the source of Debian Linux, from which Ubuntu was built).

To clarify, it appeared that the CoreUtils existed in GNU, and there was an offshoot called CoreUtils for Windows.  Apparently this was what I would be getting through GnuWin.  There were other approaches to this sort of thing (e.g., Gow, UTools, UnxUtils), but my sense at this point was that GnuWin was dominant in this category.

I looked at the list of tools included in CoreUtils (for Windows).  I didn't count them, but I thought I remembered seeing an indication that there were more than 100 of them.  They were grouped into three main categories:  file utilities, text utilities, and shell utilities.  In the file utilities group, the description of the ls command was simply "lists directory contents,"; and vdir would apparently provide a "long directory listing."  These sounded like what I needed.  Examples in the text utilities category included comm ("compares two sorted files line by line") and uniq ("remove duplicate lines from a sorted file").  Examples in the shell utilities category included sleep ("suspends execution for a specified time") and uname ("print system information").

Although I could have just clicked on a download link, I went into the folder for the latest version and saw that it had not been updated since 2005.  This made me wonder whether I should have opted instead for Gow (short for GNU on Windows), which had apparently been updated as recently as November 2011.  I found a spate of (1 2 3 4 5) brief summaries of Gow published about that time.  Their similarities raised the thought that they may have been written from similar press releases.  Not that that would necessarily be bad.  Any product being promoted in 2011 could count as fresh air against a 2005 alternative.  But it was not reassuring that none of these explained clearly whether Gow was genuinely different, or just a borrowing, from the seemingly better-documented and more widely used GnuWin.  I found a page stating that Gow had been developed by a corporation in 2010 and used for some years before being released as open source.  This appeared to be an authoritative page.  It puzzlingly characterized GnuWin as being appropriate "if you want just one or two utilities."  A list of Gow utilities seemed similar, at a glance, to the GnuWin list (above), though I noticed that it did not have vdir.  The seeming mischaracterization of GnuWin, combined with the sense of evasion in the press-release writeups, persuaded me to stick with Plan A.

So now I did download and install the executable (exe; not src.exe) version of CoreUtils (6MB).  But, weird thing, they didn't give me a way to run the program.  My Start Menu had links to several PDFs.  Actually, it was rather messed up: they gave me four shortcuts to a total of two PDFs, and some of those links were buried about five layers deep in superfluous subdirectories. They also gave me two links to CoreUtils Help files that, when I clicked on them, gave me the familiar "Why can't I get Help from this program?" message that Windows 7 kindly provided when I would try to run Help files written for Windows XP.

Obviously, I ignored the manuals' actual contents and went looking for a way to run the program.  Weird thing:  I had all these redundant and dysfunctional help materials, and a link to an Uninstall routine, but no actual "Run CoreUtils" shortcut. I was half-tempted to uninstall them as defective, when it occurred to me that, well, they're supposed to be run from the command line, not the Start Menu.  So, OK, I went to the command line and typed "ls."  Windows said, "'ls' is not recognized as an internal or external command, operable program or batch file."  Hmm.  The manual, then, if I must.  Or manuals, I should say:  a regular-looking manual and also what appeared to be the set of Linux MAN (i.e., manual) pages, both in PDF format.  Neither had installation instructions.  I went to the ls MAN page.  It seemed to say that "ls -a" would be a working command.  Well, not on my machine, it wasn't.

I rooted around and found an article on how to use CoreUtils.  It said that I would have to adjust the PATH environment variable to tell the system where to look for the CoreUtils command instructions.  My way of applying those instructions was as follows:  first, in Windows Explorer, find where the CoreUtils executables (e.g., ls.exe) were installed.  On a 32-bit Windows 7 system, the location would probably be C:\Program Files\GnuWin32\bin; on a 64-bit system, C:\Program Files (x86)\GnuWin32\bin.  With that folder selected, click on the address bar at the top of Windows Explorer, make sure the whole address was highlighted (Ctrl-A if necessary), and copy the address (Ctrl-C).  Now I went to Start > Run > SystemPropertiesAdvanced.exe (could have used sysdm.cpl and then the Advanced tab) > Environmental Variables > System Variables > highlight Path > Edit > hit the End key.  There, I typed a semicolon (";") and then pasted in what I had copied from the Windows Explorer address bar.  (Could have typed it manually, using the 32-bit or 64-bit address just shown, but this was more accurate and it also forced me to verify the actual location.)  I OKed out of there and tried ls -a again on the command line.  Did I have to reboot to make the Path take hold?  Yes.  That was it.  I had ls, and it listed files.

So now, how about getting all that information mentioned at the outset -- path, date, etc., all on one line?  First question:  how could I get command-line command help?  In Windows, it was DIR /?.  But the /? option gave me an error with ls.  "man ls" didn't work either.  Page 9 of the manual PDF said the MAN pages were no longer being maintained.  I wasn't sure if that applied to what looked like the MAN pages included with GnuWin.  There wasn't a MAN MAN page in that PDF.  Page 10 said --help might work.  I tried "ls --help" and experienced satisfaction.  What I was seeing there looked like what appeared on pages 50-52 of the man PDF, pages 60-70 of 176 (text pages 52-62) in the more explanatory help PDF.  I wasn't inclined to read 11 pages to figure out how to get my directory listing.  Skimming down through the ls --help output, I tried "ls -l -N -R."  Good, but no cigar:  the path wasn't on the same line as the filename; no improvement over DIR.

The user's guide PDF didn't seem to think that there actually was a way to print the file's path on the same line as its date, filename, etc.  And so there I was.  I had come all this way with faith in my heart for the infinite possibilities of Linux.  I fervently believed that, with GNU, anything was possible.  But now, with my limited knowledge of Linux and such, cruel reality was saying Bismillah, no! we will not let you have all that stuff on one line of output.  There actually probably was a way to do it with some other tool, like the awe-inspiring grep, available in a different GnuWin package.  But I wasn't quite ready to go there.  In this project, grep looked, for me, like a bridge too far.

I thought about posting a question in the GnuWin Help forum.  But there had only been a handful of posts there in the last couple of months.  I also thought about going down the list of other utilities contained in CoreUtilities, so as to demonstrate to myself that this hadn't been a wild goose chase.  I thought about trying Gow after all, just in case its version of ls had different capabilities.  I thought about working up a kludge in which I would do a listing of all directories first (with e.g., "dir /ad /s /b") and then try to invent a way to append the pathname to each file line.

But before pursuing those rather lame possibilities, I noticed TCC/LE, advertised as a complete, powerful replacement for Windows CMD.  (TCC was short for "Take Command Console.")  It got 3.5 stars from 43 voters at Softpedia, only a solitary vote (five stars) at CNET -- but it had apparently been updated there just a few days earlier.  At MajorGeeks, it averaged 4.07 from 38 voters.  The description said it had enhanced commands (specifically including DIR) with new options.  A search didn't encourage the sense that there was a regular category of this sort of thing, with lots of competitors.  I downloaded and installed it.  The installation process seemed pretty slick, ending with a direct ride to their forums.  The installation left me with an open CMD window with a funky prompt, though apparently it was actually their own version of a command window.  (I did have another Win7 command window open throughout the installation.  It remained functional; I was able to close and open a new one after installation.)  I typed Help at their command prompt and went straight into their GUI help dialog, which actually made me say "Wow."  It wasn't spectacular; it was just good, and helpful, which I guess counts as spectacular after a long slog.  I replaced their ugly prompt with the ordinary Windows one by typing "prompt $P$g" at the prompt, though not without first amusing myself with variants (e.g., "Now what?").

Eventually I discovered that their help dialog was more or less the same as their online help page.  The manual had a large number of further instructions on how to tinker with the prompt and, it seemed, everything else.  Typing "option" at the prompt brought up settings, but not an obvious way to preserve prompt settings between sessions; it appeared the answer to that might lie somewhere within their SET command.  Anyway, I found information on their DIR command almost instantly, and also got a cursory version of it by typing dir /? at their prompt.  It led me to PDIR, and there I found the answer I was looking for.  What I had to type in a TCC/LE command window was this:

pdir D:\ /s /(dy-m-d zc fpn) > dirlist.txt
That gave me all of the information I was looking for, on a line-by-line basis, for every file on drive D, output into dirlist.txt.  Specifically, with the options in that sequence, I got the date (y-m-d), size (with commas), and the file path and name.

I took a quick look at their list of Commands by Category.  I also saw that they had a number of video and textual tutorials.  An impressive program.  But in any case, this investigation was done.

Tuesday, February 7, 2012

Windows 7: Verify or Validate MP3s

I had a folder full of MP3s. I wondered whether any of them were bad.  In a brief previous search, I found MP3 Diags as a possible MP3 tester.  This post describes what I found when I tried it out.

Trying Out MP3 Diags

I was using MP3 Diags 1.0.07.  It had a Mac-type interface that I disliked.  That is, instead of a menu with words that had meaning to me, it showed big, gaudy icons that mostly meant nothing until I hovered my mouse over them to get the tooltip.  (The icons shrank to a more pleasant size when I shrank the program to fill only part of the screen.)  The tooltips were visible only when MP3 Diags was in focus onscreen; I couldn't see them while I was typing these words.

With those initial reactions, I placed myself squarely among those who plunge into a program without first reading its documentation.  I was admittedly more inclined toward the WFM philosophy than toward the RTFM philosophy.  Windows programs did generally seem capable of using menus, including cautionary pop-ups, putting risky functionality into Advanced tabs, and otherwise steering stressed users into safe or pre-warned channels of behavior.  I appreciated that the MP3 Diags programmer had provided many pages of documentation, among which he warned that the program was not really designed for those who were looking for a pushbutton solution.  But it had to be clear that many users would never see that warning, or would perhaps mistakenly think they understood it when they did not, and that was the basis on which I approached the program.

After hovering over all of the icons, I chose two that seemed relevant for starting purposes.  First, when I hovered my mouse over the gearlike icon at its top left corner, the tooltip said, "Scan folders for MP3 files [Ctrl+S]."  From the Windows world, a gear icon would normally mean Tools; it seemed to me that a different icon would have been better, with a tooltip that said, "Select folders to scan."  That option let me use checkboxes to designate a particular folder.  The MP3s in that folder were not large; they averaged about 400K each.  When I checked the boxes, the program was ready to begin scanning.  I canceled out of the gear icon and went to the wrench-and-screwdriver icon, at the top right corner of the program, that seemed more fitting to its purpose:  adjust settings.

Then I had to go back into the gear icon to run the program.  On a fairly up-to-date computer, it seemed to diagnose about 12 MP3s per second.  When it was done, it put up this notice:

Your files are not fully supported by the current version of MP3 Diags. The main reason for this is that the developer is aware of some MP3 features but doesn't have actual MP3 files to implement support for those features and test the code.

You can help improve MP3 Diags by making files with unsupported notes available to the developer. The preferred way to do this is to report an issue on the project's Issue Tracker at http://sourceforge.net/apps/mantisbt/mp3diags/, after checking if others made similar files available. To actually send the files, you can mail them to ciobi@inbox.com or put them on a file sharing site. It would be a good idea to make sure that you have the latest version of MP3 Diags.

You can identify unsupported notes by the blue color that is used for their labels.
At that point, the MP3 Diags screen consisted of three panes.  In the top third of the screen, MP3 Diags listed the MP3s that it had tested, with columns indicating which Notes applied to them.  In the middle third of the screen, MP3 Diags provided explanations of those notes.  In the bottom third, MP3 Diags seemed to be showing me details about the single MP3 that was currently highlighted at the top.  For instance, the bottom pane said this about one file:
1:16, MPEG-2 Layer III, Single channel, 22050Hz, 40000bps CBR, CRC=yes, frame count=2923; last frame located at 0x5d2d1
That last pane didn't seem too important for present purposes, so I focused on the middle pane.  It looked like MP3 Diags had found four kinds of things worth commenting on, in the files that I had submitted for diagnosis.  None of these were error messages; they were all just informational.  They read as follows:
fa  -  No ID3V2.3.0 tag found, although this is the most popular tag for storing song information.

ob  -  No supported tag found that is capable of storing song information.

ab  -  Low quality MPEG audio stream.  (What is considered "low quality" can be changed in the configuration dialog, under "Quality thresholds".)

an  -  No normalization undo information found.  The song is probably not normalized by MP3Gain or a similar program.  As a result, it may sound too loud or too quiet when compared to songs from other albums.
(I was not able to copy and paste these messages from MP3 Diags, and therefore had to retype them here.)  I didn't know where the letters (e.g., "fa") came from; I would have found it more helpful to see error codes grouped into areas of concern (e.g., quality, tags, playability).  The point seemed to be that there were no problems with the MP3s per se, as distinct from their tags and their quality and their normalization:  they would play without errors.  That was my concern.  These weren't songs, as MP3 Diags seemed to assume; they were just old recordings of speech, and as such did not need to be recorded at a high bitrate.  I just wanted to know whether anything had gotten corrupted.

I had been reading that middle pane with its "File Info" button clicked.  I clicked on its "All Notes" button instead.  It showed me errors like these:
aa  -  Two MPEG audio streams found, but a file should have exactly one.

ac  -  No MPEG audio stream found.
These were obviously much more worrisome, and I didn't seem to have them, so that was good.  Ah, but now that I focused on the top pane, I suspected that the four notes shown above (fa, ob, ab, and an) might apply only to the one MP3 that was highlighted in the top pane.  Was I supposed to page down through all of the MP3s to do a manual check of which ones might have which errors?  I tool-tipped the icons at the top of the screen again, looking for some kind of reporting function.  I tried the "Filter by Notes" option.  Unfortunately, since the error notes weren't grouped or hierarchically arranged under main topics (e.g., playability), it appeared that I would have to read all 23 notes and choose the ones that were worrisome -- as distinct from, say, just clicking on a main category and optionally selecting or deselecting subcategories.  Some of the categories were presented in colored print, for unknown reasons.  For instance, "aa" was brown, "an" was black, and "dj" was blue.  The brown wasn't much different from the black; I had to look close to make sure.  I wasn't entirely sure which categories to worry about:  was "dj" ("Unsupported ID3V2 version") ominous, or did it just mean something to do with tags, which were irrelevant for purposes of these MP3s?  I had to look at Wikipedia to see that ID3V2 related to information about the file, and apparently not its actual audio contents.

The error codes I chose to filter by were aa, ac, ad, ae, ak, bg, cb, ia, ib, ja, kb, kc, kd, and of.  As that last example illustrates, it was awkward to write about some of these codes (e.g., an, of) without using quotation marks, since their letters formed English-language words.  I wasn't sure, but a quick search suggested that these codes might be peculiar to MP3 Diags, and therefore easily changed, rather than being promulgated by some official MP3 authority.

Once I had selected those codes to filter by, the top pane in MP3 Diags changed.  Now I could see that various files had various issues.  It was also more obvious, now, that both the middle and bottom panes were providing information specifically about the file highlighted in the top pane.  I would have liked to see a count of how many files had error "aa," for instance.  Another useful feature would have been an option to display files in order of the nmber and/or seriousness of their errors, so that those with six or eight errors would come before those with just one.  It would also have been interesting to see whether all of the files having a certain error were clustered in the same folder, in which case I would think maybe I should just replace that whole folder with good copies from a backup.  I could refilter by folders, but apparently I couldn't filter, sort, or output a report by both folders and error codes.  In fact, it seemed I couldn't output a report at all, which meant that I wouldn't be able to write a batch file to mass-delete or mass-zip any files that might be bad.  If I selected a group of files in the top pane, it seemed that the only error codes I would see in the middle pane would be those pertaining to the first file in the group.  There were no right-click options for selected files.  And if I clicked on an error code column (e.g., ab), the program would not re-sort the files according to their values in that column.

Later, I would look in more detail at some of the error results (below).  But at this point, MP3 Diags had helped me to clarify my thinking about Mac-like software.  A while back, someone had asked me why I didn't just use Apple hardware and avoid the hassles I was experiencing with PC stuff.  I had my reasons, but it was still a question worth keeping in mind.  I knew I didn't really care for Macs, but hadn't thought much about why not, exactly.  If MP3 Diags was any indication, it now seemed to me that one answer to that question would be that much Mac stuff is lacking in what would be considered basic functionality, among PC programs (although, as this post demonstrates, that's a problem in much PC software too).  At its worst, Mac software seems to promote the idea that simplicity is superior.  And it is, if you don't need to do anything complex.  But again, that was just a passing reaction, based (at the moment) on one program that was surely not a fair representation of Mac software at its best.

It did seem quite possible that the programmer of MP3 Diags would find such remarks puzzling if not bizarre.  I realized that my reactions could well be very far from his intentions.  Again, my purpose was to convey a sense of how the program felt in use, a sort of walk-through from a new user's perspective.  Another way to phrase the message was that idiosyncratic design may be best saved for those situations where it is really necessary.

Alternatives to MP3 Diags

While I had my objections to MP3 Diags, I was not seeing any immediately obvious alternatives.  Not to say there were no contenders.  My previous search had also led to MP3valCheckmate MP3 Checker, and MP3Utility.  I did some random flailing around, looking at these and following leads to others.  An eHow webpage suggested that Dr. Tag's MP3 Repair Tool.  In a search for "MP3 validation," The SnapFiles list of "Misc. MP3 Software" included MP3 Diags, MP3val, and also MP3Test ($17 trial).  Another search led to an old thread that mentioned Foobar 2000, a highly recommended MP3 player that apparently had some kind of MP3 checking capability (or maybe it just wouldn't play bad MP3s), as well as dBpowerAmp ($14), Audiotester, Mr. Question Man (also sometimes called Burrrn), and EncSpot.

Among all those programs, there didn't seem to have been much testing.  Some were old and had not been updated.  In most cases, the ratings at cites like CNET and Softpedia were based on just a few votes (sometimes just one).  There were miscellaneous accounts -- for instance, MrSinatra reported questionable results from Audiotester.  But I felt I was swimming in very murky waters.  It seemed my best strategy might be to look at the ratings in Softpedia and CNET, supplemented by a search for reviews and a general sense of which of these tools had been most extensively used and recommended.  Based on various comments I had seen, I decided to focus on MP3val, Checkmate, MP3Utility, MP3 Repair Tool, Foobar 2000, and Mr. Question Man.

Of those six programs, only Foobar2000 was listed on CNET (rated 4 out of 5 stars by 129 voters, excellent by editors).  Softpedia listed MP3val (3.5 stars by 26 voters), Checkmate (3.0 stars by 24 users), MP3Utility (3.6 stars by 21 voters), Portable MP3 Repair Tool (2.7 stars by 30 users),  Mr. Question Man (3.5 stars by 24 users) -- and Foobar2000 (4.6 stars by 1646 voters).  I decided to start with Foobar2000.  If that didn't lead where I wanted, I would try looking at MP3val, MP3Utility, and/or Mr. Question Man.

But then I ran another search.  I was curious as to whether Winamp would provide at least the same functionality as Foobar2000.  The search inadvertently pointed me toward two other MP3 checking programs, both on Softpedia, that looked like they might actually be more widely used than some listed in the previous paragraph.  Those two programs were MP3-Check (3.8 stars by 63 voters) and MP3 Checker (3.9 stars, 25 users).  As for Winamp itself, a couple of searches seemed to indicate that it did not share Foobar2000's ability to test MP3s.  These results suggested that, if Foobar2000 didn't do the job, I should look next at MP3 Checker and MP3-Check, before turning to the others listed above.

Verifying MP3s in Foobar2000

I had expected Foobar2000 to give me a funky, multicolored player interface like Winamp.  Instead, I got a straightforward space where files would be listed.  I didn't see where it would have the ability to verify MP3s, so I went back to that old thread and saw that I would first have to load some files and then right-click on them and select the appropriate option.  I did that.  Loading files took a while.  Foobar2000 indicated that it was "processing" them.  It seemed to be handling only a few per second.  When it was done, unfortunately, a right click revealed no testing options.

On the other hand, a right-click on an MP3 in Foobar2000 did present the possibility of doing a mass conversion.  Foobar2000 offered ten output conversion formats, including WAV.  (It also offered ways of modifying the output, e.g., crossfade, skip silence).  This raised the possibility that I could do a mass conversion.  Presumably a bad file would not be capable of being converted to another format.

Or at least a bad file would not produce good sound upon conversion.  I visualized an MP3 testing program of the future.  It would extract sound samples from several different points in the MP3, including beginning, middle, and end, and would compare them against designated reference files containing samples of the kinds of noise that the tested file should contain.  Perhaps this imaginary program would concatenate copies of MP3s not fitting the profile, display the waveform, and underneath it show the name of the file, and the location within the file, from which the presently viewed sound sample was taken.  That way, users could eyeball the program's judgments as to which MP3s were conforming or nonconforming, and could have some hands-on assurance that the program was accurately detecting acceptable vs. screwed-up MP3s.

I didn't have a program like that.  But, as I say, I did have the option of doing a bulk conversion.  Apparently Foobar2000 and, as it turned out, Winamp would do it.  Cool Edit 2000 (no longer available) would do it, and so, probably, would some other audio editors.  I was afraid that conversion could take a long time, but Foobar2000 converted a test group of 20 of these small MP3s within just a few seconds, and the files played successfully.  So this was one possible route.  I guessed that Foobar2000 would convert a blank file without objecting, though, and would otherwise fail to provide some of the warnings that I had seen in MP3 Diags (above).  I decided to look at other possibilities.

MP3 Checker and MP3-Check

I looked at the Softpedia and CNET pages for Convivea's MP3 Checker.  They essentially repeated what I saw on the MP3 Checker homepage.  It sounded like a simple and capable program.  The latest version was 1.08, released on July 22, 2006.  I also looked at the homepage and the Softpedia and CNET pages for MP3-Check by AudioMoves.  On this very preliminary basis, I was leaning toward MP3-Check over MP3 Checker because the MP3-Check homepage provided a more detailed description of what it did, and because its webpage and the product both seemed to have been updated within the past year or less.

So I downloaded MP3-Check 1.40 from Softpedia and installed it.  It appeared to be designed primarily to check for MP3s that might have problems with quality or with their tags.  These were not my concerns at present; I knew that some of the MP3s I would be testing might have bad or nonexistent tags, low bitrates, low sample rates, low volume, or might use joint stereo -- to cite the five criteria that MP3-Check allowed me to select and, to varying degrees, to adjust.  I turned off all of those options except the tag check, which did not clearly appear capable of being turned off.  I checked the option to Create Status Logfile.  Then I ran the program on the same folder that I had tested with MP3 Diags (above).  I could see indications in the program's status bar that it was checking files very quickly.

When MP3-Check was done, it put up a little notice indicating how many files it had checked.  It calculated an average time of about 55ms per MP3, for these little MP3s that I had it check.  When I clicked OK on that notice, it opened its logfile.  The log showed me the names of the files it had checked, and their sample and bit rates.  The log was tab delimited, so I could copy and paste directly into Excel and sort by the various columns (e.g., bitrate), to see if anything looked odd.  There were several files with very low bitrates.  I dug those out and listened to them.  Two of them were corrupted, and I was able to replace them with backups.  But those were the only bad files I was able to find this way.  MP3-Check did indicate that a large number of my files had tag problems, as I expected.  It also indicated that there was one "Unordinary MP3" in my list.  I wasn't sure what was the matter with that file.  It seemed to play OK.  Otherwise, I was done with MP3-Check.

MP3-Check vs. MP3 Diags vs. Mr. Question Man

I wondered if MP3 Diags had detected the two or three bad files that I had just identified using MP3-Check.  I hadn't closed MP3 Diags yet, so now I went back to look at the results it had shown me.  Looking through those results was not easy.  There was no Ctrl-F option to find a specific file by name.  There was also no Ctrl-A option to select all files in the list.  PgUp worked, but Home didn't, so I would have to page or scroll to get to the top of the list.  Once I was there, there didn't seem to be any Shift-End or other key combination that would let me select the whole list that way either.  I could select the whole list by starting at the top and doing a Shift-PgDn until I got to the bottom, or by scrolling all the way down, but of course this would take a long time with a large list.  And what was the point?  Once I had the whole list selected, Ctrl-C worked for only one item at a time, and there was also no right-click option, by which I could copy the list and paste it into a spreadsheet or Notepad for further searching and comparing.  Worse, my attempts to highlight the whole list with Shift-PgDn caused MP3 Diags to freeze up.  It came back to life after five or ten minutes, but this was not encouraging.

I thought maybe I could find those few bad files, if MP3 Diags had detected them, by narrowing my filter to just those categories that might have caught a corrupted file.  But I thought I had already done that, when I decided to disregard tags and such (above) and focus instead on audio stream issues. 

It seemed that MP3 Diags was a good start on a potentially great program.  But I was uncomfortable with what seemed to be its core idea:  never mind about the details; just click on the proper selection and we will take care of fixing your MP3s.  It was a nice dream, and their colorful GUI seemed to support it, but so far the program had not won my confidence.  I knew it was quite possible for a magical, black-box program to make things worse.  I appreciated that these were all freeware programs, and that their creators had kindly made them available to the rest of us.  It was just that I was looking for one really good program to do this job right and not make more headaches for me.

At some point in the process, I took a look at Mr. Question Man.  Its webpage had not been active since 2006, and its description on Softpedia did not make clear whether it would repair MP3s, as distinct from merely providing information about them. But I went ahead with it anyway, on the strength of its positive user votes, few but mighty.  It did turn out to be informational only.  I appreciated its option for configuring the Isolinear Optical Chips Latency, on the Settings tab devoted to PapalaPapIHaveToCustomizeEverything.  Funny program.  I was thinking the writer should try political journalism.  S/he might have a positive impact there.

MP3 Checker

I downloaded and ran MP3 Checker.  It made a good initial impression, with practical options like "Do not report MP3s with minor glitches as BAD" and "Move MP3s with errors to the quarantine directory" (typos corrected here).  I didn't like that I couldn't resize its window.  And in the time it took me to write that last sentence, it had finished its scan of a rather substantial number of MP3s; it took the focus onscreen; and since I was in the middle of typing, my keystrokes seemed to be just what it needed to shut itself down.  It was like a miniature tornado had ripped across my screen, kicking up a little dust but apparently not doing anything significant.

I started it up and ran it again.  It had remembered all of my previous settings except the folder where I wanted it to look for MP3s.  This time I made a point of finishing my typing before I started it, so I wouldn't inadvertently shut down its closing announcement again.  What I saw this time -- again, after running for less than a minute -- was not good.  It said that it had completed the scan and had scanned 0 files, processing 0 total bytes of data, verified 0 good MP3s, and detected 0 bad MP3s.

There had to be something wrong.  A couple dozen users had given this thing fairly good marks.  It had seen my MP3s -- I could see it listing their names in its status bar.  It believed it was doing something with them.  I was missing something somewhere.  Maybe its users had been running Windows XP -- maybe somehow that made a difference?  I had no idea.  A discussion thread conveyed the impression that MP3 Checker was a casual project and seemed to produce erroneous results.  With all due respect to other reviewers who liked it, I concluded that this was not the program for me.

MP3val

As noted above, several other MP3 validation tools had received ratings in the vicinity of 3.5 stars from a couple dozen users each.  MP3val was one of those.  It came with the executable file mp3val.exe, which would run from the command line with several options including -f (try to fix errors) and -si (suppress info messages).  I chose to run it as a portable GUI program.  There were virtually no settings options.  Basically, I just pointed it to the folder containing the MP3s and told it to scan them.  It loaded the list of files in that folder, and then gave me options of scanning or repairing all or selected files within that folder.  I told it to scan them.  It ran down the list in a spreadsheet-like layout with two columns:  file name and state (i.e., condition).  In a small pane at the bottom, it displayed what I guessed I would have seen if I had run the command-line version:  the name of the file it was analyzing, a warning that the file contained no supported tags, and a statement of the file's properties (e.g., its number of frames, type of MPEG, number of tags, CBR).  It was proceeding quickly but not instantly.  It did appear to be doing a genuine scan.

When it was done, I tried to sort the list of files by clicking on the header of the State column.  That didn't work.  The procedure was, instead, to go into View > Scanned Files with Problems.  It showed me a list of troubled MP3s.  When I selected one, the pane at the bottom told me what the problem was.  Here were some of the warnings displayed there:
MPEG stream error, resynchronized successfully.

No supported tags in the file.

VBR detected, but not VBR header is present.  Seeking may not work properly.

It seems that file is truncated or there is garbage at the end of the file
The manual (a simple HTML file included in the portable folder) listed a number of other possible error messages.  I would soon be comparing these error messages against those generated by MP3Utility (below).

The manual did not provide an indication of how I might save the scan results into a file.  Ctrl-A didn't work, but I was able to select all of the files in the Problem list by using Ctrl-Shift-End from the top.  But once they were selected, Ctrl-C didn't work to copy them, and a right-click just gave me options to delete, scan, or repair the selected files.  I looked in vain for a log file that it might have created in its program folder; apparently its list of problem files was saved either in RAM or in a temporary directory somewhere.  (It belatedly occurred to me that I could perhaps access a log to get the information I had been unable to extract from MP3 Diags, above, but it was not clear to me what its MP3Diags.dat files was trying to say, so for practical purposes that workaround didn't work.)  It probably would have been possible to get the list of problem files from the screen, using a capture-and-OCR program like Aqua Deskperience (which I had bought) or JOCR or SysExporter, though that route would be painful with a long list of files.  It would probably be possible, and surely easier, to use the command-line version of the program to get a list of files.

MP3Utility

As noted above, I had found two other MP3 validators that had averaged 3.5 stars or better from at least 20 users.  These were MP3Utility and Mr. Question ManThe Softpedia page for MP3Utility, a portable, seemed to indicate that the program's last revision was in 2009.  The webpage and, even more, its Readme.txt also provided an encouraging amount of detail.  I got the impression of a careful, thoughtful effort to identify and handle flawed MP3s.

MP3Utility offered few but potentially useful options, such as the possibility of adding it to the right-click context menu.  When I ran it on my test folder, it identified 16 bad MP3s, moved them to a designated folder, and allowed me to save its log file for later reference.

I took a look at those results.  For 15 of those 16 bad MP3s, the logged error was of this type:  "First sync error at approx.1:31 (80% through audio)."  For the other one, the error was, "Can't locate first valid frame header within 5,000 bytes of beginning of file."  The MP3Utility Readme said that the program would identify several kinds of errors, which I summarize as follows:
Unable to open file (file is protected by another application or was moved after being initially loaded into MP3Utility)

File too short, or End of file encountered in first audio frame

Can't locate first frame header

Last audio frame truncated (can be ignored in almost all cases)

Last audio frame too long (can probably be safely ignored)

Error reading frame header xxx (i.e., sync error; serious error except possibly when it occurs at end of file)
So the logged results of the search of my MP3s did not match up exactly with this list, but apparently the only errors in my MP3s were in the last category:  I had 16 files with sync/header errors.  The log stated that it had "found errors/warnings in 16 files," so evidently MP3Utility didn't think any of my files had any of the other errors listed here.

Comparison of Errors Found

How did the results from MP3Utility compare against the results of other programs?  I had made a slight mistake, for comparison purposes:  I had gone ahead and replaced two bad files after running MP3Check, as noted above.  But otherwise the errors identified by MP3Utility should have matched up exactly with the errors identified by the other programs (above) that did produce a list of bad files.  The programs of particular interest, at this point, were MP3 Diags, MP3-Check, and MP3val.

I looked at MP3 Diags first.  I wondered how its relatively extensive list of "notes" would match up with the list of errors just given.  It seemed that MP3Utility was superior on the first one, "Unable to open file," insofar as there was no acknowledgement of any such possible error in MP3 Diags.  In other words, MP3 Diags would apparently give the user the impression that all files had been checked, even if some of them were locked or not found.  This incorrect information may have seemed of no concern from the MP3 Diags perspective, since the program appeared to be oriented toward giving the user a complete (long) list of errors and then, after running a fix, presumably declaring most if not all of them to be repaired.

The second MP3Utility error, "file too short," appeared at first to be divided into at least three categories in the MP3 Diags errors:  "no MPEG audio stream found" (which MP3 Diags labeled an "ac" type error), "invalid MPEG stream - fewer than 10 frames" (type "ak"), and "File contains null streams" (type "kd").  But possibly I had misunderstood what a "stream" was.  I thought a stream was the audio data, as distinct from some sort of header and/or tailer that would contain non-audio data (e.g., tags).  But it turned out that, at least in mp3HD format, you could have an MP3 that would have two data streams.  Apparently that was not possible when MP3 Diags was created, else its programmer would not have included the "aa" error message, which stated that a file should have exactly one audio stream.  Then again, some MP3 Diags error descriptions (e.g., "kc") did seem aware of this.  Anyway, MP3 Diags identified a number of MP3s with these problems:  many with an "ac" error, many with an "ak" error, and six with a "kd" error.

MP3 Diags did not provide a way to right-click or double-click on a file listed in its onscreen error report, so I searched manually to check some of the files listed under those three error categories.  First, I noticed that three of the "kd" files were reported as having "ak" errors as well.  These seemed to be exceptionally troubled MP3s.  All three of these were in the folder containing Bad MP3s that MP3Utility had segregated.  So, good, the programs seemed to agree about those.  How about the "kd" files not containing "ak" errors?  Those were in the Bad MP3s folder too.  So MP3 Diags and MP3Utility seemed to agree that "kd" errors were bad (though possibly MP3Utility had moved one or more of those files to the Bad MP3s folder for some other reason).  But obviously MP3Utility did not share the MP3 Diags concern with many other files, else the Bad MP3s folder would have contained far more than just 16 files.  It did not appear that MP3 Diags was trying to produce a careful technical analysis that researchers and others could use for multiple purposes.  If that had been the case, it would presumably have been possible to export the MP3 Diags error results to a log file.  Fairly or not, I was reminded of those anti-malware programs that seemed to exaggerate the number and significance of threats to one's computer security.  Spot checks of several other files containing both "ac" and "ak" (but not "kd") errors did not lead to any obvious problems:  the files seemed to play OK.

As noted above, the next MP3Utility error, "Can't locate first frame header," was apparently serious enough to qualify a file as a Bad MP3.  I couldn't tell which MP3 Diags error would be similar to this one.  The only MP3 Diags errors that referred explicitly to headers were "bg" and "cb," but both of those were described as being issues that would matter only to "some players."  The one Bad MP3 in which MP3Utility found this error was not included among those listed by MP3 Diags as having either "bg" or "cb" errors.

I decided not to examine the next two errors in the foregoing list of errors that MP3Utility would identify, since the Readme said those could probably be safely ignored.  This took me to the last item in the list.  The idea there seemed to be that MP3Utility had found a frame header, but it couldn't be read, and this caused or was related to a sync error, and that was serious.  MP3 Diags did not have any error messages referring to "sync errors" per se.

Since I was not doing too well in an attempt to compare apples to apples, between MP3 Diags and MP3Utility, I decided to try another strategy.  My idea was to filter the MP3 Diags output for those errors that sounded most serious, according to the MP3 Diags error descriptions, and then compare the results against the set of files that MP3Utility had moved to the Bad MP3s folder.  But as I went down the MP3 Diags list, I couldn't really tell if any were serious.  The one exception was "ac" ("No MPEG audio stream found") -- but as noted above, I had sampled some "ac" files and they played, so I didn't understand how MP3 Diags could say that it found no MPEG audio streams in them.  I mean, they were MP3 files, and MP3 is a kind of MPEG.  I tried renaming one of them as a WAV and playing that in IrfanView, and that produced an error ("Can't read file header"), and the error went away when I changed it back to an MP3 extension.  It did seem to have an MPEG stream.

In a modification of that alternate strategy, I filtered the MP3 Diags list for all error messages that sounded like they could involve significant problems in playback (except for "kd," which I had already examined separately, above).  The ones I selected were "aa," "ac," "ad," "ak," "bg," "cb," "ib," and "kc."  MP3 Diags didn't give me a union/intersection choice -- that is, I couldn't indicate whether I wanted the program to display only those files that had *all* of these problems -- so instead it showed me all of the files that had *any* of these problems.  And as noted earlier, MP3 Diags also didn't give me a way to sort this list according to the numbers or types of errors.  I paged down through the list and manually selected a half-dozen files that had at least seven of these problems.  They all played OK -- including a couple that MP3Utility had placed into the Bad MPGs folder.

My conclusion about MP3 Diags, at this point, was that -- for whatever reason -- it was displaying large numbers of error messages that didn't seem to have significance for purposes of playback (as distinct from, say, tag editing).  In so doing, its very limited options meant that users would be at considerable risk of missing the potentially small number of files with real problems.  As noted above, this could make sense for the user whose available time was commensurate with the number and length of files being checked, but it could be overwhelming for others.

Possibly users would eventually see the files with major problems, if they proceeded to let MP3 Diags repair their files.  That is, maybe there would be only a few seriously troubled files left in the list, after the repair process ran.  But I didn't want to let MP3 Diags have its way with my files if I didn't actually need to do that.  After all, the problematic files seemed to be working OK, and these other programs weren't telling me that I needed repairs on large numbers of MP3s.  So I didn't get to that stage of seeing what files would remain on the MP3 Diags list after it ran a repair.

Moving along, then, how about a comparison among MP3Utility, MP3-Check, and MP3val?  As noted above, MP3-Check had identified only a couple of troubled files, and I had replaced them.  I hadn't kept their names, but I had a backup of the tested folder, so I ran MP3-Check there.  Ah, yes, now I remembered which files they were -- now that I was looking at them again.  There were four of them.  The MP3-Check log showed them as being recorded at the very low bitrate of 8kbps; and when I compared them against copies from an old backup, I could hear that there was something very wrong with them.  And in part, maybe that was the point of programs like MP3 Diags -- maybe there were lots of little problems that you wouldn't notice.  Maybe the file would sound like it was fine, until you compared it against another version, or played it with a different player.  (The minimum bitrate threshold in MP3 Diags could not be set low enough to distinguish those 8kbps files from others recorded at 16kbps, which was a setting that some old or otherwise limited audio devices would use.  That is, it was unlikely that those four files should have been stored at 8kbps, and my listening test had revealed that there was something wrong with them; but it was quite likely that I would have some files recorded at 16kbps.)  I was listening to these files in IrfanView, which was able to play almost anything; maybe I would have been having a very different reaction if I'd had to use other software.

Anyway, I had to run MP3Utility and MP3val against that backup folder too, to get a good comparison among the three programs.  The MP3Utility log showed me the same list of 16 bad files as before.  It contained none of the four files that MP3-Check said were recorded at 8kbps -- files that I had manually confirmed were corrupted.  MP3val gave me a far larger list of problem files, but it identified only one of those four.  It seemed, then, that if I used either MP3Utility or MP3val, I might want to supplement it with a program, like MP3-Check, that would determine each file's bitrate.

Next, I compared the list of 16 problem files identified by MP3Utility against the larger list generated by MP3val.  All were included in that larger list.  I suspected that the remarks about the philosophy of MP3 errors, as laid out in the Readme for MP3Utility, might give me some guidance in adjusting the MP3Utility options, such that it would detect more errors -- possibly the very same ones as MP3val had identified.  I was not feeling any particular need to look into this at present.  It appeared that MP3val would do everything that MP3Utility would do, so I tentatively decided to go with MP3val.

The Conversion Alternative

As just noted, MP3val identified only one of the four low-bitrate files listed by MP3-Check.  MP3val said that the problem with that file was, "This is a RIFF file, not MPEG stream."  This gave me an idea.  I checked a few of the other files in which MP3val had identified errors.  The "no VBR header" problem seemed very common.  It seemed that there might be advantages, for some purposes, in developing a concept of the ideal MP3 file -- what it would need to have in terms of tags and so forth -- and then building that into a bulk conversion process.  Then all of the files would pass almost any test, assuming I disregarded inappropriate tests like the bitrate threshold, suitable for music but not text, found in MP3 Diags.  In other words, I would run the conversion, and thereafter I would not have this motley collection of all sorts of errors, produced by various pieces of hardware.  Maybe truly flawed files would stand out more obviously in that sort of arrangement.

I decided that the conversion approach was interesting but unnecessary at this point.  It would perhaps be more appealing if, someday, I investigated the question of what standards were best for archival purposes.  That is, there were probably people out there, somewhere, who had decided that 64kbps WMA was the most stable, durable, reliable format for long-term voice data archives.  Or something like that.  Standards and formats would change and become obsolete from time to time.  Some such conversion might make sense, once I knew what I was doing.  But for right now, there was no point doing a mass conversion to 56kbps, or some other number drawn from a hat.

Fixing the Problems

The MP3 Diags manual contained some advice:  "If you like your files and they don't bother you, then you probably shouldn't change them."  Such advice, seemingly reasonable, rested on the assumption, consistent with some aspects of that program's design and help system, that people would be using MP3 Diags to do detailed exploration of a limited number of individual music files that they would be listening to in their full length.  Like most advice, unfortunately, there could be many situations to which it would not apply.  One need only visualize a paralegal who was under time pressure to verify and sort dozens of multihour deposition recordings, or a corporate peon who was expected to clean up thousands of tech support call recordings, not to mention myriad casual users who had the belief, perhaps mistaken, that it was possible to have a trustworthy program that could identify and fix major errors in their song files.  It seemed that users would be better protected by programs that used a standard design, so as to help them recognize when they were getting in too deep.  As this post demonstrates, most users would probably find it prohibitively time-consuming to try to read the help files and master the eccentricities of various unfamiliar MP3 validation programs, in a search for one that did what they wanted.  In other words, it could seem rather misanthropic to provide a big red "Fix" button along with a buried warning, "Never use the Fix button."

It presently appeared that MP3val had done a good if imperfect job of identifying problem files.  Given its decent reputation (and the fact that I had a backup), I put the BAD MP3s (removed by MP3Utility) back into the main folder, along with the other MP3s, and told MP3val to scan that folder.  When it was done, I told MP3val to repair all files.  When that was done, I went into MP3val's View > Scanned Files with Problems.  All of the bad files were reported as fixed.  I spot-checked a few.  They seemed to be fine.

I was surprised to see that, after MP3val ran, it still listed a number of files in the PROBLEM category.  The problem, in every case, was that "no VBR header is present."  Evidently this was something that MP3val could not fix.  I thought that a conversion approach (above) might solve that sort of problem.  There did not seem to be any urgency about it -- by this point, I was getting the sense that I had almost no MP3s that were absolutely unplayable and thus needed to be restored from a backup -- so I figured I could postpone repair of the "no VBR header" problem until I had learned more about archival formats (above).

I noticed, also, that MP3val had created .bak (backup) copies of the files that it had repaired.  This was apparently the meaning of the Preferences > Delete backup files option:  apparently MP3val worked by creating a backup file in the same folder as the MP3 file, making its changes, and then optionally deleting the backup.

Conclusion

I looked at a number of MP3 validation programs.  I had limited time and expertise in which to do a comparison.  I was able to eliminate some programs from consideration fairly easily.  Others were closer to the mark, and deserved a more careful look.  If I returned to this project in the future, I thought I might try to invest the time in reading the entire MP3 Diags help file and trying to get past its inflexible and alien interface.

For various reasons, MP3val presently seemed closest to what I needed.  It was possibly overkill in the sense of identifying numerous problems, apparently minor, beyond the apparently more significant problems identified by MP3Utility.  Even those more significant problems were not truly huge:  the files seemed to play normally in IrfanView, both before and after I used MP3val to fix them.  As far as I could tell, MP3 Diags was very much overkill, in the sense of identifying numerous problems that the help file then advised me to ignore, as long as my files were playing correctly.

MP3val and others did not draw my attention to a few files that had somehow gotten corrupted into a lower bitrate -- or something.  I was not sure what had happened to those files, but I could see and hear that they had lost quality compared to old backup copies.  MP3-Check provided bitrate information that I was able to copy and paste into a spreadsheet, where I could sort by bitrate to highlight such potentially problematic files.  There may have been a faster approach to this particular issue through an MP3 player/information program like Foobar2000.

Foobar2000, Winamp, and some audio editors were capable of doing batch conversions of large numbers of files.  If I did return to this project at some point, I thought it might be worthwhile to see if I could use a conversion program to fill in missing tags, homogenize bitrates, and otherwise convert these MP3s, created on different pieces of hardware, into a single, consistent format that would eliminate most if not all errors.  I would probably use MP3 Diags, MP3val, MP3-Check, and/or some other informational program to guide me in arriving at the ideal form for such a conversion.

I felt that it would probably make sense to postpone that conversion inquiry until some future point when I would investigate archival formats.  That is, I knew that formats were capable of becoming extinct, and I suspected that there was probably research and perhaps a consensus on what filetype, bitrate, and other characteristics were most likely to be supported into the indefinite future.  Ideally, I would be able too do just one conversion, check it with just one capable MP3 checker by that point, and put such concerns to rest.

Sunday, January 29, 2012

JPG: Can't Read File Header - Unknown File Format or File Not Found

I was looking at various JPGs.  I noticed that a number of them produced this error message when I tried to view them in IrfanView.  I had checked the box to activate IrfanView's Unicode plug-in as suggested, and anyway these were not exotic file names.  So I didn't know what this error would mean for these JPGs.

A search indicated that numerous people had encountered this error.  I decided to start by verifying that this was not just a quirk of IrfanView.  It didn't seem to be; I also wasn't able to view these JPGs as icons in Windows Explorer, and when I tried to view them in Firefox, I got an error:  "The image [filename] cannot be displayed because it contains errors."  When I tried Internet Explorer, I got "Your web browser has blocked this site from using an ActiveX control in an unsafe manner."  Windows Photo Viewer said, "Windows Photo Viewer can't open this picture because the file appears to be damaged, corrupted, or is too large."  Chrome didn't show an error message; it just gave me a blank page.  Photoshop said, "Could not complete your request because an unknown or invalid JPEG marker type is found."  Microsoft Paint said, "Paint cannot read this file.  This is not a valid bitmap file, or its format is not currently supported."

Eliminating the Easy Solutions

A search led to indications that some problems of this type could be due to the program, as I had feared in the case of IrfanView.  For instance, one webpage indicated that a faulty Skype extension could produce the foregoing Firefox error.  Presumably an attempt to open the JPG in some other program, as above, would help to clarify whether it was a program issue rather than a JPG issue.  A discussion thread raised the prospect that this kind of thing could result from various kinds of file system or drive problems.  Other webpage said that USB flash drives (especially improperly removed) or Picasa could be an issue.

In some cases, a backup could be a solution, possibly beginning with a DoubleKiller search for other files having the same filename (just in case there might be another copy of the same file somewhere on the computer).  That discussion thread also suggested that it could be a machine-specific problem, but it wasn't in my case:  same problem when trying to open on another computer.

As noted in a previous post, building on an earlier effort and leading to some additional refinements, it was possible to use IrfanView to detect corrupted JPGs scattered around the computer.  The basic idea was to do a search for *.jpg (in e.g., a command window, or using a file finding program like Everything) and then run IrfanView (using either File > Batch or command line methods) to see which JPGs would fail to convert to another format (e.g., PDF).

I had previously reviewed commercial software for fixing JPGs. The prices were generally high and my confidence in them was not great. I only had a few dozen corrupted JPGs and wasn't eager to spend much money on fixing them.  One commenter, responding to my post, said that she'd had generally good results with JPEG Recovery Pro ($50), but she still seemed to be looking for a better solution.  Voters on CNET had given it less than two stars.

One reviewer on CNET (like others) said the demo version of Corel's Paint Shop Pro had been useful.  On CNET, it got 3.5 stars from 466 users, though it didn't look like the dozen users who had rated the current version were quite as pleased with it.  I downloaded the latest version from CNET.  It was large (366MB) and it took a while.  CNET said it would be a 30-day trial version, $60 purchase price after that.  (I could also have downloaded from Corel's website.  Oddly, when I started to do that, the download dialog said their version was only 282MB.)  Unfortunately, when I tried to open a few of my corrupted files, Paint Shop said, "An error occurred while trying to read from the file."  Same outcome with a half-dozen different corrupted files.

Digging into the Files

My efforts (above) suggested that I might or might not be able to find a program that would help to automate the repair of corrupted JPGs.  Assuming I did find such a program, the next question would be whether it would work on all corrupted JPGs.  I was not finding a clear, obvious solution.

In other words, it seemed that, sooner or later, I was going to find myself among those who were talking of manually editing JPGs to fix them.  I hadn't done that before.  I had no idea whether that sort of process could be even partially automated.  But the next step seemed to be that I should see if I could fix at least some simple problems in JPGs.

In a thread cited above, someone said that I could open a JPG in Notepad and could tell, from its first few characters, what kind of file it was.  A GIF would tend to begin with "GIF89," a JPG would begin with "ÿØÿà," and a PNG would start with "‰PNG."  I looked at a couple of my JPGs.  Sure enough, the uncorrupted ones did begin with with "ÿØÿà."  But the corrupted ones I examined didn't have anything like any of these three options.  I knew, anyway, that it wasn't a case of the file being saved with the wrong extension.  If that had been the problem, IrfanView would have caught it (given the program options I had selected) and would have offered to change it to whatever the correct extension should be.

Another site offered a step-by-step guide to the process of editing the JPG.  The editing required a hex editor.  That site recommended Frhed, whose interface was far friendlier than that of HexEdit.  I opened one of my corrupted JPGs in Frhed.  It looked a lot less crazy than it had looked in Notepad.  According to a user-friendly version of the step-by-step guide, the JPG consisted of two sections:  header and image.  Corruption resulted from having a bad header.  Or so we hoped.  The solution was to replace the bad header with a good one.  So I made a backup of the files I would be working on, and set to work.

The user-friendly guide recommended xvi32 instead of Frhed.  It was rated 4.2 out of 5 stars (Very Good) by 47 users (76,213 downloads) at Softpedia -- vastly more than Frhed -- so I downloaded and ran that instead.  I had to bump up its font a bit -- 8-point type seemed unduly ascetic.

Now that I was getting organized, I looked at my first corrupt JPG.  Unlike the other one I had just glanced at, it did not have anything except zeroes.  This file was completely toast.  The next one had data.  The guide said I should look for "ff da" in the hexadecimal data, so I did a Ctrl-F and searched for FF DA.  (For some reason, xvi32 insisted on entering capital letters.)  My search seemed to think that it had succeeded:  it stopped at something that read 9F FA.  Call me crazy, but that did not look exactly like FF DA to me.  I tried searching the same file in Frhed.  A search for ff da found nothing, and a search for just ff didn't find much.  Going for a trifecta, I tried in HexEdit -- and there, I did find ff da. 

These hex editors, sounding an uncertain trumpet, inspired me to search for that funky string noted above -- ÿØÿà -- in Notepad.  It wasn't there.  I tried five other JPGs in Notepad.  No ÿØÿà in any of them.  I felt lost.

According to the user-friendly guide, a hex search for ff da in xvi32 should have had some luck.  I searched a good JPG in Notepad for ÿØÿà and, sure enough, there it was, right at the start of the file.  I took a look at that same good JPG in xvi32 and it, too, found ff da in a hex search -- and this time it really was ff da.  So were we saying that my bad JPGs did not have the ÿØÿà or the ff da that would be necessary for a manual repair job?  Was that why neither Paint Shop Pro nor IrfanView nor anything else had been able to do anything with these files -- were they all completely fubar?  Were people using these hex editors, searching for ff da, when instead they could have achieved the same thing with IrfanView or PSP?

Diagnosis

I had a folder full of JPGs.  Some of them worked; some didn't.  Having made a backup, I started at the beginning and viewed each of these files in IrfanView.  I had set my IrfanView properties so that it would go immediately to the next JPG upon hitting an arrow key or upon deleting a file.  In other words, I could just hold down the Del key until it came to an error, and then hit Enter and right-arrow to get to the next one.  This would delete each good JPG, which was fine, since I didn't need to be editing this copy of it.

That left me with 159 apparently bad PDFs.  Wow.  More than I expected.  This could take a lot of manual editing.  IrfanView couldn't open them, which probably meant nothing else could either.  Just to be sure, I tried opening several dozen of them in Paint Shop Pro.  No dice.

I wondered if there was a fast way of searching all 159 of these files for ÿØÿà.  Copernic Desktop Search didn't find anything like that.  Well, what if I glued a bunch of them together with the COPY command and then viewed the one huge file in Notepad?  To test this approach, I went into a command window and typed "COPY File1.jpg + File2.jpg Newfile.txt," where File1 and File2 were good JPGs.  Notepad told me that File1.jpg and File2.jpg each had exactly one occurrence of ÿØÿà.  How about Newfile.txt?  Yes, indeed, it had two occurrences of ÿØÿà.  So this approach seemed to work, at least for purposes of preserving occurrences of ÿØÿà in a concatenated file.

To run this little test, I needed a more powerful concatenator, unless I was willing to type out 159 filenames:  COPY wouldn't use wildcards (e.g., COPY *.jpg Combined.txt) -- but I had forgotten that /b would fix that:  COPY /b *.jpg Combined.txt.  (I chose a .txt extension so that (a) Notepad would open it automatically and (b) Combined.jpg would not get mixed up in copying itself into itself with the *.jpg wildcard.)  I ran that and then opened Combined.txt.  It was a large file, of course, so Notepad took a while to open it.  Once it was opened, I did a search for ÿØÿà.  I found hundreds of occurrences -- far more than 159.  Did this mean that my concatenation messed things up, or did it mean that some files had multiple occurrences of ÿØÿà?  I tried concatenating just 10 bad JPGs:  I had named them using sequential numbers, starting with ZZZ_0001.jpg, so I was able to use a wildcard to select ten:  COPY /b ZZZ_003*.jpg Thirties.txt.  The results were confusing.  Eventually I wrote a batch file to open each file individually in Notepad.  The batch file contained lines like these:

start notepad.exe ZZZ_0013.jpg
start notepad.exe ZZZ_0027.jpg
I thought I might crash the system if I opened 159 sessions of Notepad at once, so I broke it into four parts of about 40 lines each.  I ran the first one and did a Ctrl-F and then Ctrl-V in each one to paste ÿØÿà on the search line.  Now I had my answer.  Some of these files (e.g., ZZZ_0074.jpg) contained many iterations of ÿØÿà, while others contained none.  I went through them all.  After a while, it wasn't hard to guess that the ones that seemed to be filled with Chinese characters (and there were quite a few of them) would have no occurrences of ÿØÿà, while some but not all of the files containing more familiar if gibberishy characters (e.g., 1 Òhhh) would have at least one such occurrence.  There might also have been a way of speeding up the process by doing spot checks, since it seemed that files near to one another (probably originating from the same folder) (e.g., ZZZ_0058.jpg and ZZZ_0059.jpg) tended to follow the same pattern of having or not having occurrences of ÿØÿà.  In the end, 27 of the 159 had at least one occurrence of ÿØÿà, and 132 did not.

Fun with Hex Editors

These facts seemed to call for two separate approaches.  For the 132 jpgs that had no occurrences of ÿØÿà, maybe the situation was that the front ends had gotten lopped off, and that's why Paint Shop Pro et al. couldn't make anything of them.  What would happen if I just arbitrarily rammed a header onto each of these files, down to the ÿØÿà point?  The answer to that might give me some clues for the minority of files that did have multiple occurrences of ÿØÿà.

In a survey of good JPGs, I noticed that most began with this:
ÿØÿà JFIF         ÿ
There seemed to be at least one invisible character in there, so the best approach (outside of a hex editor) seemed to be be to copy it from the start of a working JPG in Notepad (i.e., not from this webpage), and save it as header.txt.  Then, starting with ZZZ_0013.jpg, the first of my bad JPGs, I typed this:
COPY /b header.txt + ZZZ_0013.jpg new0013.jpg
and then I tried to open new0013.jpg.  IrfanView gave me an error:  "Decode error!  JPEG datastream contains no image."  Tried it with a couple other files; same result.  A brief search suggested that this "decode error" problem could be just as bad as the original one.  So it appeared that this COPY approach was not the answer.

Back at the user-friendly guide, I confirmed that, in their view, the new header approach required me to locate the hex string FF DA, which I had not been able to do in many files.  Given the uncertainties I had encountered in the several hex editors (above), I wondered if there was a way to output the hex contents of a JPG in text form, so that I could do ordinary searches in Notepad (or whatever) to confirm that there was no FF DA in these files.  The solution was easy enough:  Frhed (but not HexEdit or xvi32) had an option, File > Export as hexdump, that gave me a text file displaying the hex data.  So if I wanted to see a file's hex in a text file, I could use that; and if I wanted to see a file's ASCII in a text file, I could use Notepad.  Xvi32 did have a File > Print option that gave me pretty printed pages (in e.g., PDF or hard copy), displaying both hex and ASCII, so I could have searched for either text or hex values in its PDF output.

But now, this was odd.  When I searched ZZZ_0030.jpg for "ff da" in Frhed, it found nothing; but when I searched Frhed's hex dump text file for "ff da" in Notepad, it found multiple occurrences.  Ah, but the problem seemed to be that Frhed was searching the ASCII side, not the hex.  Frhed's search box told me to consult the online help for guidance, but the program contained no link to any webpage as far as I could tell.  Xvi32's Find option let me search for either text or hex, but as noted above, it often led to 9F instead of FF.  Back in Frhed's hex dump file, I searched for the first occurrence of ff da.  It was on row 0050b8 (which the hex dump displayed as 0050b835, mistakenly running together the row number, it seemed, with the first column of data).  I tried to locate row 50b8 in xvi32, but there was no such row.  I guessed that 0050b8 must refer to a specific location (such as the "ff" in "ff da"), so what I was calling "rows" would actually have different numbers within the same file, according to how many data points were being displayed on a single row onscreen in the hex editor.  Armed with that theory, I did now see an occurrence of FF DA in xvi32, near where location 0050b8 should be; and when I clicked on that occurrence, the status bars in both Frhed and xvi32 displayed indications that I was at "hex address" or "offset" 50CD.

OK, so what could I do with this information?  There seemed to be some confusion here.  Those webpages had said that I was looking for ÿØÿà.  An ASCII code list told me that the ASCII code for ÿ was 152.  (This meant that I could type ÿ by holding down Alt and hitting 152 on the keyboard's numeric keypad.  It turned out that not all ASCII code lists agreed:  some versions of the extended ASCII table (most, it seemed) would say that 152 would give me a tilde (~), but that wasn't the tale told by my keyboard.)  According to my preferred ASCII list, the ÿ had a decimal value of 152 and a hex value of 0x98.  A converter told me that 152 in decimal = 98 in hexadecimal, so that added up.

So, ahh, now I thought maybe I was figuring this out.  Looking at ZZZ_0030.jpg in xvi32, I noticed that selecting FF in the hex area would highlight ÿ in the ASCII area.  As far as xvi32 was concerned, ÿ = FF, not 152.  I tried FF in my hex converter.  It said FF in hex = 255.  Well, and of course it did:  FF was as high as hex would go.  In hex, you don't count from 0 to 9; you count 0 to 9, and after 9 you continue on with a, b,c, d, e, and then f.  What we call 16 in decimal is called F in hex.  After F, you start over back at 0 in hex, just as you start back over at 0 after 9 in base-10 (decimal) counting.  So FF in hex actually meant 0FF:  it was similar to 099 in decimal.  After 099 comes 100 in decimal; after FF comes 100 in hex.  So never mind my keyboard:  these hex programs were interpreting FF as 255, and that was the last possible number in the 256-character extended ASCII set (beginning with zero).  In these ASCII code lists, 255 was not represented by ÿ or any other character.  Apparently it had some special meaning.  To clean up a loose end, I found an occurrence of hex 98 in xvi32, clicked on it, and saw that, sure enough, it was linked with the ASCII tilde.

I looked at a good JPG file in Frhed.  It began, as shown above, with "ÿØÿà JFIF ÿ" -- that is, with this set of hex values:
ff d8 ff e0 00 10 4a 46 49 46 00 01 01 01 01 2c 01 2c 00 00 ff
After that ending ff, the contents of the good JPGs seemed to diverge.  That 17th character seemed to be where the image content began.  Would I get a working file if, instead of pasting in that header.txt file (which, being represented in ASCII, was apparently not able to capture all of the hex nuances), I pasted these 17 (or maybe 16) codes at the start of a bad JPG?  Or, wait.  After the hex dump search experience (above), was I sure that this sequence, or part of it, was not already in those files?  The Notepad search for ÿØÿà had produced mixed results, but maybe that wasn't the right way to go about it.

(Note that the second character, Ø, was represented by d8.  There were apparently several different Ø-like characters in use in different languages.  The hex calculator indicated that d8 in hex meant 216 in decimal.  But when I typed Alt-216, I got ╪, not Ø.  That seemed to be an error, according to indications of what I should have gotten in the Latin-1 (ISO-8859-1) character set.  The answer was that I should have been typing Alt-0216, not Alt-216.)

In Frhed, I opened one of those JPGs in which, using Notepad, I had found no occurrences of ÿØÿà.  I did a search (Ctrl-F) for ÿØ, which I could either have pasted into the search box or entered via Alt-152, Alt-0216.  I did find an occurrence of ÿØ.  I noticed that it came shortly after a big section full of zeroes, which made me think that much of that particular file might be gone forever.  I tried again, this time searching for the full ÿØÿà (Alt-152, Alt-0216, Alt-152, Alt-133).  Nothing found.  So Frhed seemed consistent with Notepad in that particular search, at least in this file.

By this point, I was a bit lost.  It did occur to me that I might be able to automate the triage of potentially salvageable JPGs by doing a hex dump, counting the occurrences of 00 (nothing, empty space) as a percentage of the total number of hex values in the file, doing some sampling of partially zeroed but still readable JPGs, and identifying a threshold (10%?  20%) beyond which a JPG would not be worth saving.  But I wasn't there yet, because my JPGs weren't readable at all, and they did have non-zero data.  Most of them, that is; I had found a total of two that were completely empty.

A Bit of Clarity

I went back to the first of my bad JPGs that had no occurrences of ÿØÿà.  My hex editors all seemed to have some way to insert characters or a whole file.  The latter, offered by xvi32 and Frhed, seemed easier, so in xvi32 I went into File > Insert.  I had not yet created the file that I wanted to insert, so now I went into Notepad, pasted my string of 17 hex characters (above), saved it as Header.txt, and proceeded to insert that into my bad JPG in xvi32.  Oops:  that pasted the hex codes (ff d8 ff e0 ...) as text, not as hex.  Xvi32 also gave me an Edit > Insert > Hex string option, so I tried that.  That worked.  I saved the bad JPG and tried opening it in IrfanView.  This gave me a new error:  "Decode error!  Bogus marker length."  Unfortunately, a search led nowhere from that.  Another search produced more, including a FileFormat.info webpage that said this:
The first two bytes of every JPEG stream are the Start Of Image (SOI) marker values FFh D8h. In a JFIF-compliant file there is a JFIF APP0 (Application) marker, immediately following the SOI, which consists of the marker code values FFh E0h and the characters JFIF in the marker data, as described in the next section. In addition to the JFIF marker segment, there may be one or more optional JFIF extension marker segments, followed by the actual image data.
This was helpful.  It seemed that all I really needed, after all, was the FF d8 bytes.  The rest of "ÿØÿà JFIF" was perhaps related to JFIF compliance, but I didn't know if I needed that to produce a file that a program like IrfanView could read.  So I went back into xvi32, went to the 17th byte (i.e., the last one I had just entered) and used the Edit > Delete to Cursor option to delete what I had just added, and then used Edit > Insert String to add back ff d8.  I saved and tried again.  Now we were back to the "Can't read file header" error in Irfanview.  Another FileFormat webpage seemed to say that, to have a JPG file, all you needed was the first four bytes (ff d8 ff e0).  In xvi32, I went into File > New, inserted those four bytes, and saved that file as Test.jpg.  IrfanView gave me an error:  "JPEG datastream contains no image."  I inserted four empty bytes (i.e., eight zeroes) after those four header bytes but still got that error.  I replaced those with bytes 5 through 8 from a good JPG and tried again.  Still the same error.

Another search led to a webpage that seemed to explain something I had noticed in another FileInfo webpage:  it seemed that a JPG file was (or at least could be) defined as one beginning with FF D8 and ending with FF D9.  This page also explained that JFIF was an alternative to EXIF, but I wasn't sure whether I needed either of them.  It seemed that I hadn't really added four data bytes, when I added bytes 5 through 8:  I was just adding the JFIF part of the header.  Assuming I had to have either EXIF or JFIF, in xvi32 I now modified Test.jpg so that it contained what appeared to be the standard JFIF header (ff d8 ff e0 00 10 4a 46), then added four bytes (90 60 1B 88), then added FF D9 at the end, saved, and tried again.  Still "contains no image."  I looked at Test.jpg in Notepad.  Interestingly, it looked like I had made a start on one of those Chinese-looking files.

A more focused search tended to confirm the thought that I was getting in over my head and/or that there just might not be a solution.  People were talking about serious programming, and they were also giving me the impression that, of course, the image data for a JPG would state, or be influenced by, its size, color, compression, and other factors.  This was probably why the user-friendly webpage advised finding a header from a JPG of similar size, if possible, taken by the same camera and edited with the same software.  I didn't have that kind of knowledge about these particular JPGs, so my chances of adding a good header were limited.
The user-friendly webpage hadn't actually been very clear, to me, so I returned to the original advice page that it was trying to present in more user-friendly terms.  That page confirmed that there would typically be "several" occurrences of ff da in a JPG.  This suggested that a file without any such occurrences, searched in a hex editor or dump rather than in Notepad, could be beyond saving.  I had decided I wasn't going to make that determination today, though.  If I couldn't save something now, I was going to zip it up and save it until maybe some better tool came along.

The advice here was to look for the *second* occurrence of ff da, occurring somewhere around 2000 to 4000 bytes into the file.  I hadn't understood that from the other page.  Everything up to that second occurrence was supposedly part of the header; everything after it was image data.  So I would be replacing all of that header section with a similar section from a good JPG.  If this was right, then my attempt to create Test.jpg (above) appaerently needed a second occurrence of ff d8, followed by something resembling image data, in order to work.

At about this point, I discovered that I might have been confusing ff d8 and ff da.  Both appear in the foregoing paragraphs, and I was no longer sure which one I was supposed to be interested in.  A look at a working JPG called Good.jpg indicated that the first two ASCII characters (ÿØ -- which, pronounced yo!, could be a great way to start a JPG) were represented by hex FF D8.  But the original advice page was saying that the boundary between header and image data was marked by FF DA, not FF D8.  So apparently I confused that.  In Good.jpg, FF DA -- that is, ASCII characters ÿÚ, produced (as I now realized) by Alt-0255, Alt-0218 -- first appeared at location (would it be called "offset" or perhaps "byte number"?) 261. This was not nearly as far into the file as the advisor had suggested.  Perhaps it varied with the contents of the file.  But, no, again, this was the first occurrence of FF DA, not the second.  In xvi32, I hit F3 to repeat the search.  But xvi32 again took me to an instance of 9F DA, not FF DA.  I tried the same thing in Fhred, finding once again that it searched for ASCII, not hex; so I searched Good.jpg, in Fhred, for ÿÚ.  It said the offset or address of the first hit was at 609 or, in hex terms, 0x261.  But it could not find any more occurrences.  Was that why Fhred had taken me to the irrelevant 9F DA -- because there was only one FF DA?  (Probably not, I decided later; probably it did that because it was accepting Ÿ as equivalent to ÿ, since I had not specified a case sensitive search.)  HexEdit, too, appeared to be finding only one occurrence of FF DA in Good.jpg.

A Glimpse of Light

It seemed that I would have to try to make this work using the first rather than the second occurrence of FF DA in Good.jpg.  In Fhred, with the cursor blinking on DA at offset 609, I went into Edit > Select Block.  There, I typed x0 as the Start of Selection, and left the End of Selection at x262.  I clicked OK.  This selected everything from the start of the file to DA at offset 609.  Then I realized I was doing this in the wrong file.  But it was OK.  There seemed to be another way to proceed.  In xvi32, I went into File > Export as Hexdump.  I left the same range (x0 to x262) and selected Export to File and Just Hex Digits on a Line.  I saved it as Header.txt.  Then I closed Good.jpg.

Hopefully that gave me a working header.  In Fhred, I went into one of my bad JPGs and searched for FF DA.  It found nothing.  I double-checked, doing the same search in xvi32.  It found an occurrence of FF DA.  Plainly, I had still not quite gotten the hang of using these hex editors, or else maybe Fhred really was buggy, as the writer of the user-friendly webpage believed.

Viewing Bad.jpg in xvi32, the first occurrence of FF DA that xvi32 found was near the end of the file.  That couldn't be an end-of-header marker, could it?  The hex address was 4E6D4, indicating that there was a lot of data before this point.  My guess was that this was one of the later occurrences of FF DA, not the early occurrence that would mark the header.  Could I fix this bad JPG by just attaching Header.txt at the start of the file?  In Fhred, I used Ctrl-Home to go to the start of Bad.jpg.  There, I saw nothing that looked like ÿØ, which presumably would have appeared at the start of any good JPG.  I went into File > Import from Hexdump.  I named the newly created Header.txt as my source and clicked OK.  I got a question:  "Does this data have the same format as the Fhred display?  This data contains only whitespace and hexdigits. (unlike Fhred display)."  I assumed that a header would naturally get a question like this, so I clicked Yes to proceed.  It said, "Unexpected end of data found.  Cannot continue!  Do you want to keep what has been found so far?"  I clicked Yes.  It gave me a blank screen.  As you might have predicted, this did't fix Bad.jpg.

I tried again, this time following the instructions more closely, in case that made a difference.  Specifically, in xvi32, I saved Good.jpg as Header.txt, searched it for FF DA (case sensitive), went to the byte immediately after DA, and then went into Edit > Delete from Cursor.  Finally:  Header.txt really did contain only the information up through FF DA.  I saved it and opened Bad.jpg.  Still in xvi32, and with the cursor located at the start of Bad.jpg, I went into Edit > Insert > Header.txt.  This, I hoped, would prepend a good header to the image body of Bad.jpg and heal it.  I saved Bad.jpg and tried opening it in IrfanView.  Sadly, I got "Bogus marker length."  Paint Shop Pro couldn't open it either.

I tried again with a different Bad.jpg.  This was one of the files that I had identified (above) as having many instances of ÿØÿà, whereas the previous Bad.jpg (i.e., the one I had just been experimenting with) had none.  I searched for the first instance of FF DA, used Ctrl-Shift-PgUp to mark everything up and through FF DA, pressed Del to delete it, and then inserted Header.txt at the start of the file.  I saved it and tried opening it in IrfanView.  Again, "Bogus marker length."  I tried again, this time going to the second instance of FF DA.  "Bogus marker length" once again.

Wrap-Up

I was out of time for this project.  Perhaps some ideas would come to me later, or I would become aware of some new program or technique.  As always, comments and suggestions were welcome.  In the meantime, all I could do at this point was to archive these bad JPGs in a zip file and put them aside.