Showing posts with label iso. Show all posts
Showing posts with label iso. Show all posts

Wednesday, May 23, 2012

A Million-Day Calendar with Explicit Julian-Gregorian Comparison

I wanted to look up a historical date.  Specifically, I wanted to know which day of the week it occurred on.  As I was looking for an answer to that question, I gradually came to the impression that there did not exist a standard calendar.  I decided to build one.  This post describes that process.

Someone may already have created what I was looking for.  But I wasn't finding it.  What I was looking for was, simply, the Official Calendar.  Of the United States, of the Catholic Church, it didn't matter -- just an official calendar that some reputable body had actually committed to print (preferably with explanations, and without errors).

What I was finding, instead, was lots of rules about how to calculate an official calendar, as well as various tools that would assist in those calculations.  This was fine, as far as it went.  But we don't generally tell people who prefer the Celsius temperature scale to just use the Fahrenheit and convert it.  Instead, people living in places that use Celsius have thermometers that show them the literal answer, without the need for a manual conversion process.  I wanted something like that for calendar dates.

Ultimately, I created a calendar covering a million days, starting on January 1, 500 BC. I produced that calendar as a spreadsheet, printed it as a PDF, and made both available for download. I don't often revisit this post. As of an update in early 2023, these materials are available for download through my SourceForge project page or at Box.com or MediaFire. See also my download blog post.

The PDF is a 38MB, 10,000-page document.  I would not recommend printing more than necessary.

Assumptions and Calculations
Built into the Million-Day Calendar

I chose Excel 2010 to develop the calendar because that version of Excel could accommodate somewhat more than a million rows.  I did not use Excel's built-in date arithmetic, though, because of its known errors.  That is, I did not ask Excel to calculate the necessary dates automatically.  Instead, I calculated them in a semi-manual process.  The process was not entirely manual, because I did not calculate row by row, day by day, for each of the million days shown.  Instead, I developed formulas that would count forwards or backwards from a certain date, and I applied those formulas to the million rows, usually broken into several segments due to historical changes in calendar calculation.  There were some manual adjustments as well.

I found that a million days would cover approximately the period from January 1, 500 BC to the year 2238 AD.  This seemed like a good range for most purposes. For dates outside this range, there would still be the option of using a formula or calculator, or of adding another tab to extend the spreadsheet.

As shown in the preceding paragraph, I was inclined to use AD and BC to refer to calendar eras.  AD was short for Anno Domini (Latin for "in the year of the [or "our"] Lord"). AD and BC (short for "Before Christ") were thus based on an early medieval calculation of the number of years before or after the birth of Jesus. This religious origin was an addition to other religious origins (e.g., "Thursday" deriving from "Thor's Day"). Instead of AD and BC, an apparent minority of non-Christians preferred to use CE (short for "Christian Era" or "Common Era" or "Current Era") and BCE.

Traditional chronology did not incorporate a year zero (i.e., 0 AD or 0 BC).  That is, the calendar went directly from 1 BC to 1 AD.  The original concept may have been that there was no need for a year zero, since Jesus was not born until the start of the first year of his life (incorrectly calculated as 1 AD).  This variation would make no practical difference in the AD era:  for example, the number 2012 represented the year in which this post was written.  It would lead to difficulties in the BC era, however.  For instance, the rule on leap years (involving division by 4) would produce a leap year in the year 4 AD and, before that, in the year 0; but since there was no year 0, the prior leap year was in 1 BC.  Hence, traditional BC dates did not fit exactly with the rule that leap years are evenly divisible by 4.

The calendar in effect at the time of Jesus was the Julian calendar, introduced by Julius Caesar in 46 BC -- a year which, by decree, was 445 days long.  The Julian calendar was revised several times, finally stabilizing in 4 AD.  For present purposes, the key innovation of the Julian calendar was the decision to define the year as equal to 365.25 days, adjusted via leap years in every year evenly divisible by 4 (e.g., 2008, 2012, 2016).  The Julian calendar eliminated the leap month Mercedonius but did not otherwise significantly change the names or lengths of months.  For purposes of year numberingepochs (i.e., reference years) in the early centuries of the Julian calendar commonly used regnal systems based on the current ruler or other officials (e.g., "January 1 in the second year of the reign of the Emperor Justinian"), but there was a semi-chaos of other epochs as well.  For instance, the Anno Mundi era started from calculations of the date on which the world was created, and the Ab urbe condita era started from the hypothesized date when Rome was founded.

The big change after the institution of the Julian calendar came in 1582 AD, when Pope Gregory XIII introduced the Gregorian calendar.  The Gregorian reform assumed the use of AD rather than regnal or other epoch systems; the AD epoch concept had been gradually spreading during the Middle Ages.  Gregory's principal contribution was to revise leap year calculations.  Over the centuries, the Julian calendar had become increasingly inaccurate with respect to the actual equinox.  That is, the calendar might say that it was March 21 -- the time for Easter -- and therefore daytime and nightime should each be about 12 hours long; but in fact, according to the clock, that day would already have arrived more than a week earlier.

In other words, the Julian calendar was falling behind the real world because the calendar was inserting too many leap years.  The extra leap days were making the Julian calendar late:  it would say the date was only March 11, when it really should have been March 21.  Gregory thus removed ten days from the calendar for October 1582, to catch up, and also changed the leap year calculation slightly.  The Gregorian rule for leap years was that every year evenly divisible by 4 would still be a leap year, except that years evenly divisible by 100 would not be leap years unless they were also evenly divisible by 400.  So 1700, 1800, and 1900 would not be leap years, but 1600 and 2000 would be.

This adjustment was still not perfect, but because of gradual slowing in the Earth's rotation, it was apparently pretty close.  The slowing issue, which I did not explore, may have been related to the difference between the tropical year and the sidereal year.  The Julian and Gregorian calendars were apparently based on the tropical year, which was the amount of time that it took the Sun (as seen from Earth) to come back to the same place as it was on the previous vernal (spring) equinox.  The sidereal year was an alternative to the tropical year:  it was the amount of time that it took Earth to return to the same relative position as it had occupied a year earlier, as measured with reference to certain stars.

These findings about the Julian and Gregorian calendars called for some decisions, for purposes of constructing a million-day calendar.  One such problem had to do with the present day.  My computer might tell me that it was May 6, 2012.  This would be a date in the Gregorian calendar.  Its appearance on my computer, my wristwatch, and everywhere else would testify to Gregory's widespread success.  I knew, however, that there was also a Chinese New Year and a Jewish calendar and all sorts of other calendars that still had meaning for various cultural and religious purposes, as well as the similarly named but essentially unrelated Julian Year system used in astronomy.  Even the Julian calendar continued to be used in Eastern Orthodox churches.  I decided that the intended spreadsheet approach to the million-day calendar might enable others to add these alternative calendars as they wished.  Because of the size of the spreadsheet and the relative rarity and potential complexity of these other calendars, however, I decided that I would not try to build any of these alternatives into the calendar myself, but would instead focus on the Julian and Gregorian calendars that predominated in the West during the timeframe addressed in the million-day calendar.

Another problem had to do with adoption dates.  The Gregorian adjustment of October 1582 specified that the Julian calendar would end on October 4, 1582; the days of October 5 through October 14 (inclusive) would not exist; and the Gregorian calendar would begin on October 15, 1582.  This rule was adopted at very divergent rates:  immediately, in several Roman Catholic countries, but elsewhere with considerable delays and confusion continuing into the 20th century.  The problem here, then, was that October 5, 1582 did not exist in Spain, and yet someone in England could be staring at a letter dated October 5, 1582, and that would make perfect sense according to the Julian calendar, which would continue to be used in England until 1752 (at which point England would need to delete eleven days, not ten, to get in sync with the Gregorian reform).  During the transition period in England, people commonly used the terms "Old Style" (abbreviated as "O.S." in English, and as "st.v." in Latin) to refer to the Julian date, and "New Style" ("N.S." or "st.n.") to refer to the Gregorian date.

As just described, the Gregorian calendar officially began (and was officially implemented in some places) on October 15, 1582; the Julian calendar officially ended on the preceding day, which (according to the Julian) was October 4, 1582.  But one could also say that October 4, 1582 (Julian) was the same as October 14, 1582 (Gregorian).  This way of looking at the matter would require proleptic (i.e., anachronistic) calculations.  Specifically, there would be a proleptic Gregorian calendar for all dates before October 15, 1582 on the Gregorian calendar, and there would also be a proleptic Julian calendar for all dates before January 1, 4 AD on the Julian calendar.  October 13, 1582 (Gregorian) would be the same as October 3, 1582 (Julian); October 12 (G) would be the same as October 2 (J); and so forth, back in time.

Since the Gregorian calendar did not exist before 1582, the statement that the Battle of Hastings occurred on October 14, 1066 would imply that it was October 14 according to the Julian calendar, not the Gregorian.  While it could be confusing to cite proleptic Gregorian dates for events that were made part of history according to the Julian calendar, there seemed to be some applications for which a proleptic Gregorian calendar could be useful.  For example, someone might be interested in determining whether a certain event happened on the actual equinox, as distinct from the date represented as the equinox in the Julian calendar.  In developing the million-day calendar, I thought it would thus be useful to display Julian and Gregorian dates side-by-side, so as to confirm the accuracy of the calendar and/or of others' conversions between the two, as described more fully below.

To a much greater degree than the proleptic Gregorian calendar, it seemed that the proleptic Julian calendar could be useful for a variety of historical situations.  The concept here was, in essence, that one could work backwards to construct a Julian calendar for dates long before Julius Caesar, and could use that calendar to construct a list of standard dates when various historical events occurred.  Although sources rarely seemed to specify what calendar they were using, it appeared that the proleptic Julian calendar was in fact being used widely for this purpose.  There would certainly be scholarly disputes as to the conversion of ancient chronologies to Julian calendar terms (so as to interpret, for instance, a statement that a certain event occurred in the 245th year since the founding of Rome), but at least the calendar system itself would be consistent over centuries.

Developing and Testing the Million-Day Calendar

I added proleptic Julian calendar calculations to the million-day calendar. I started these calculations by adding a separate Julian Days table to the spreadsheet. The concept of the Julian Day was proposed by Joseph Scaliger in 1583. Julian Days were simply a count of days, beginning (for astronomical and historical reasons) with Day Zero at 12:00 noon on January 1, 4713 BC. (Julian Days could include decimal values for fractions of a day, such as 0.083 = 2 PM.) So, for instance, Julian Day 7 arrived at noon on January 8, 4713 BC.

There were no years in the Julian Day system, but Julian Days could be used to calculate the proleptic Julian calendar, in which every fourth year would be treated as a leap year.  Because there was no Year Zero in the Julian calendar, Scaliger's first year of 4713 BC was a leap year.  (That is, in a system that had a Year Zero between 1 BC and 1 AD, 4713 BC would have been called 4712 BC.)  The resulting calculations produced Julian dates, in the spreadsheet, that were consistent with those reached by John Herschel in his Outlines of Astronomy (1849, p. 595).  Specifically, January 1, 4004 BC was Julian Day 258,963; the destruction of Solomon's Temple (which Herschel put on May 1, 1015 BC) was on Julian Day 1,350,815; and Rome's founding (which Herschel put at April 22, 753 BC) was on Julian Day 1,446,502.  Moving into the million-day period beginning on January 1, 500 BC (Julian Day 1,538,799), the spreadsheet matched Herschel's calculation that the Julian calendar reformation of January 1, 45 BC occurred on Julian Day 1,704,987; the Islamic Hijra calendar began on Julian Day 1,948,439 (July 15, 622 AD); and the official last day of the Julian calendar (October 4, 1582) was Julian Day 2,299,160.  It tentatively seemed that the spreadsheet's Julian calendar portion was accurate.

I also added Day of Week calculations to the spreadsheet, beginning with the common assertion that January 1, 4713 BC was a Monday (in, implicitly, the proleptic Julian calendar).  For the dates cited in the preceding paragraph, these calculations indicated that January 1, 4004 BC was a Saturday; May 1, 1015 BC was a Friday; April 22, 753 BC was a Tuesday; January 1, 500 BC was a Thursday; January 1, 45 BC was a Friday; July 15, 622 AD was a Thursday; and October 4, 1582 was a Thursday.  Further, I extended the Julian calendar beyond its official end to Thursday, November 7, 2238 AD (Julian Day 2,538,798).  According to the spreadsheet (and also the Julian Day arithmetic, i.e., Julian Day 2,538,798 minus Julian Day 1,538,799), that was the millionth day (inclusive) from Thursday, January 1, 500 BC.  These particular Julian Day numbers and day-of-the-week calculations matched the values produced by an online date calculator appearing on a NASA webpage.  It tentatively seemed that the spreadsheet's Julian Day calculations were corresponding accurately with Julian calendar dates.

Next, I produced a proleptic Gregorian calendar in the million-day calendar, adjacent to the Julian calculations.  The starting point for this calendar's calculations was its commonly recognized starting date of Friday, October 15, 1582.  As noted above, the preceding day of October 14 on the Gregorian calendar (G) (if such a date had officially existed on that calendar) would have been Thursday, October 4 on the Julian calendar (J).  So the spreadsheet's presentation of Julian and proleptic Gregorian dates had to match up on the row containing the values of October 14 (G) and October 4 (J).  That is, both had to have the same Julian Day value of 2,299,160.  From October 14, 1582, I extended the Gregorian calendar back to January 1, 500 BC.  I decided not to extend this proleptic Gregorian calendar back into the period before 500 BC, though there were situations in which such an extension might have been useful.

There were some interesting things in the relationship between the proleptic Gregorian calendar and the Julian calendar.  At the starting point in the 16th century AD, the Gregorian dates were later than the Julian.  As just noted, October 14, 1582 (G) was equivalent to October 4, 1582 (J).  The Gregorian allowed fewer leap years, so the difference between it and the Julian began to narrow with each additional century (except for those evenly divisible by 400), going back in time.  The ten-day difference of 1582 thus became a nine-day difference on the first previous day when the formulas for the two calendars differed:  there was no February 29, 1500 (G), but there was a February 29, 1500 (J).  By the time one arrived back at the third century AD, the difference between the two calendars vanished.  That is, as noted by Peter Meyer, the two calendars had exactly the same dates from March 1, 200 AD to February 28, 300 AD.  This was no coincidence.  Gregory had designed his reform so that Easter would occur at about the same time as it had occurred in 325 AD, when the Council of Nicea (also spelled Nicaea) discussed such matters.  So during the century ending on February 28, 300 AD (J), both calendars showed the same dates (e.g., February 1, 300 (J) = February 1, 300 (G), and both are Julian Day 1,830,664).  Before the third century, the Gregorian calendar predated the Julian by progressively larger amounts, until January 1, 500 BC (J) would be represented as December 27, 501 BC (G).  Going back still farther, dates on the Julian calendar would continue to fall three days later every 400 years, so that January 1, 4713 BC (J) would arrive a month earlier on the Gregorian, in late November 4714 BC.  On the other extreme, in the centuries following 1582 AD, the Gregorian dates became progressively later than those of the extended Julian, until November 7, 2238 AD (J) was equivalent to November 22, 2238 (G).

I checked the foregoing dates and days of the week using another online calculator as well, produced by Fourmilab Switzerland.  I began by entering Julian Day numbers and then seeing what results this calculator would produce for Julian and Gregorian calendar dates.  This calculator took the approach of inserting a Year Zero in the proleptic Gregorian calendar, so its statement of BC dates differed from the values shown in the spreadsheet by one year.  For example, the Fourmilab calculator indicated that January 1, 45 BC (J) was equal to December 30, 45 BC (G), whereas the spreadsheet would put the latter as December 30, 46 BC (G).  Fourmilab's approach seemed incorrect in this regard.  For mathematical purposes (as in e.g., the ISO 8601 approach, below), there would need to be a Year Zero; but the historical reality seemed to be that proleptic calculations in both Julian and Gregorian calendars did not have a year zero.  Fourmilab was not alone here; the conflation of mathematical consistency with historical fact had evidently produced some confusion in other computing situations as well.  At any rate, after adjusting for that divergence in BC years, the results of the Fourmilab calculator did match up with those yielded by the spreadsheet and the NASA calculator.  This calculator and the spreadsheet also agreed that February 1, 200 AD (G) was Julian Day 1,794,140 and was also February 2, 200 AD (J).  (The NASA calculator did not do proleptic Gregorian calculations.)

I looked at one other online calculator, produced by CSGNetwork.  I did not attempt a redundant comparison against all of the dates listed above.  Instead, I focused on the especially problematic period of the first two centuries AD.  In that timeframe, the CSGNetwork calculator seemed to be in error.  Specifically, a "Calendar Date Entry" of January 1, 1 AD yielded Julian Day 1,721,425.5.  The NASA and Fourmilab calculators and the spreadsheet agreed that January 1, 1 AD (J) should rather be Julian Day 1,721,423.5 or 1,721,424.  So if "Calendar Date Entry" in the CSGNetwork calendar was intended to refer to a Julian calendar date, its Julian Day output was incorrect.  It did not appear that the calendar intended to refer, rather, to a Gregorian calendar date of January 1, 1 AD, because it then stated that its Julian Day value of 1,721,425.5 was equivalent to January 3, 1 AD (G).  In that latter regard, it was correct.

To some unknown extent, online calculators presumably used formulas that had been devised to facilitate date calculations.  For example, Bill Jefferys presented a formula for converting Julian Days (and, perhaps, dates on the Julian calendar) to the proleptic Gregorian calendar, but indicated that it would be inaccurate before 1582, and especially for years before 400 AD.  Paul Dohrman offered a procedure for converting Julian to Gregorian, and J.R. Kambak offered one for conversions from Gregorian calendar dates to Julian Days.  Dohrman's approach, as I understood it, required these steps:
  1. Truncate to centuries (e.g., 622 AD becomes 6).  In the case of BC dates, treat them as negatives and start by subtracting a year first (e.g., 499 BC becomes -500, which becomes -5).  This calculation produces X.
  2. Calculate 0.75X minus 1.25.  So 622 AD » 6 » 3.25 (using » as shorthand for "becomes"), and 499 BC » -5 » -5.
  3. Truncate decimal points.  So 622 AD » 6 » 3.25 » 3.  This is the number of days to add to the Julian date to find the Gregorian.
This procedure produced some results consistent with the spreadsheet and the Fourmilab calculator, converting July 15, 622 AD (J) to July 18, 622 AD (G), and January 1, 500 BC (J) to December 27, 501 BC (G) (after Year Zero adjustment).  This procedure did not seem to work in the first two centuries AD, however.  For example, in the case of July 1, 1 AD (J), Dohrman's approach seemed to yield the incorrect value of June 30 (i.e., century 0 * 0.75 – 1.25) rather than June 29 (G).

There also seemed to be a problem with Kambak's long formula for converting Gregorian dates to Julian Days.  It is possible that I did not copy or interpret that formula correctly.  The version that I tested was as follows, where Y = Gregorian year, M = Gregorian month, D = Gregorian day, and JD = Julian Day:
JD = 367Y – 7(Y+(M+9)/12)/4 – 3((Y+(M–9)/7)/100+1)/4 + 275M/9 + D + 1721029
As I translated this into an Excel formula (placed into cell D2), it read as follows (assuming the values of Y, M, and D were entered into cells A2 through C2, respectively):
=367*A2-7*(A2+(B2+9)/12)/4-3*((A2+(B2-9)/7)/100+1)/4+275*(B2/9)+C2+1721029
That formula's results varied from those produced by the Fourmilab calculator for certain dates checked above, such as July 1, 1 AD (G) and October 14, 1582 (G).  The variance in these instances was very small, however.  Specifically, the values for those two dates produced by the formula and the Fourmilab calculator were 1,721,606 vs. 1,721,606.5, respectively (for July 1, 1 AD (G)) and 2,299,159 vs. 2,299,159.5, respectively (for October 14, 1582 AD (G)).  That is, the Fourmilab calculator exceeded the formula's output by only 0.5 day in each case.  Unfortunately, this variation was not consistent.  For July 15, 622 AD (G), the Fourmilab calculator produced a value of 1,948,435.5, which was 0.5 day smaller than the Julian Day value of 1,948,436 produced by the formula.  Moreover, for November 22, 2238 (G), the Fourmilab calculator's output of 2,538,797.5 was 1.5 days larger than the figure of 2,538,796 produced by the formula.  In each of these several instances, the spreadsheet agreed, again, with the results produced by the Fourmilab calculator, after rounding the latter's 0.5-day output upward.  It appeared, in short, that this formula was very close but not entirely accurate.

By this point, checking of the spreadsheet had begun to transition into critiques of the ways in which various calculators and other tools had interpreted and applied various sources (e.g., Tantzen, 1960). I took this as a preliminary indication of the potential usefulness of the million-day spreadsheet, at least where an explicit presentation of dates might facilitate visualization of calendar developments.  While further usage and testing would be helpful in identifying points at which errors might have crept into the spreadsheet, it did preliminarily appear that the spreadsheet could provide a useful tool for date calculations and conversions.

The ISO 8601 Refinement

I developed the Gregorian section of the spreadsheet in one additional way. The International Organization for Standardization (ISO) had produced a standard prescription (known as ISO 8601) for calculating dates.  This prescription appeared likely to be useful for a variety of purposes, so the spreadsheet contains a column devoted to it.

The ISO 8601 standard adopted Gregorian date numbers. One effect of the standard, for present purposes, was to prescribe standard ways of representing dates. There was a YYYY-DDD ordinal date option, which used the day of the year, where day 366 would have a value only in leap years (e.g., 2012-366 = December 31, 2012).  In the spreadsheet, I used the year-month-day format (e.g., 2012-05-06 = May 6, 2012). ISO year values were ordinarily displayed with four characters (e.g., padded with leading zeros in 0023 rather than 23) for consistency.

A second effect of ISO 8601 stemmed from its adoption of a Year Zero, with apparently the same effect as what was sometimes called astronomical year numbering.  In this approach, before the epoch of 1 AD, the absolute value of the ISO year was one less than the traditional year (e.g., ISO year 0000 = 1 BC; ISO year –0001 = 2 BC). So the million-day calendar started on ISO date -0500-12-27 (i.e., December 27, 501 BC (G)). The numerical approach of ISO 8601, using minus signs instead of "BC" and likewise dispensing with "AD," had the advantage of avoiding controversy regarding the use of those two traditional modifiers.  The Fourmilab calculator (above) appeared to be implementing an ISO 8601 approach in its calculation of BC dates.

With the Gregorian calendar presented in ISO format, it would have been possible to apply another kind of check to the spreadsheet's day-of-the-week column.  This check would have used what was known as the Doomsday technique.  That technique, useful for quickly calculating the day of the week for a given date, seemed unnecessarily complicated within the million-day calendar spreadsheet, where one could simply use the Julian Day.  That is, since Julian Day 0 occurred at noon on Monday, January 1, 4713 BC, every Julian Day evenly divisible by 7 would be a Monday.  This way of calculating the day of the week, for a given date on the Gregorian calendar, seemed to produce the same results as I had calculated by using a formula that copied, into each day-of-week cell, the name of the day that appeared in the 7th preceding row.

Official and Local Calendars

As previously noted, Gregory intended that the last day of the Julian calendar (October 4, 1582) would be followed by the first day of the Gregorian calendar (October 15, 1582).  That intention was followed in a number of countries and, at this writing, was implemented in various online calculators (e.g., those appearing on U.S. Naval Observatory and NASA webpages).  It appeared that 1582 was the most plausible candidate for the year in which the world converted from the Julian to Gregorian calendars.  In short, this combination of proleptic Julian (to 4 AD), Julian (from 4 AD to 1582 AD), and Gregorian (since 1582 AD) appeared to form the most credible version of the world's official calendar.  The spreadsheet thus expresses what appears to be the Official Calendar that I had sought at the outset.

Some remarks appearing in preceding paragraphs have already acknowledged certain aspects of that de facto official calendar.  For one thing, the concept of the Julian Day was built from a starting date calculated according to Julian reckoning, but came to serve as a means of cross-reference between the Julian and the later Gregorian calendars.  So the spreadsheet column that presents the Julian Day number corresponding to a particular day on the Julian or Gregorian calendar does not belong solely within either the Julian or Gregorian sections of the calendar.  Rather, it seemed to be best presented in the spreadsheet's Official Calendar section.

Likewise, a given date would be a Monday, or a Tuesday, or some other day of the week, regardless of the date number given to it on the Julian or Gregorian calendars.  So it would have been redundant to present separate day-of-week columns in each of those calendars' parts of the spreadsheet.  Instead, the day of the week appears just once, in the Official Calendar section.

That section also presents the official date, in two different formats.  First is the traditional format, using BC or AD indicators of era.  These traditional dates are provided in the somewhat condensed but still recognizable YYYY-MM-DD form.  As such, their components (e.g., the number of the month) are accessible for further date calculations, as users may desire, with the aid of Excel text functions (e.g., MID, FIND).  The column presenting the Official Date in Traditional Format is thus the specific statement of the Official Calendar in approximately the form that now appears to be used by most people.

Second, the spreadsheet also presents the official date in ISO format -- specifically, with minus signs and a Year Zero, modifying the traditional presentation.  To emphasize, this is the official date.  It uses the Julian calendar for dates before October 15, 1582, and therefore is not the ISO 8601 date.  It is simply an indication of how the traditional, official date looks when stated in ISO style for purposes of numeric calculations.

As noted above, substantial portions of the world did not adopt the Gregorian reforms in 1582.  The spreadsheet is adaptable for purposes of developing localized versions that may accommodate reforms implemented in later years.  In the process of preparing this post, I also found a useful calendar with local customizations at TimeAndDate.com, though a brief look suggested the presence of inaccuracies like those identified in other calculators (above).

Uses of the Million-Day Calendar

This post has explained the creation of a million-day calendar covering the period from 500 BC to 2238 AD.  That calendar is provided in spreadsheet format, one row per day.

This spreadsheet format seems to have facilitated identification of potential errors in certain tools designed to assist in use of, and interactions between, the Julian and Gregorian calendars as well as the Julian Day and ISO 8601 date systems.  It may prove useful in other contexts calling for calculations, demonstrations, or cross-comparisons among calendars and systems, including some that users may add.

The spreadsheet presentation may also be useful in less technical, more data-oriented applications.  Within the limits of computing power and spreadsheet capacity, there may be tasks that call for an ability to add columns of information, to be filled at a rate of one item per day (or week, or other time period).  For instance, at this writing, I would like to find a database (if one exists) that would show something like the leading headline of the day -- the sort of thing that one might expect to find on the front page of the New York Times, for instance, if that newspaper had existed on the day of the Battle of Hastings.  If no such database exists, perhaps this spreadsheet, shared among a number of potential contributors, could help to bring about its existence.

Tuesday, May 8, 2012

Creating a Bootable Windows 7 USB Drive for Installation / System Repair / Recovery - First Cut

Normally, if I booted a computer from a Windows 7 installation DVD, I could get into System Recovery Options (e.g., Startup Repair, System Restore, Command Prompt) that would let me run various diagnostics.  Unfortunately, my laptop did not have a CD/DVD drive.  So if I wanted to see those Windows 7 startup repair options, it seemed that I would have to find a way to do so by booting the computer from a USB flash drive instead.  This post describes the steps I took to develop a USB drive that would give me those options.  It also incidentally describes how to make a bootable copy of the Windows 7 installation DVD on a USB drive.

One approach was to put the entire Windows 7 installation DVD on a USB drive.  The DVD contained about 3GB of material, so this would require a USB drive of 4GB or larger.  Another approach was to put just a Windows 7 System Repair or Recovery CD on a USB drive.  This would require only about 150MB, so I could use a smaller, older, cheaper, or otherwise unused USB flash drive.  The Recovery CD option might load faster than a full Windows CD, but it would not be useful for installation or for recovering system files.

Either way, the first step was to get the necessary files.  The Windows installation files would traditionally be purchased on a DVD, but it was also possible to download them.  Similarly, the Windows 7 System Repair Disc was ordinarily a CD, but it could be copied or converted to files on a hard drive.

To get a System Repair Disc, I had to search my computer for "system repair disc."  That didn't work in my case -- I must have renamed the relevant shortcut -- so I searched for various combinations of "create," "system," "repair," and "recovery."  I could also have used Control Panel > Backup and Restore > Create a system repair disc.  The option of downloading the file(s) needed for a system repair CD was apparently disappearing.  In any case, eventually I found and used the link to a little Windows 7 program whose title bar read simply, "Create a system repair disc."  This created the recovery CD.

Next, the files that weren't already in ISO format needed to be converted to ISO.  The downloaded versions of Windows 7 evidently came in ISO format.  By contrast, the installation DVD and the recovery CD were not in ISO format.  To convert them to ISO, I started by using Magic ISO Maker.  It warned me that it would not create an ISO larger than 300MB, but this seemed to be a bluff to motivate an immediate purchase.  Format Factory would apparently have been one among many freeware alternatives.  When I remembered that ImgBurn would create ISOs from files or discs, however, I deleted the Magic ISO output and used ImgBurn instead, since it had worked well for me in other sorts of projects in the past.

Once I had an ISO, I had a choice between two different approaches to get it properly unpacked and operational on the USB drive.  A dedicated USB drive would focus solely on one version of Windows 7 (e.g., 32-bit vs. 64-bit, Home vs. Ultimate).  This dedicated approach seemed likely to be relatively simple and reliable, and would probably be all that most users would need.  By contrast, a multiboot USB drive would allow the user to install and/or run two or more different operating systems (potentially including e.g., Windows XP and Linux).  I decided to go with the dedicated, single-system approach.

I started with the Windows 7 system recovery CD, which ImgBurn had now converted to a file I called Win7SysRepair.iso.  There seemed to be several ways to put this ISO onto a bootable USB drive.  One approach involved using Grub4DOS.  Another was to use Microsoft's Windows 7 USB/DVD Download Tool.  I ran that Tool.  It called for a few simple steps.  First, I plugged in the little 512MB USB flash drive on which I was going to install the Windows 7 system recovery CD files.  Then I pointed the Download Tool toward the newly created Win7SysRepair.iso.  I clicked the USB Device button, and the Tool found the USB drive.  I clicked Begin Copying and confirmed that it was OK to erase the USB drive.  The tool said, "Creating bootable USB device."  The first time I tried it, it failed, with this error message:

We were unable to copy your files.  Please check your USB device and the selected ISO file and try again.
I assumed this was due to interference from AntiRun, which I was using to keep an eye on USB drives.  I shut down AntiRun and tried again.  But no, the Tool failed the second time too.  To troubleshoot this problem, I ran a search and saw that this was a rare error.

The problem seemed to be that the Tool was formatting the USB drive as NTFS.  I thought the solution would be to go to Start > Run > diskmgmt.msc and quick-reformat the USB drive with a FAT32 file system (using a volume label of no more than eight characters).  But I still got the same error.  Another source said the problem was that the Microsoft programs (diskmgmt.msc and also the Tool) failed to use the Clean command.  In other words, my USB stick had residual formatting from some previous use.

The advice was to fix this problem by opening a command window with Administrator rights and type "diskpart" at the prompt.  This started the DiskPart program, with its own DISKPART> prompt.  The next step was to type "list disk" to see what drives were connected to the computer.  This showed me that, as expected, the last disk was the smallest:  491MB.  That was surely my USB drive.  (It seemed pretty important not to be reformatting the wrong drive.)  That 491MB drive was Disk 2.  So I typed "select disk 2."  It informed me that Disk 2 was now selected.  I typed "list disk" again to check and, sure enough, there was an asterisk next to Disk 2.  So I was ready to type "clean."  It said, "DiskPart succeeded in cleaning the disk."  With that done, I could type these remaining commands in DiskPart, one at a time:
create partition primary
select partition 1
active
format quick fs=fat32
assign
exit
I exited the command window and tried Microsoft's Windows 7 USB/DVD Download Tool again.  It still failed.  I tried again, this time using a different USB drive.  This time was even worse:  previously, it had failed at the 99% mark, but with this drive the copying process didn't even start.  I tried using an ISO built from a System Recovery CD created on another computer, running a different version of Windows.  But the Windows Download Tool said this:
Invalid ISO File

The selected file is not a valid ISO file.  Please select a valid ISO file and try again.
I got that error twice, with ISOs created by ImgBurn and also by Magic ISO Maker.  It was time to give up on the Microsoft Download Tool, reformat the USB drive, and try another approach.

I went back to look at the Grub4DOS approach mentioned above.  I wouldn't be using it to install multiple bootable operating systems on my little 512MB USB flash drive, but it looked like a straightforward process anyway; I figured maybe the education would come in handy later.  For this approach, I needed to download and install MultibootISO.  I found what appeared to be a popular, current version of this program on a Pendrivelinux webpage.

On closer inspection, what we downloading was now called YUMI (short for Your Universal Multiboot Installer).  YUMI was apparently a successor to both MultibootISO and Universal USB Installer.  YUMI was portable; no installation required.  YUMI didn't have a built-in option for installing Windows 7.  I got the feeling that YUMI was not going to replace MultibootISO for this particular task.  Nonetheless, I tried.  In YUMI, I selected "Try an Unlisted ISO."  YUMI didn't complain that the ISO was invalid.  It seemed to think it had succeeded.  Sadly, the USB drive wasn't bootable, at least not in the laptop where I tried it.  I tried again and, whoa, success!  Apparently I had just not hit Esc quickly enough to bring up my laptop's bootable USB drive menu when the laptop was first starting up, or maybe I had hit Esc too many times and escaped my way right out of that menu.  But now, on this second go, YUMI gave me the Windows 7 recovery CD functionality, running from my USB drive.

Well.  This YUMI thing was pretty cool.  When I started this post, I thought I would just be content with the Windows 7 installation DVD. For that purpose, my spare 4GB USB flash drive was sufficient.  But now I wanted to try YUMI with a large USB drive that would accommodate the Windows 7 installation DVD as well as other operating systems and other bootable CDs.  But this would have to await purchase of a 16GB or larger USB flash drive.

Saturday, March 13, 2010

Windows XP SP3 CD Won't Boot

I was trying to install Windows XP on a Compaq Presario CQ60-420US laptop. I had just gotten it back from HP service in Texas, where they supposedly replaced the motherboard to fix a nonworking wired network interface card (NIC). They had reinstalled Vista on it, which I thought was very nice of them, and now the NIC still did not work for purposes of accessing the Internet, but did work for half of a crossover cable connection with another computer. That is, the other computer could access the laptop, but not vice versa.

So I had decided to try to see if WinXP would fare better than Vista on these problems. I wiped out the Vista partition, used GParted to create a new partition, and stuck the WinXP installation CD into the CD drive. Unfortunately, XP would not boot from the CD. GParted had just booted from the CD, so I was pretty sure it wasn’t the drive hardware per se. With the WinXP CD inserted, the computer would recognize that there was a CD in its drive, and would say, “Press any key to boot from CD,” but this just caused the thing to grind away, apparently trying to read and act upon the information on the CD, before eventually (after maybe five minutes) giving me a BSOD (blue screen of death).

I found that this problem occurred with a WinXP CD, containing slipstreamed SP3, that I had used to install XP on another computer; but it appeared somewhat different with an SP2 XP CD.   With that CD, the system went all the way through "Setup is loading files" and said, "Setup is starting Windows" before giving me the BSOD.  Then again, this may have been what was happening when the SP3 CD was grinding away; perhaps it was just not reporting all of the drivers it was loading.

To fix this, one suggestion said to disable BIOS virus detection. But the BIOS did not have a virus detection option. Someone else said laptop CDs are not eager to boot from DVDs, but this was definitely WinXP on a CD.

The BIOS Setup seemed willing to let me try to boot from a USB drive. I wondered if I could install WinXP from a 4GB USB jump drive that I had lying around. To try it, I rearranged the Boot Order so that USB Diskette on Key / USB Hard Drive came first. I wasn’t too sure how to do the rest of this process, so I started by just copying the WinXP CD directly to a 4GB USB drive. While that was underway, I researched the question and found that there were many websites on how to do this, and all of them sounded complicated.

At the same time, I continued trying to figure out what was wrong with the laptop’s DVD drive. One source said that they solved the problem by flashing the BIOS on the laptop. This seemed like a more comprehensive kind of solution, so I decided to try this second – right after inserting the USB drive with a copy of the Windows XP CD and verifying that, no, it wouldn’t install WinXP by itself; booting from that USB drive would just give me “Invalid system disk. Replace the disk, and then press any key.”

It seemed odd that the system would boot a Vista CD but would require a BIOS update in order to boot a WinXP CD.  Or at least it had booted a Vista installation CD previously.  I decided to try that again.  Sure enough, with the Vista installation CD inserted, I got “Press any key to boot from CD or DVD” and then “Windows is loading files.”  I thought about trying to slipstream Vista into a WinXP CD, but got some warnings that maybe this would not be a good idea.

So, continuing with the current plan, I went to the HP-Compaq download page and looked for a WinXP BIOS update for this laptop.  All I saw there was a firmware update named sp43474.exe, for the GSA-T50L optical drive, dated April 2009.  Firefox 3.5.8 wouldn’t download it to my other computer; I had to right-click and use the IE View extension to download successfully.  The problem now was that this update was an .exe file, which I would only be able to install from within a running Windows installation.  Catch-22!

Just to be sure, I double-clicked on the sp43474.exe file on the other computer.  It said, “This software update will upgrade the firmware of your system’s optical drive GSA-T50L to rev SC05.”  I had done a printout to PDF from a system information utility before sending the laptop to Texas, so now I searched it for a reference to this model of DVD drive.  It said that, actually, what I had (at least before sending the laptop to Texas) was an Optiarc DVD RW AD-7561S ATA Device.  This seemed to be made by Sony, whereas the GSA-T50L was made by LG.  I supposed that both might ultimately have been made by the same Chinese sweatshop somewhere, but I preferred to try to find a driver for the Optiarc.  But I was curious why my laptop would have an unofficial CD drive.  I was not able to find any references to the Sony Optiarc AD-7561S drive on the Sony Optiarc page, so apparently it was an older or in some sense unsupported model.  I stumbled across a recent thread in which someone else was tired of hassling with HP, couldn’t find a firmware update, and was at the point of just installing the wrong update to see what would happen.  Same problem in another thread.  I even found a thread on an official HP forum where they were talking about this problem.  As one participant in that last thread said, “There does not seem to be any updates for the 7561s at all.”  So, OK, updating the firmware was not an option; the more likely option was to send the machine back to HP for a replacement, assuming they were willing.  I couldn’t do that now -- I needed the laptop -- so we were back at the option of installing XP from the USB.  I went through a long effort of that nature, but ultimately abandoned it and came back to this post.

While that long effort was underway, I began looking into other possibilities.  One was to adapt an Acronis True Image backup (or you could try Macrium for freeware) of a Windows XP installation from another machine, so that it would work on this laptop's different hardware.  The Acronis option that I would need, to avoid a BSOD, seemed to be Universal Restore, which would apparently cost at least $365 (list).  Alternately, it seemed that OEMs use Sysprep to deploy the same operating system installation on many computers, but it looked like more hassle than it would be worth for just one system.  I also looked into the possibility of installing XP via network connection.  This led me to a site that specialized in boot disks.  I didn’t see how I could use any of that, unfortunately -- at least not without a substantial additional time commitment, which was impossible.

The effort to install XP from the USB did not pan out, so at this point I returned to this post.  There seemed to be two basic options.  Either find a way to develop or adapt an Acronis image from another machine so that it would work on the laptop, or get to a command prompt (via USB drive, network connection, or otherwise) and run an installation command (seemingly something like X:\i386\winnt /s:X:\i386, where the /S switch defines the location of the startup files) to install WinXP from scratch.

I wondered if (a) Windows updates were being designed and revised to defeat the Acronis approach and (b) it would be possible to bail out of a WinXP basic installation before installing many hardware drivers, and make an Acronis image at that point.  I tried that, but WinXP would crash almost immediately; and as with other efforts, it would not boot into Safe Mode either.  I figured that putting the Acronis image on the laptop would at least have installed the I386 folder there, so now maybe it was just a question of getting to a command prompt.  I didn't need BartPE on a USB stick for that; I could boot BartPE from a CD.  I had a CD version of BartPE from 2007.  I guessed it would probably work, since XP was much older than that.  But it seemed that whatever caused the laptop to crash when I had the WinXP CD inserted was also going to reject WinXP as part of a BartPE CD.

I also had an Ultimate Boot CD for Windows (UBCD4Win).  The UBCD ground away for a long time, eventually showed me the WinXP splash screen -- and didn't crash!  Instead, it gave me a choice, "Select shell to start." It still seemed to be in its process, so I waited, and it chose the automatic option for me.  It gave me an option of setting up networking, but I didn't know how to answer the question as to which sort.  I tried to cancel, but it seemed intent upon continuing, so I just used the default options.  I went to Start > Command Prompt and typed DIR C: (using capital letters here just to indicate the actual command; DOS is not case-specific) and, no, the I386 folder was actually not installed there yet.  I put the WinXP CD in the CD drive and typed DIR X: because it looked like X was the letter that UBCD had assigned to the CD drive.  I typed COPY X:\I386 C:\ and it copied files.  Then it froze.  I killed that window and tried running Start > Programs > File Management > Explorers.  None of the explorer programs would run.  Eventually I realized this was because I had removed the UBCD disc.  I put it back in and tried again.  The explorers still didn't work.  I had belatedly realized that COPY was probably not the best command, so I tried XCOPY X:\I386 C:\I386 /E /H /Y.  It copied lots of stuff, and not the way I intended.  Now all the stuff that should have been in the C:\I386 folder was instead in the root (C:\) folder.  Oops.  I wound up deleting it all and starting over.  There were just a few files in C:\ that I couldn't delete (i.e., biosinfo.inf, bootfix.bin, config.sys, and mstask.inf), and of course several folders (i.e., Documents and Settings, I386, Program Files, and Windows).  Problem:  there was no WINNT.EXE file to be found.  On my other computer, it was in C:\I386, but it was not in that folder on either the laptop or the WinXP CD.  I carried it over via jump drive.  It took UBCD a minute to recognize the jump drive, but it did.  I typed WINNT.EXE and hit Enter.  I got this:
Windows XP Setup
This program does not run on any 32-bit version of Windows.
Use WINNT32.EXE instead.
Setup cannot continue.  Press ENTER to exit.
So, OK, I jumped WINNT32*.* over to C:\I386.  (The USB flash drive was recognized as drive G, as I discovered by feeling around.  That is, I tried D: and got a partition; tried E: and got a partition; tried F: and got an error; tried G: anyway and there it was.)  But it froze.  It would copy the WINNT32.* files, but not the WINNT32*.DLL files, not even one at a time.  I tried running C:\I386\WINNT32.EXE anyway.  It gave me an error:
The file C:\I386\WINNT32U.DLL could not be loaded or is corrupt.  Setup cannot continue.
I didn't have that particular one on the jump drive, so I went back to the source machine and tried to jump it over.  But I got the same problem:  the command window froze and had to be killed.  I rebooted the computer with an Ubuntu 9.10 CD and then copied all of the WINNT32*.* files (and a WINNT32 folder) from the source computer to the laptop's C:\I386 via jump drive.  Back in UBCD, I tried running that same WINNT32 command and got that same "could not be loaded or is corrupt" error.  One post suggested this sort of message might be due to insufficient RAM.  A Microsoft webpage said that I should double-click on WINNT32.MSI.  I wasn't in a GUI, so I just tried typing WINNT32.MSI at the command line.  This opened a dialog asking me what program I wanted to use to open this kind of file.

Since I had a command prompt using UBCD, I tried a different approach.  The command I used this time was:
X:\i386\winnt32.exe /syspart:C: /tempdrive:C: /makelocalsource
But this gave me an error, stating that this command was not recognized.  I tried it again, inserting a BartPE jump drive and replacing X: with H: since H was where the jump drive was.  This worked, but it just gave me the same error message I had gotten a long time earlier:
Setup cannot continue because upgrade functionality is disabled and your copy of Windows XP only allows upgrades.
I tried replacing H with C, since I had already copied I386 to C.  This just gave me the message about WINNT32U.DLL being unloadable or corrupt.  One post led me to think that this problem might be unique to my WinXP SP3 disc, so I loaded an SP2 disc and tried winnt32.exe with that.  That did not seem to achieve anything:  the disc spun up and then down and I was back at the command prompt.  I tried it again with the SP3 CD.  Same result.  Apparently I had mistyped the command previously.

I thought maybe I could run winnt if I could find a 16-bit DOS equivalent.  My search led to FreeDOS, where I downloaded FDFullCD.iso.  I burned it and booted the laptop with it.  I indicated that I wanted to boot FreeDOS, and then chose option 3, FreeDOS Live CD with HIMEM + EMM386.  But I got this error:
Selected page frame e000 not available, searching automatically
No suitable page frame found.  EMS functions limited.
I gathered that EMM386 was not a good option in this case.  I rebooted and tried again, this time choosing option 4, HIMEM only.  This produced a bunch of error messages, but at least it put me to an A: prompt.  Drive B appeared to be a RAM drive.  The program did not recognize drive C.  I thought this must be because it was an NTFS drive, but no, FreeDOS did not recognize any drive, including the CD drive.  I booted FreeDOS again, and this time chose option 5, FreeDOS Live CD only.  I still got some error messages, including "There is no CDROM, or the wrong CD-ROM!"

I guessed that whatever kept this CD-ROM drive from recognizing WinXP was also impairing its recognition of the FreeDOS CD somewhat.     In that case, I wondered if I could install the contents of the WinXP CD from a bootable FreeDOS USB drive.  To make that, I followed instructions to install Unetbootin on my Ubuntu machine, and then used Unetbootin to make a FreeDOS bootable USB stick.  When that was done, I checked its properties and saw that Ubuntu indicated the filesystem type was msdos.  I copied the /I386 folder from the slipstreamed WinXP SP3 CD to the USB stick, put the USB into the laptop, and rebooted.  After a Unetbootin screen, I was back at the FreeDOS menu.  I tried option 3; same errors as before.  I tried option 5.  This time, it recognized the USB drive as drive C.  Then I realized that this was no help; it still couldn't see the partition on the hard drive where I wanted to install WinXP.  Well, could I reformat that partition as FAT32?  I formatted another USB drive as FAT32, put it into the laptop, and rebooted from the FreeDOS stick.  It did recognize the FAT32 USB drive.  Now, could I install WinXP on a FAT32 partition?  It looked like it was supposed to be possible.  I pulled out the USB drives, rebooted the laptop with GParted, and replaced the NTFS partition where I had planned to install WinXP with a FAT32 partition.  (I figured this would be much faster than trying to convert from NTFS to FAT32.)  Of course, now I was wondering whether the FAT32 alternative would also have made a crucial difference with some of the attempts to install from UBCD.  When GParted was done, I rebooted with the FreeDOS stick.  Sure enough, now I could see the FreeDOS stick as drive C and the WINXP partition as drive D.  I tried using XCOPY, but it didn't work.  I wasn't sure how to copy subfolders etc. in FreeDOS.  Rather than research it, I removed the FreeDOS USB, rebooted with UBCD, plugged in the FreeDOS stick, found that WINXP was now drive C and the FreeDOS stick was drive G, and therefore used XCOPY G:\I386\*.* C:\I386 /E /H /Y to copy the I386 folder from the FreeDOS to WINXP.  This time around, that ran very smoothly.  I rebooted with the FreeDOS USB stick.  I went into the I386 folder on the WINXP partition (which was drive D in this case), and typed WINNT.  Windows XP Setup asked me where the XP files were located.  It defaulted to D:\I386, so I left it at that.  Next, it said this:
Windows XP Professional Setup
Setup did not detect SmartDrive on your computer.  SmartDrive will greatly improve the performance of this phase of Windows Setup.
You should exit now, start SmartDrive, and then restart Setup.  See your DOS documentation for details about SmartDrive.
I looked into SmartDrive.  Along the way, I ran across the ingenious suggestion that I could use an IDE to USB adapter and, with that, could use the internal CD drive on a desktop computer as an external drive for some other computer (if I had needed to do that).  (This would require leaving the CD drive's power connected in the host computer.)  SmartDrive itself was an MS-DOS disk cache program, to speed up file transfer times.  It appeared that FreeDOS might have its own cache arrangement, presumably faster if I could get HIMEM and/or EMM386 to load.  The best I could do was to reboot, choose FreeDOS option 3, ignore the FreeDOS boot error messages, and continue past the WinXP setup message about SmartDrive.  WinXP setup next said, "Please wait while Setup copies files to your hard disk."  It copied files for a while, and then froze.  I had heard something about how WINNT would only start the process, so I rebooted with UBCD, went into C:\I386, and typed "winnt32.exe /syspart:C: /tempdrive:C: /makelocalsource" and, this time, it worked!  It gave me "Welcome to Windows Setup" and then asked for my WinXP's product key.  I went through the various options, accepting the defaults (including the offer to upgrade my drive to NTFS), and it began the installation process.  It estimated that it would take 52 minutes, instead of the usual 39.  But then something happened.  I turned away, and when I looked back, we were back at the UBCD prompt.  I ran WINNT32 again, just like before, but this time I said No to the option of upgrading to NTFS.  It didn't matter.  I ran it a third time, and again it crashed, shortly after it finished "Copying Installation Files."  No error message or anything.  I rebooted UBCD and tried installing again from the I386 folder that I had added to the bootable FreeDOS USB drive, using the command "G:\i386\winnt32 /syspart:C: /tempdrive:C: /makelocalsource" (because G was where the system saw the FreeDOS USB drive).

So now I wondered whether I could boot to a very WinXP-compatible 32-bit command line.  I started with Vista.  My Vista installation was not booting at this point, probably because I had told GParted to hide the Vista partition, so I inserted the Vista DVD and rebooted.  I went with its Repair Your Computer option but didn't proceed with the repair, choosing instead the steps necessary to get a command prompt.  I inserted the FreeDOS USB and found it at drive H.  But before trying it, I went to C:\I386 and ran that WINNT32 command again.  It proceeded just as it had done in UBCD, except I didn't recall having to agree to the license agreement in the UBCD process.  It crashed -- it put me back to the command line -- just as it had done in UBCD.  I tried again, this time starting from the I386 folder in the FreeDOS USB.  I even swapped out the Vista CD for the WinXP CD, just in case that would make any difference.  It didn't.  Another crash.  I took out all bootable items, rebooted, and got "Disk error.  Press any key to restart."  So, no, it had not by some magic completed the process.

I rebooted UBCD.  On another computer, I copied C:\Windows\system32\chkdsk.exe to a new Utilities folder on the FreeDOS USB.  I plugged in the FreeDOS USB, navigated to C:, and typed D:\Utilities\chkdsk /r.  It ran.  I told it to force a dismount.  It quickly checked files and slowly checked free space.  It found no problems.  I rebooted with GParted and had it check the filesystem on the WINXP partition.  No problem there either.  I wondered if the presence of even a hidden Vista partition was screwing things up for WinXP.  I deleted both partitions and recreated WINXP.  The WinXP CD would still not boot.  The Vista CD would still boot, and it gave me a command line from which I could run winnt32.  Even so, it crashed again.  I wondered whether the problem was with my SP3 CD.  I made another folder on the FreeDOS USB, naming it I386-SP2, and copied the I386 folder to it from the WinXP SP2 CD that I had used in making the SP3 CD (i.e., having the same product key).  I tried running WINNT32 again from the Vista command prompt, using this I386-SP2 folder as the source.  This gave me an error:
Setup was unable to build the list of files to be copied.
The system cannot find the path specified.
I verified that the WINXP partition was still drive C, and that it contained an I386 folder.  The problem seemed to be that the I386 folder has to be named I386 -- nothing more, nothing less.  I renamed I386-SP2 to be I386 and re-ran the WINNT32 command.  It crashed again, in exactly the same way as before.  I checked C: to see what WINNT32 was actually doing there.  The answer: not much.  It was creating files named $WIN_NT$, with the extensions .~BT and .~LS, along with a file named textsetup.sif, and it had copied NTLDR over.  And that was about it.  I deleted those $WIN_NT$ files and tried WINNT32 again, this time using the product code from my other WinXP CD.  It crashed, of course.

Someone said that a problem something like this one sounded like it might be due to a bad CPU or other hardware.  Of course, that was always possible.  I wasn't ready for that yet; after all, everything else seemed to work OK.  But I was running out of other theories.  Then I stumbled into a search from which I gathered that "downgrade" was an official searchable term, especially where Vista was concerned.  PC Magazine had a two method article, but neither method would work for me since I wasn't able to boot from the XP CD.  I found what looked like the consummate downgrading guide, with specific information for my laptop model.  For a CQ60 with an Intel CPU, it seemed to say that I should slipstream the ICH9 SATA driver (ICH9M-E/M SATA AHCI Controller) into the XP installer using nLite.  To get the ICH9, I downloaded and unzipped f6flpy3289.zip, but I did not understand why the highly regarded CherylG was telling me that I would find my driver inside iaAHCI.inf.  In the process of digging around for the driver she named, I contracted a virus, which prompted me to get up-to-date on Spybot Search and Destroy and on the Web of Trust add-on for Firefox.  I eventually found the driver on an HP webpage (though not the downloads webpage for my laptop).  I ran it on another computer (it was an .exe file) and it tried to install itself but then said the computer didn't meet the requirements, which I assume meant something about its particular CPU.  The conclusion there seemed to be that it would help to run it on a computer whose hardware was very different from the target computer, so that it would just extract but would not install.  I didn't know if it would run on the laptop from the Vista prompt, but anyway this was not the concept of slipstreaming, so I went back to the first one that CherylG had pointed toward.  Then I downloaded nLite and looked at the recommended guide for slipstreaming the driver.  Now that I saw the guide, I understood why CherylG was pointing me toward an .inf file.  She apparently didn't mean that I somehow had to extract the driver from iaAHCI.inf; I just had to include that .inf file in the slipstream.  This seemed to be the only thing I had to slipstream, so I went ahead with it.

The slipstreaming process began by copying the entire SP3 CD to a folder on the hard drive that I called SP3CD.  I started to install nLite, which required the Microsoft .NET Framework 2.0.  Installing that required trying to update .NET Framework 1.1 and then downloading a tool to uninstall it when the updates failed, and then manually doing registry edits etc. to uninstall it when the tool failed.  That ran overnight.  Next morning, when that was done, I followed the MaxEasyGuide to slipstreaming with nLite.  The first time, for some reason, it did nothing.  Second time, it made a file that I called SP3-SATA.ISO, containing the ICH9M-E/M SATA AHCI Controller driver from within iaAHCI.inf -- and now I understood what CherylG was saying.  Next question:  what was I supposed to do with this thing?  I decided to open it with UltraISO and copy its I386 folder to another folder.  Then I ran Beyond Compare to see how different it was from the original slipstreamed SP3 I386 folder.  It looked like there were a couple dozen differences in files.  So, OK, the nLite process seemed to have done something.  I wasn't sure I had done all of these steps right, but I decided to give it a shot.  I used Beyond Compare to synchronize the I386 folder on the FreeDOS USB with the I386 folder I had just extracted from SP3-SATA.ISO.  Whoa.  Many, many differences.  I forgot -- I had been using the SP2 I386 folder.  I changed that around and re-ran the comparison.  Still tons of differences!  Oh, I knew why.  It was because of the timechange.  The files were mostly the same, but some had been created an hour before the others of similar name.  I told Beyond Compare to ignore unimportant differences and did a full refresh, but that made no difference.  I bit the bullet and upgraded 6,992 files on the FreeDOS USB.  While that was underway, I re-ran the nLite and Beyond Compare processes, just to be sure.  Everything looked good.  I put the updated FreeDOS USB drive into the laptop and re-ran the WINNT32 command.  Alas, that wasn't the answer.

I found another thread that seemed to say that CherylG's advice on drivers was for an AMD CPU, whereas my CQ60-420US had an Intel CPU.  But some respondents said they were definitely using the Intel drivers and it still wasn't working.  Unfortunately, the drivers they pointed to were no longer availlable at the Intel website.  Daniel Potyrala advised that I download and run CPU-Z in order to find out not only my CPU type but also my chipset.  I couldn't very well run CPU-Z without an operating system.  Since it seemed that having Vista on the same hard drive did not explain why I could not even run WINNT32, it seemed to be time to restore the Vista partition using GParted and Acronis.  Adam Pash assured me that it didn't matter which came first, WinXP or Vista, so I left the WinXP partition in place (in hopes that it would someday be operational) and just added the Vista partition behind it.  I downloaded the appropriate version of CPU-Z, jumped it over to the laptop, and installed and ran it.  CPU-Z said that my mainboard chipset was an Intel GL40, which meant that I needed to slipstream . . .  the ICH9M-E/M SATA AHCI Controller driver, just like before.  So Mr. Potyrala was wrong; CherylG had given me the right tip.  But then Fernando1 said there were different drivers for 32-bit and 64-bit CPUs.  My T4200 Pentium (Penryn) was a 64-bit.  It was hard to tell the two apart; the driver names were the same.  I repeated the nLite process.  This time, the Textmode driver I selected was the ICH9 SATA AHCI Controller (Desktop ICH9R).  There were, again, some differences, so once again I updated the FreeDOS USB drive and tried it in the laptop, this time typing  "I:\i386\winnt32 /syspart:F: /tempdrive:F: /makelocalsource" because the FreeDOS USB was at I: and the WinXP partition was now at F.  That didn't work, so I tried once more, this time selecting the ICH8R/ICH9R SATA RAID Controller.  That didn't work either.

It was time to quit.  The webpages to remember, for future reference, included Quang Do's post in an HP forum, listing drivers and their sources; ditto Riskyone101; and perhaps a post, by wimb, that I really didn't understand.  The approach to remember was to use Unetbootin in Ubuntu to quickly create a FreeDOS bootable USB drive, if I needed 16-bit support, or just a Vista DVD or UBCD, if I could go straight to 32-bit support; to format the target drive in FAT32 rather than NTFS if necessary; to use nLite to create a slipstreamed ISO, assuming I could find the right drivers, and UltraISO to extract files from that ISO to load via USB drive if necessary.  I didn't know why WinXP could not run on the laptop, and I wasn't sure what else to try at this point.

Friday, January 9, 2009

Ubuntu DVD Burning Problem

I was trying to burn a DVD in Ubuntu Hardy Heron 8.04. Using Brasero, I got this message:

Burning error Error while burning: An unknown error occured. Check your disc.
Or, for those who are searching for phrases, the spelling should be: "An unknown error occurred." I found nothing when searching for the correctly spelled phrase, but the incorrect one brought up a few Google hits. Before digging into them, I clicked on the "View log" button. There, I saw what appeared to be two error messages:
BraseroGrowisofs stderr: File [file path and name] is larger than 4GiB-1. BraseroGrowisofs stderr: -allow-limited-size was not specified. There is no way do represent this file size. Aborting.
Again, presumably the spelling was supposed to be, "There is no way to represent this file size." Apparently the problem was that I was trying to burn a too-large file. It was perplexing that it was too large, because 7-Zip had created it as one in a series, supposedly in the right size for DVD burning. I killed Brasero and tried again to burn the DVD, this time using CD/DVD Creator (the default Ubuntu file burner, which seemed to be the same as the Nautilus program that some people referred to. This time, I got
Error writing to disc There was an error writing to the disc: Unhandled error, aborting
There was no option of reading an error log. When I closed that dialog, I got "An error occurred while writing." The Google search gave me a couple dozen hits this time, but in refined form it really only gave me one or two. The first discussion that came up offered some ideas: (1) they (in 2005) recommended using KB3 as my disc-burning program; (2) they said that ISO filesystems cannot handle files larger than 2GB, and beyond that I would need to use the UTF file system; and (3) they said you get better results if you put your files into an ISO disc image first, and then burn the DVD from the disc image. I started with option (3), using Brasero to create an ISO image. It didn't appear to give me a UTF option, and in the ISO option it gave me the same "no way do represent this file size" error, plus another "HUP" error. Turning to option (1), one source recommended tinkering with the large ZIP file using PgcEdit, which I had no interest in doing. That source also said it might be a problem unique to Hardy 8.04. Another source said KB3 wasn't the solution, but KB3 looked like it had a better user interface, so I decided to install it and take a look, just in case they had somehow upgraded it to handle the problem. Using Ubuntu's System > Administration > Synaptic Package Manager, I searched for KB3 but didn't find it. From its name, I guessed it might be a KDE package, which wasn't the default with Hardy and (to my knowledge) wasn't installed on my computer. I decided to try a different approach. I was using VMware Workstation as a virtualization tool in Ubuntu, which meant that I was running copies of Windows XP in several virtual machines (VMs) on Ubuntu. So I went into VMware, opened one of those VMs, and tried starting a CD burning program. It didn't seem to be recognizing the DVD drive from inside the VM, but eventually I found that a version of Nero would at least burn the large ZIP file to an ISO file, named "CURRENT 01," using its Image Recorder. What I actually got, though, was three ISO files: CURRENT 01.iso, CURRENT 01.iso.001, and CURRENT 01.iso.002. Then, back in Brasero, I selected the option that said, "Burn image: Burn an existing CD/DVD image to disc." I pointed it to the folder where I had put those three ISO files. It would only let me indicate one of them, so I indicated CURRENT 01.iso. It burned the DVD, but ended with this:
Burning error Error while burning: some files may be corrupted on the disc.
The log had a number of messages of this type:
BraseroReadom stderr: addr: 7808 cnt: 64
I closed the error dialog and looked at the DVD in Ubuntu's File Browser (a/k/a Nautilus). It did report that it had burned the entire 4.4GB file to DVD, so apparently selecting CURRENT 01.iso to be burned was the right move. The file I had burned was named Reference.7z.001. It had the .001 extension because I had told 7-Zip to save the Reference folder (containing a bunch of materials I was keeping for reference purposes) in 4.4GB chunks. So this was just the first of several that I would then have to copy back to the hard drive and restore via 7-Zip in order to recover my complete Reference folder. Using Synaptic, I searched and saw that I had already installed 7-Zip in Ubuntu (I had done the zipping using 7-Zip in a Windows virtual machine), under the name of p7zip-full. Using File Browser, I searched File System for p7zip and found it in /usr/lib/p7zip. But it didn't appear I could run it directly as a standalone program, or at least it didn't respond when I double-clicked on it or right-clicked and selected Open. I copied the 4.4GB file back to an NTFS drive to test it, perhaps by using 7-Zip in Windows rather than in Ubuntu, but I kept getting "Error while copying 'Reference.7z.001'." The resulting file, back on the NTFS drive, was substantially smaller -- only about 1GB. So apparently the burn had failed. I tried again. Another failure; pretty much the same problems as before. This approach was not working. I went back into the WinXP VM, and this time I used Nero to create an .NRG Nero image file instead of an ISO. It did the same thing -- created three files, none larger than 2GB. Then I rebooted the computer into native Windows (i.e., not just a virtual machine) and tried burning a DVD from the ISO, using Nero there too. Nero reported that the burn was successful, so it seemed the NRG version would not be needed. I copied Reference.7z.001 from the DVD back to a separate folder on the hard drive and used 7-Zip in WinXP to extract its files. But it wouldn't work right; apparently I needed all of the Reference.7z files (i.e., Reference.7z.002, etc.) before I could check this thing. So, OK, I burned another DVD (there was only one more Reference file, i.e., Reference.7z.002) and then copied its contents back to the hard drive and tried again to test the results by running 7-Zip to unzip the complete contents of the Reference folder. Since there was space on the second DVD, I also burned a couple of other 7-Zip files onto it and copied and unzipped them on the hard drive too. They all worked fine; all my files were back. So the problem was indeed with the Ubuntu DVD-burning software. The hard drive and DVD drive hardware were working fine; 7-Zip was fine; Nero was fine. I just couldn't burn a DVD in Ubuntu if any of the files were larger than 2GB.