Ubuntu 10.04 vs. Windows XP: Can't Delete a File

Many Windows users have encountered files that they cannot delete.  I had been able to handle most of those problems pretty easily.  The tools I used were Unlocker, to delete the problem file from within Windows conveniently, and Ubuntu Linux, when Unlocker failed.

I was able to switch to Ubuntu pretty quickly, if necessary.  I was using Ubuntu 10.04 on a dual-boot system.  I had installed VMware Workstation 7.1 in Ubuntu, and had created a virtual machine (VM) running Windows XP SP3 in Workstation.

But in this case, I wasn't having file deletion problems in Windows; I was having file deletion problems in Ubuntu.  Specifically, I was running into difficulties when trying to use an external USB hard drive.  For instance, when I tried to delete a file from the external hard drive, I first got this message:

Cannot move file to trash, do you want to delete immediately?

No problem there, to my knowledge; it was a 2.2GB .avi (i.e., video) file, so maybe it was too big for the trash.  So I clicked Delete.  This gave me "Error while deleting," and when I clicked on "Show more details," I got this:  "Error removing file:  Input/output error."  Similarly, when I tried to move some .avi files manually from the internal hard drive to the external USB drive, I got "Error while copying" and, in the details, "Error opening file [filename]:  Input/output error."

Through a relevant search, I came across a post that made me think I should try to approach this from the Windows side.  That seemed ironic at first, but then I reflected that the target drive was NTFS.  That is, it was formatted for use by Windows.  I wasn't having this problem with my Ubuntu ext3-formatted drives.

So I went into Windows and tried to do the same maneuvers I had just tried in Ubuntu:  connect to the USB drive; delete one file from the external (NTFS) USB drive; move files from an internal (ext3) drive to that external USB drive.  I didn't get too far.  Windows said the folder on the USB drive "is not accessible.  The file or directory is corrupted and unreadable."  Was that what "input/output error" in Ubuntu meant?

I ran Disk Management (Start > Run > diskmgmt.msc), right-clicked on the USB drive > Properties > Tools > Error-checking > click both boxes > Start.  I clicked on out of there and rebooted so that the error check would run.  But this did not repair the drive.  On a second try, I watched more closely.  The VM did reboot, but it did not run a disk check.  Apparently the problem was that the external USB drive did not become connected to the VM until after the operating system had already restarted, by which time it was too late for a disk check.  So I rebooted the VM again, this time with a WinXP installation CD in the CD/DVD drive, and ran CHKDSK /R from the Recovery Console.

CHKDSK ran for a while and then said, "The volume appears to contain one or more unrecoverable problems."  This particular external drive consisted of a hard drive in an external enclosure.  I took the drive out of the enclosure and put it directly in the computer.  This time, CHKDSK ran successfully.  I repeated CHKDSK until it reported no more errors, and then put the drive back in its enclosure.  Now I was able to delete the file.  Problem solved!

Ubuntu 9.04: Backing Up and Copying Webpages and Websites

As described in a previous post, I had been using rsync to make backups of various files.  This strategy was not working so well in the case of webpages and websites, or at least I wasn't finding much guidance that I could understand.  (Incidentally, I had also tried the Windows program HTTrack Website Copier, but had found it to be complicated and frustrating.  It seemed to want either to download the entire Internet or nothing at all.)

The immediate need driving this investigation was that I wanted to know how to back up a blog.  I used the blog on which I am posting this note as my test bed.

Eventually, I discovered that maybe what I needed to use was wget, not rsync.  The wget manual seemed thorough if a bit longwinded and complex, so I tried the Wikipedia entry.  That, and another source, gave me the parts of the command I used first:

wget -r -l1 -np -A.html -N -w5 http://raywoodcockslatest.blogspot.com/search?max-results=1000 --directory-prefix=/media/Partition1/BlogBackup1

The parts of this wget command have the following meanings:

• -r means that wget should recurse, i.e., it should go through the starting folder and all folders beneath it (e.g., www.website.com/topfolder and also www.website.com/topfolder/sub1 and sub2 and sub3 . . .)
  
• -l1 (that's an L-one) means stay at level number one.  That is, don't download linked pages.
• -np means "no parent" (i.e., stay at this level or below; don't go up to the parent directory)
• -A.html means Accept only files with this extension (i.e., only .html files)
• -N is short for Newer (i.e., only download files that are newer than what's already been downloaded).  In other words, it turns on timestamping
• -w5 means wait five seconds between files.  This is because large downloads can overload the servers you are downloading from, in which case an irritated administrator may penalize you
  
• The URL shown in this command is the URL of this very blog, plus the additional information needed to download all of my posts in one html file.  But it didn't work that way.  What I got, with this command, was each of the posts as a separate html file, which is what I preferred anyway
• --directory-prefix indicates where I want to put the download.  If you don't use this option, everything will go into the folder where wget is running from.  I came across a couple of suggestions on what to do if your path has spaces in it, but I hadn't gotten that far yet
  

Incidentally, I also ran across another possibility that I didn't intend to use now, but that seemed potentially useful for the future.  Someone asked if there was a way to save each file with a unique name, so that every time  you run the wget script, you get the current state of the webpage.  One answer involved using mktemp.  Also, it seemed potentially useful to know that I could download all of the .jpg files from a webpage by using something like this:  wget -e robots=off -r -l1 --no-parent -A.jpg http://www.server.com/dir/

The first download was pretty good, but I had learned some more things in the meantime, and had some questions, so I decided to try again.  Here's the script I used for my second try:

wget -A.html --level=1 -N -np -p -r -w5 http://raywoodcockslatest.blogspot.com --directory-prefix=/media/Partition1/BlogBackup2

This time, I arranged the options (or at least the short ones) in alphabetical order.  The -p option indicated that images and style sheets would be downloaded too.  I wasn't sure I needed this -- the basic html pages looked pretty good in my download as they were -- but I thought it might be interesting to see how much larger that kind of download would be.  I used a shorter version of the source URL and I designated a different output directory.

I could have added -k (long form:  --convert-links) so that the links among the downloaded html pages would be modified to refer to the other downloaded pages, not to the webpage where I had downloaded them from; but then I decided that the purpose of the download was to give me a backup, not a local copy with full functionality; that is, I wanted the links to work properly when posted as webpages online, not necessarily when backed up on my hard drive.  I used the long form for the "level" option, just to make things clearer.  Likewise, with a bit of learning, I decided against using the -erobots=off option.  There were probably a million other options I could have considered, in the long description of wget in the official manual, but these were the ones that others seemed to mention most.

The results of this second try were mixed.  For one thing, I was getting a lot of messages of this form:

2010-01-01 01:43:03 (137 KB/s) - `/[target directory]/index.html?widgetType=BlogArchive&widgetId=BlogArchive1&action=toggle&dir=open&toggle=MONTHLY-1196485200000&toggleopen=MONTHLY-1259643600000' saved [70188]

Removing /[target directory]/index.html?widgetType=BlogArchive&widgetId=BlogArchive1&action=toggle&dir=open&toggle=MONTHLY-1196485200000&toggleopen=MONTHLY-1259643600000 since it should be rejected.

I didn't know what this meant, or why I hadn't gotten these kinds of messages when I ran the first version of the command (above).  It didn't seem likely that the mere rearrangement of options on the wget command line would be responsible.  To find out, I put it out of its misery (i.e., I ran "pkill wget" in a separate Terminal session) and took a closer look.

Things got a little confused at this point.  Blame it on the late hour.  I thought, for a moment, that I had found the answer.  A quick glance at the first forum that came up in response to my search led me to recognize that, of course, my command was contradictory:  it told wget to download style sheets (-p), but it also said that only html files would be accepted (-A.html).  But then, unless I muddled it somehow, it appeared that, in fact, I had not included the -p option after all.  I tried re-running version 2 of the command (above), this time definitely excluding the -p option.  And no, that wasn't it; I still got those same funky messages (above) about removing index.html.  So the -p option was not the culprit.

I tried again.  This time, I reverted to using exactly the command I had used in the first try (above), changing only the output directory.  Oh, and somewhere in this process, I shortened the target URL.  This gave me none of those funky messages.  So it seemed that the order of options on the command line did matter, and that the order used in the first version (above) was superior to that in the second version.  To sum up, then, the command that worked best for me, for purposes of backing up my Blogger.com (blogspot) blog, was this:

wget -r -l1 -np -A.html -N -w5 http://raywoodcockslatest.blogspot.com --directory-prefix=/media/Partition1/BlogBackup1

Since there are other blog hosts out there, I wanted to see if exactly the same approach would work elsewhere.  I also had a WordPress blog.  I tried the first version of the wget command (above), changing only the source URL and target folder, as follows:

wget -r -l1 -np -A.html -N -w5 http://raywoodcock.wordpress.com/ --directory-prefix=/media/Partition1/WordPressBackup

This did not work too well.  The script repeatedly produced messages saying "Last-modified header missing -- time-stamps turned off," so then wget would download the page again.  As far as I could tell from the pages I examined in a search, there was no way around this; apparently WordPress did not maintain time stamps.

The other problem was that it did not download all of the pages.  It would download only one index.html file for each month.  That index.html file would contain an actual post, which was good, but what about all the other posts from that month?  I modified the command to specify the year and month (e.g., http://raywoodcock.wordpress.com/2009/03/).  This worked.  Now the index.html file at the top of the subtree (e.g., http://raywoodcock.wordpress.com/2009/03/index.html) would display all of the posts from that month, and beneath it (in e.g., .../2009/03/01) I had named subfolders for each post, each of which contained the index.html file displaying that particular post.  So at this rate, I would have to write wget lines for each month in which I had posted blog entries.  But then I found that removing the -A.html option solved the problem.  But if I ran it at the year level, it worked only for some months, and skipped others.  I tried what appeared to be the suggestion of running it twice at the year level (i.e., at .../wordpress.com/ with an ending slash), with --save-cookies=cookies.txt --load-cookies=cookies.txt --keep-session-cookies.  That didn't seem to make a difference.  So the best I could do with a WordPress blog, at this point, was to enter separate wget commands for each month, like this:

wget -r -l1 -np -N -A.html -w5 http://raywoodcock.wordpress.com/2009/01 --directory-prefix=/media/Partition1/WordPressBackup

I added back the -A.html option, as shown, because it didn't seem to hurt anything; html pages were the only ones that had been downloaded anyway.

Since these monthly commands would re-download everything, I would run the older ones only occasionally, to pick up the infrequent revision of an older post.  I created a bunch of these, for the past and also for some months into the future.  I put the historical ones in a script called backup-hist.sh, which I planned to run only occasionally, and I put the current and future ones into my backup-day.sh, to run daily.

But, ah, not so fast.  When I tried this on another, unrelated WordPress blog, it did not consistently download all posts for each month.  I also noticed that it duplicated some posts, in the sense that the higher-level (e.g., month-level) index.html file seemed to contain everything that would appear on the month-level webpage on WordPress.  So, for example, if you had your WordPress blog set up to show a maximum of three posts per page, this higher-level webpage would show all three of those.  The pages looked good; it was just that I was not sure how I would use this mix in an effective backup-and-restore operation.  This raised the question for my own blog:  if I ever did have to restore my blog, was I going to examine the HTML for each webpage manually, to re-post only those portions of text and code that belonged on a given blog page?

I decided to combine approaches.  First, since it was yearend, I made a special-case backup of all posts in each blog.  I did this by setting the blogs to display 999 posts on one page, and then printed that page as a yearend backup PDF.  Second, I noticed that rerunning these scripts seemed to catch additional posts on the subsquent passes.  So instead of separating the current and historical posts, I decided to stay with the original idea of running one command to download each WordPress post.  I would hope that this got most of them, and for any that fell through the crack, I would refer to the most recent PDF-style copy of the posts.  The command I decided to use for this purpose was of this form:

wget -r -l1 -np -N -A.html -w5 [URL] --directory-prefix=/media/Backups/Blogs/WordPress

I had recently started one other blog.  This one was on Livejournal.com.  I tried the following command with that:

wget -r -l1 -np -N -A.html -w5 http://rwclippings.livejournal.com/ --directory-prefix=/media/Backups/Blogs//LiveJournal

This was as far as I was able to get into this process at this point.

The Ultimate Intellectual Piracy

The United States may soon cease to be the world’s greatest threat to global peace.

That statement may come as news to those who were not aware that the U.S. ever was a great threat to global peace. And surely, by many measures, it is not. How many countries have a Peace Corps? How many provide real estate for a United Nations?

Let us not quibble about the U.S.’s standing as the world’s leading seller of weaponry. Let us not debate the invasion of Iraq, or the devastation and brutality of the Vietnam War (or, indeed, the murder rates in our own cities).

In fact, let us withdraw that statement about the United States altogether. Because soon, it may not matter anyway.

The U.S. has had its Ugly Americans abroad, as well as its piggish Yuppies cluelessly asking themselves, “Why do they hate us?” in the wake of 9/11. But at least some such Yuppies did ask, and some actually seemed to want to hear an answer. That, however, may not be the way of the future.

Surely there will always be a liberal, educated fringe of Chinese individuals who have not only visited the West but who have also come to appreciate the good things that the West’s liberal, educated fringe try to achieve. Such individuals will undoubtedly be grossly outnumbered, though, by those Chinese people who, in good middle-class American style, neither know nor care, very much, what the rest of the world may think or believe.

Most Chinese may be like most Americans – concerned, that is, with what’s on the barbecue or in the fridge, and not so concerned with what someone with a cause, somewhere else in the world, seems to be complaining about.

China has a reputation, these days, of copying what other people invent. There is no law of nature that limits such copying to the good things. Those who plunder goods from others’ homes may inadvertently haul away some dust and cockroaches as well.

In particular, China may be copying some flaws from America of the 1960s. Chinese responses to Tibet certainly make it seem that way.

On Tibet, as in Vietnam, the world speaks out, in the name of fairness and humanity. The Dalai Lama, like a latter-day Ho Chi Minh, actually talks as if he believed that the leaders of the superpower were reasonable people. But those leaders have their eyes on domestic opinion, and domestic opinion is clear enough.

It took a long, long time for Americans to make up their minds, take action, and ultimately end the Vietnam War. Even in a fairly open democracy, with many fictions exposed by a relatively free press, it took years on end for society to get sick of its own anti-communist rhetoric.

Americans of that era knew what they believed. They knew it because it was what someone had told them, and it was also what their friends seemed to believe. We cannot expect anything different from the Chinese, and we are not getting anything different. The effort to talk sense to Chinese people – even educated, westernized ones – about Tibet, these days, seems much like the effort to talk sense to Americans in 1968 (or, actually, in 2003).

Tibet is not the point. Tibet is merely the illustration. If the Chinese people are presently able to support internal or localized nationalist hype á la the Alamo or Cuba, in the future they may also be able to support nationalist hype focused abroad, á la Saigon or Baghdad.

Things are changing very quickly, these days, in the U.S.-China balance. It is easy to notice the shifts in the balances of finance, military power, and influence. But other things are shifting as well.

Sooner than we expect, people of the whole world may begin to encounter the Ugly Chinese. If such a thing happens, it will not be because Chinese individuals are interpersonally ugly. Much to the contrary, they hail from a culture that seems to foster deference and agreeableness. It will happen, not because of who the Chinese people are, but despite that.

Power tends to corrupt, and the Chinese people are gaining power. There are things they want and, as shown in Tibet, there are things they will take – not because they are right, educated, or caring, but simply because they will be increasingly able to follow their beliefs and feed their desires.

It will be too bad if China copies us so diligently in that mistake. But the writing does appear to be on the wall. Chinese public opinion is harshly set against a fair deal for the people of Tibet. Leaders in China, like leaders in the United States, do not generally tell their constituents to set their nationalism aside in favor of respect or decency to others.

Ultimately, the problem is with the accumulation of power itself. When a nation becomes as big as China or the U.S., it tends to expect its leaders to achieve outsized things at the expense of other peoples. There had never before been a superpower like the United States. And in its own, different but conceivably far worse way – as we may soon begin to see – there may also never be a superpower like China.