American Brand Name, Chinese Computer

10 days ago I wrote an article for O'Reilly about the premature failure of my Sylvania netbook. I ended up deciding to buy an HP Mini 110 Mi Edition (more on my choices soon) and ordered it custom built to my specs directly from Hewlett-Packard. It was competitively priced (and slightly less expensive than what's on HP's website now) and I was promised that the netbook would ship within six days. The turnaround time was much better than what Dell offered. I've read mainly positive reviews of this particular netbook, including one by Ladislav Bodnar on DistroWatch Weekly. As many of you probably know I write fairly regularly for Ladislav and I definitely to respect the man and his opinions. The netbook did, in fact, ship on the sixth day: from Shangai, China.

I knew that most if not all the netbooks, laptops, and notebooks sold today are built in a handful of factories located either in mainland China or Taiwan. I was under no illusion that buying an American brand name meant an American product. What surprised me is that HP apparantly doesn't even have stock of this netbook in the U.S. and doesn't even do simple customizations here.

In recent weeks Dell announced it was closing a Winston-Salem, North Carolina manufacturing plant and giving the work to "third parties" in Mexico and other countries. Despite President Obama's campaign rehtoric about keeping American manufacturing jobs in the United States companies keep outsourcing the work offshore or moving their own facilities to other countries with less expensive labor costs, poor or no environmental regulations, and little or no protections for workers. Meanwhile unemployment in the United States continues to rise. Here in North Carolina our unemployment rate is well above the national average.

I see this continuing loss of American manufacturing capacity, which is already a small fraction of what it once was, as a long term disaster for this country. Anyone who knows their history knows that when the U.S. entered World War II it rapidly converted the then unmatched manufacturing facilities to wartime production very quickly. If a major world conflict started today or in the near future I fear that we have no ability to do the same thing again. A pretty smart guy named Albert Einstein, who was a pacifist prior to the Second World War, held much more pragmatic views after the rise of Nazism. He famously said, "So long as there are men there will be wars." Sadly I believe he was right. That makes it the height of foolishness to leave the United States in a position where it cannot be ready for war quicky no matter how much we value peace.

I've come to believe a little protectionism would go a long way, particularly when it comes to industries vital to our economy, our defense, or both. I am mindful of history and the last thing I'd want Congress and the President to do is pass something like the Smoot-Hawley Tariff Act during a deep recession or what, I believe in retrospect, we will call a depression. The last thing I want to do is severely curtail international trade. Having said all of that I am no longer the believer in free trade that I once was. I do feel trade will need to be a little less free and a bit more regulated to insure both American jobs and American defense capability.

The kind of severely under-regulated free trade we now have has meant lower prices for consumers. The fact that my new computer is made in China is one of the reasons it is so inexpensive. I would gladly pay more for one made in the United States. I can remember when this country was the leading manufacturer of personal computers. Today, to even recapture a small share of PC manufactuing in the U.S. would require government intervention.

Every President we've had since Ronald Reagan, Republicans and Democrats alike, have repeated the mantra of free trade over and over again. I wonder how Republicans who claim to be so concerned about national defense can justify promoting the wholesale export of manufacturing which has happened over the past 30 years. Similarly, I wonder how Democrats who claim to be so deeply concerned about the environment and workers' rights can do precisely the same when many of the countries which have succesfully imported manufacturing capacity and jobs have no such concerns. Any claim of concern about human rights by either party, by liberals or conservatives, rings hollow when we export our jobs and manufacturing capacity to countries that have abominable human rights records. It seems corporate profits and corporate lobbyists trump any and all other concerns. Greed remains the real G-d of early 21st century America.

It's way past time we examine the real costs of "free trade" instead of just parroting well worn lines about the benefits. I think if we do an honest cost-benefit analysis of free trade we will find that it is anything but free and, indeed, may do more harm than good as it is structured today.

Circling The Wagons

As anyone who reads my blog regularly knows by now I do a fair amount of writing about Linux and Linux distributions for a number of websites. Over the past year I've been paid to do so, moving me out of the "just another blogger" category and into the tech journalist category. Once upon a time when someone called me a "journalist" I would dispute the label but writing professionally has changed that.

Writing honestly about Linux distributions is not a way to become popular or make friends. When a given distribution, any distribution, has problems and a reporter writes about it there are always fans who will circle the wagons and/or go on the attack. I am very used to that by now. It's no surprise at all that has happened with a few CentOS loyalists. What is surprising is that it is continuing more than two weeks after I last wrote about the subject. Ken Leyba's post on the Cooking With Linux blog gets things seriously wrong on a number of levels.

A netbook is nothing more than a small notebook. Mr. Leyba is completely off base when he claims otherwise. Machines like this have been used in business for about forever. I remember the CEO and other senior execs of a company I supported when I contracted to IBM Global Services using the itty bitty Toshiba Libretto with a 7" screen for travel back in 1999. Nothing has changed all that much other than the fact that you no longer pay a premium for tiny and that has helped make small machines popular. Linux makes them more productive.

Enterprise Linux is marketed for the desktop. To claim that CentOS, an EL clone, is only for servers and that my experiment was somehow invalid because it was done on a desktop/notebook/netbook is patently ridiculous. When I was consulting for Red Hat in 2004-2005 I visited a number of companies that were and undoubtedly still are using Red Hat Enterprise Linux on desktops, workstations, and yes, on laptops. Those enterprise customers would have considered a failure to deliver a Firefox patch that closed a number of critical security vulnerabilities to be a serious problem. The idea that a Firefox patch is irrelevant on an enterprise distribution as some have claimed is simply preposterous.

CentOS has to prove they can get patches out on a timely basis to be taken seriously as an enterprise product. Their track record in that area over the past year has been atrocious. It wasn't one Firefox package. It was a year of things arriving late, sometimes months late.

The response from CentOS developers to the security issue in the comments of my business-centric article for O'Reilly Broadcast on the subject was actually spot on:

"Your point about the security updates is well founded, we try a lot harder to make sure we get things right and we have a much more involved process to establish when the 'right' is. An easy way to work through this would be if Red Hat were to share more info with us. Not sure if that is likely to happen and what the timeframe for that would be, but over the course of the next few months we hope to have a more transparent process in place that lets users track exactly what is going on, where and how."

When patches are a few days late it can, indeed, have something to do with the upstream vendor. When they are two months late that seems entirely unlikely. The fact that a member of the CentOS development team is owning the problem making a commitment to "get things right" in the future is a very positive step.

My complaints about the dependency on third party repositories and the lack of packages in general is an issue that is hardly unique to CentOS. I raised the same issue when I reviewed Slackeare 12.1 last year. It's a significant issue for any desktop/laptop user, not just on netbooks. The need to go to third parties for packages to adequately support newer hardware isn't netbook specific either.

I also feel that the issues around the open letter to Lance Davis called into question how that distro is being managed. Yes, it was blown way out of proportion by the tech media and I said as much in my O'Reilly Broadcast article. That one issue may well have been solved and I certainly bear the CentOS devs no ill will. That doesn't change the fact that Scientific Linux has done a better job with getting patches out on a timely basis. It is not dependent on a few volunteers and has the backing of and funding from major laboratories and universities all over the world. From a business perspective that makes Scientific Linux a safer choice for an Enterprise Linux clone.

The attempt to replicate my business environment on my netbook was a valid experiment. I thought other Linux users, those who think highly of Red Hat Enterprise Linux as I do, might want to do the same. The original DistroWatch Weekly feature article documented the difficulties in doing so. The post here which Mr. Leyba responded to was the fourth of five if you include the original DistroWatch piece and the business-centric O'Reilly piece. Considering that I linked back to the preceding articles should be a very clear message that the one post shouldn't be taken out of context but as part of a larger whole. In any case it wasn't a "rant" against CentOS as Mr. Leyba claims, but rather it tied the proverbial ribbons on the end of my experiment and explained why I decided the whole thing was more trouble than it was worth.

I did make a mistake in putting issues regarding servers and desktop-specific issues in one blog post. While many of the issues involved impact both areas the fact that I did not make a clear enough delineation between the two almost certainly generated misunderstanding. In that sense I did fail to communicate clearly. Mea culpa.

It's important to note that the experiment really wasn't a total failure. In the end I did get the netbook hardware to be 100% functional running CentOS. I also managed to improve performance significantly. What I also did was document the difficulties involved. I don't think there was anything wrong with doing so.

Mr. Leyba has shown integrity by allowing me to respond to him and to his readers directly on the Cooking With Linux blog. I do appreciate that and I will certainly afford him the same opportunity if he wishes to comment.

I thought I had put this issue to bed and moved on a couple of weeks ago. Clearly some people had other ideas. That's perfectly fine. In the final analysis nothing has really changed. I stand by all the pieces I wrote on the subject of CentOS. I'm using Scientific Linux instead and that will continue to be my recommendation for anyone who wants a no cost, no support Enterprise Linux clone either in the server room or on the desktop.

A Perfect Illustration of Why I Now Choose Scientific Linux Over CentOS

The following comment was posted in response to my business oriented article about the CentOS situation for O'Reilly Broadcast. It perfectly illustrates why I have made the decision I did:

---

By Peter Griffin on August 6, 2009 10:15 AM | Reply

I've been running CentOS as a file/intranet server since 5.0 was released. I started becoming concerned during the protracted period that it took to get 5.3 out. Not about the "lateness" in getting 5.3 out, but the complete lack of security updates in the interim for my 5.2 system. This "No updates available" went on for over a month. My version of Firefox trailed behind Red Hat's by two versions.

We can go back and forth about how many developers CentOS has vs. Scientific Linux. To me, that is academic if the Scientific Linux developers get their distro out several weeks in advance of CentOS, and more importantly, provide more timely security patches. If I'm not mistaken, Scientific Linux also supports older "dot" releases, such as 5.1, 5.2, etc., while CentOS does not. Not an issue for me, but it does indicate a little more thoroughness on the part of the Scientific Linux developers, few in number as they may be.

---

This wasn't a hasty decision. It wasn't made in a vacuum. Firefox wasn't the reason I talked about late patches, only the most recent example.

I wish the CentOS project and its developers well. Maybe in the future the CentOS developers can correct the issues that I've raised and I will reconsider the distro. They would not only need to get security patches out on a timely bases but they would also need some sort of institutional backing to insure their future before I will consider recommending them again.

The End of the CentOS Netbook Experiment

I no longer have CentOS running on my netbook. It won't be back. At the moment I am not recommending CentOS for anything, not even servers. On my netbook and on desktops in general it has very little to do with the overhyped and exaggerated claims that miscommunication between the developers would lead to the death of CentOS. I have a story I'm writing forO'Reilly Broadcast about that CentOS misadventure, a combination of self-inflicted pain when the developers aired their dirty laundry in public and some in the tech press sensationalizing a story. On servers that story actually does play a significant part in my decision making. Before the CentOS fans out there get all angry at me and start with the inevitable flames let me explain my decisions.

I've decided that RHEL/CentOS just isn't for the typical desktop. The repositories are sparse compared to other distros and I would have had to compile quite a number of apps and dependencies for things I use every day. It was just plain too much work. Yes, I am aware of and tried RPMForge and EPEL and Odiecolon Repo and CentOS Extras. ELRepo proved tremendously useful for firmware and drivers. All of these repos provided useful packages. I tried using yum-priorities to keep all the repos from conflicting with one another and for a time that even seemed to work. With all of those third party repos I still was missing way too many things I use all the time.

I also abhor depending on third party repositories of variable quality. Yes, most of the packages I ended up using were quite good. Some had issues. The fact is that I just do NOT want to rely on multiple sources of packages which I may or may not truly trust. I want the distributor to provide a decent selection of software which they maintain with a decent level of quality assurance. CentOS just doesn't provide that for desktop applications. This was one of my main complaints when I reviewed Slackware 12.1. As much as Slackware fans berated me for this complaint I still don't trust that multiple repos will always play nicely together. They don't. I also do not want to have to build my own packages all the time. I write reviews and write about Linux professionally nowadays. I have to try new things all the time. CentOS is just not well suited for that.

Speaking of new things, I had to go to a third party repo and to compile a webcam app (as there is none worth having in any of the repos) just to make all of my netbook hardware work. My choice of apps was further complicated by the old libraries and tools included with CentOS. Older code makes perfect sense for a stable server environment which, after all, is what CentOS and the upstream Enterprise Linux are designed for. If I want to compile a newer desktop app which depends on newer libraries it may simply mean that the app isn't going to build.

Next comes the performance issue, or more correctly the lack of performance issue. After much tweaking and shutting off of unnecessary services I still found CentOS to be slower than any other distro on the netbook. (This also applies to my aging Toshiba laptop which has hardware fully supported by CentOS.) Even allegedly bloated distros running KDE 4 were faster than CentOS running Xfce. CentOS was and is the only distro I've tried on the netbook that was sluggish at all. Might I have found more stuff to rip out and more performance tuning to do? Sure! I probably could have made it better. The big question is this: Why bother? Was I really getting that much advantage running a business environment on my netbook? I decided the answer was no.

So, between lack of apps, multiple sources and old code CentOS was pretty well doomed on my netbook anyway. The coup de grace came with my last update. After rebooting the system would hang when the ACPI module was loaded. Sure, I could go into single user mode and troubleshoot and fix the problem. I have no doubt about that. I just decided I couldn't be bothered.

OK, so CentOS isn't for netboooks or desktops. You've undoubtedly noted that I said I'm not recommending it on servers either. None of the above really applies to servers, of course. The issue, of late, is the speed, or should I say slowness, of CentOS security patches. This is of vital concern to business and organizational users. When Mozilla released Firefox 3.0.12, a security patch which closed five vulnerabilities identified as "critical", Red Hat had an updated package the very same day. That's what a professional enterprise distro has to do. Downstream Sceintific Linux had a package ready the next day. It took CentOS over a week. This isn't the first such case, either. CentOS has been erratic at best about getting security patches out. The Firefox package was simply the straw that broke the proverbial camel's back.

The net result is that I am now recommending Scientific Linux for people who need a RHEL clone for their business or organization. The story about Lance Davis, the developer that went missing for a time, while overhyped and exaggerated in terms of the impact on the future of CentOS, is relevant here. Scientific Linux is backed by Fermilab, CERN and other major labs and universities. As a result it has a level of funding and stability that an independent project like CentOS does not have. One of the reasons Red Hat does such a good job selling their Enterprise Linux offerings is the support they offer and the strength of the company behind the distro. Scientific Linux may not offer the support or charge for the subscriptions, but the organizational backing insures its future and makes it a stronger choice than CentOS. The recent news forced me to take a long, hard look at Scientific Linux and I decided it was a better choice.

So, for me, CentOS is gone. I do wish the project well. I just hope they find a way to reassure their user community that they can be stable and reliable. The recent bad press has hurt them in that regard.

The Ongoing "CentOS 5.3 On A Netbook" Saga

Last week I wrote a feature for DistroWatch Weekly on the trials and tribulations of installing CentOS 5.3 on my Sylvania g Netbook Meso. I was very pleased when Dag Wieërs, a CentOS developer and long time packager and maintainer of an excellent repository of additional packages for Red Hat Enterprise Linux, posted in the comments section with some useful suggestions and a request that I document how to get the netbook completely functional in the CentOS wiki. That's a project I'll take on when I have everything working to my satisfaction. In the meanwhile I thought it might be useful to report my progress here. That's the kind of thing blogs are for, right?

Anyway, following Dag's suggestion I ripped out the wireless driver I compiled from source and tried to use what CentOS provides. As before that simply didn't work. Then I installed the rt73usb-firmware package from ELRepo. This worked after a fashion. CentOS did recognize my wireless card and, three passwords later (for the keyring and twice for my own WPA2 protected network) I actually was able to connect. I did have one instance since yesterday when NetworkManager locked up, something I used to see in Ubuntu all the time. That, plus the relatively slow performance and all the GNOME dependencies are just a few of the reasons why I tend to truly hate NetworkManager. Unfortunately nobody has a wicd package for CentOS. If I stick with CentOS I will definitely have to package wicd and write a how-to documenting replacing NetworkManager with it.

I also followed Dag's suggestion for installing the kmod-video4linux package from ELRepo in the hope that it may contain drivers for my webcam. No distro supports the webcam out of the box so far as I can tell. Only the factory preinstalled Ubuntu Netbook Remix 8.04 worked without significant effort. Testing with the video conferencing software is a royal pain and possibly not a fair test so I decided to install either wxcam or cheese or both. Guess what? None of the CentOS repos have packages for those either. Grrr!

I did install Xfce 4.4.2 from the CentOS Extras repository and, as expected, it's a bit faster and less resource intensive than GNOME. I'd have liked a somewhat newer version but I'm happy with the results.

I also received an e-mail from Radu-Cristian Fotescu, who writes the Planète Béranger blog, asking me to test his recently rebuilt Odiecolon repository for RHEL/CentOS 5. He has some more up-to-date multimedia packages than other repos so I decided to give it a go. His repo is also known to conflict with the much larger and definitely necessary RPMForge repository. In order to avoid "rpm hell" I am trying to keep things in order with yum priorities. So far it's doing a good job of excluding conflicting packages.

If all this sounds overly complex, well... it is. Back when I wrote my somewhat controversial review of Slackware 12.1 one of the things I complained about was the dependency on third party repositories of variable quality. RHEL/CentOS as a desktop distribution definitely suffers from the same shortcoming. Getting all the repos to play nicely together, even with yum-priorities, appears to be an arcane art. Only time will tell whether or not I've mastered it. I will also need to compile a bunch of apps from source and will probably end up building and contributing significant numbers of packages if I intend to stick with CentOS.

I should also add the performance is still nowhere near as good as my favorite desktop distro, VectorLinux or even as good as Ubuntu Netbook Remix. With further tweaking and streamlining I'm fairly certain I can improve things enough to be reasonably happy.

If it weren't for the fact that Red Hat Enterprise Linux is the absolutely dominant business distribution I honestly wouldn't bother with all of this. The main reason for the effort is to be able to use the netbook for business related demonstrations and to have a fairly uniform operating environment for my systems. Check back for further progress reports.