Friday, August 21, 2009

Circling The Wagons

As anyone who reads my blog regularly knows by now I do a fair amount of writing about Linux and Linux distributions for a number of websites. Over the past year I've been paid to do so, moving me out of the "just another blogger" category and into the tech journalist category. Once upon a time when someone called me a "journalist" I would dispute the label but writing professionally has changed that.

Writing honestly about Linux distributions is not a way to become popular or make friends. When a given distribution, any distribution, has problems and a reporter writes about it there are always fans who will circle the wagons and/or go on the attack. I am very used to that by now. It's no surprise at all that has happened with a few CentOS loyalists. What is surprising is that it is continuing more than two weeks after I last wrote about the subject. Ken Leyba's post on the Cooking With Linux blog gets things seriously wrong on a number of levels.

A netbook is nothing more than a small notebook. Mr. Leyba is completely off base when he claims otherwise. Machines like this have been used in business for about forever. I remember the CEO and other senior execs of a company I supported when I contracted to IBM Global Services using the itty bitty Toshiba Libretto with a 7" screen for travel back in 1999. Nothing has changed all that much other than the fact that you no longer pay a premium for tiny and that has helped make small machines popular. Linux makes them more productive.

Enterprise Linux is marketed for the desktop. To claim that CentOS, an EL clone, is only for servers and that my experiment was somehow invalid because it was done on a desktop/notebook/netbook is patently ridiculous. When I was consulting for Red Hat in 2004-2005 I visited a number of companies that were and undoubtedly still are using Red Hat Enterprise Linux on desktops, workstations, and yes, on laptops. Those enterprise customers would have considered a failure to deliver a Firefox patch that closed a number of critical security vulnerabilities to be a serious problem. The idea that a Firefox patch is irrelevant on an enterprise distribution as some have claimed is simply preposterous.

CentOS has to prove they can get patches out on a timely basis to be taken seriously as an enterprise product. Their track record in that area over the past year has been atrocious. It wasn't one Firefox package. It was a year of things arriving late, sometimes months late.

The response from CentOS developers to the security issue in the comments of my business-centric article for O'Reilly Broadcast on the subject was actually spot on:
"Your point about the security updates is well founded, we try a lot harder to make sure we get things right and we have a much more involved process to establish when the 'right' is. An easy way to work through this would be if Red Hat were to share more info with us. Not sure if that is likely to happen and what the timeframe for that would be, but over the course of the next few months we hope to have a more transparent process in place that lets users track exactly what is going on, where and how."
When patches are a few days late it can, indeed, have something to do with the upstream vendor. When they are two months late that seems entirely unlikely. The fact that a member of the CentOS development team is owning the problem making a commitment to "get things right" in the future is a very positive step.

My complaints about the dependency on third party repositories and the lack of packages in general is an issue that is hardly unique to CentOS. I raised the same issue when I reviewed Slackeare 12.1 last year. It's a significant issue for any desktop/laptop user, not just on netbooks. The need to go to third parties for packages to adequately support newer hardware isn't netbook specific either.

I also feel that the issues around the open letter to Lance Davis called into question how that distro is being managed. Yes, it was blown way out of proportion by the tech media and I said as much in my O'Reilly Broadcast article. That one issue may well have been solved and I certainly bear the CentOS devs no ill will. That doesn't change the fact that Scientific Linux has done a better job with getting patches out on a timely basis. It is not dependent on a few volunteers and has the backing of and funding from major laboratories and universities all over the world. From a business perspective that makes Scientific Linux a safer choice for an Enterprise Linux clone.

The attempt to replicate my business environment on my netbook was a valid experiment. I thought other Linux users, those who think highly of Red Hat Enterprise Linux as I do, might want to do the same. The original DistroWatch Weekly feature article documented the difficulties in doing so. The post here which Mr. Leyba responded to was the fourth of five if you include the original DistroWatch piece and the business-centric O'Reilly piece. Considering that I linked back to the preceding articles should be a very clear message that the one post shouldn't be taken out of context but as part of a larger whole. In any case it wasn't a "rant" against CentOS as Mr. Leyba claims, but rather it tied the proverbial ribbons on the end of my experiment and explained why I decided the whole thing was more trouble than it was worth.

I did make a mistake in putting issues regarding servers and desktop-specific issues in one blog post. While many of the issues involved impact both areas the fact that I did not make a clear enough delineation between the two almost certainly generated misunderstanding. In that sense I did fail to communicate clearly. Mea culpa.

It's important to note that the experiment really wasn't a total failure. In the end I did get the netbook hardware to be 100% functional running CentOS. I also managed to improve performance significantly. What I also did was document the difficulties involved. I don't think there was anything wrong with doing so.

Mr. Leyba has shown integrity by allowing me to respond to him and to his readers directly on the Cooking With Linux blog. I do appreciate that and I will certainly afford him the same opportunity if he wishes to comment.

I thought I had put this issue to bed and moved on a couple of weeks ago. Clearly some people had other ideas. That's perfectly fine. In the final analysis nothing has really changed. I stand by all the pieces I wrote on the subject of CentOS. I'm using Scientific Linux instead and that will continue to be my recommendation for anyone who wants a no cost, no support Enterprise Linux clone either in the server room or on the desktop.

Thursday, August 6, 2009

A Perfect Illustration of Why I Now Choose Scientific Linux Over CentOS

The following comment was posted in response to my business oriented article about the CentOS situation for O'Reilly Broadcast. It perfectly illustrates why I have made the decision I did:

I've been running CentOS as a file/intranet server since 5.0 was released. I started becoming concerned during the protracted period that it took to get 5.3 out. Not about the "lateness" in getting 5.3 out, but the complete lack of security updates in the interim for my 5.2 system. This "No updates available" went on for over a month. My version of Firefox trailed behind Red Hat's by two versions.

We can go back and forth about how many developers CentOS has vs. Scientific Linux. To me, that is academic if the Scientific Linux developers get their distro out several weeks in advance of CentOS, and more importantly, provide more timely security patches. If I'm not mistaken, Scientific Linux also supports older "dot" releases, such as 5.1, 5.2, etc., while CentOS does not. Not an issue for me, but it does indicate a little more thoroughness on the part of the Scientific Linux developers, few in number as they may be.

This wasn't a hasty decision. It wasn't made in a vacuum. Firefox wasn't the reason I talked about late patches, only the most recent example.

I wish the CentOS project and its developers well. Maybe in the future the CentOS developers can correct the issues that I've raised and I will reconsider the distro. They would not only need to get security patches out on a timely bases but they would also need some sort of institutional backing to insure their future before I will consider recommending them again.

Wednesday, August 5, 2009

The End of the CentOS Netbook Experiment

I no longer have CentOS running on my netbook. It won't be back. At the moment I am not recommending CentOS for anything, not even servers. On my netbook and on desktops in general it has very little to do with the overhyped and exaggerated claims that miscommunication between the developers would lead to the death of CentOS. I have a story I'm writing forO'Reilly Broadcast about that CentOS misadventure, a combination of self-inflicted pain when the developers aired their dirty laundry in public and some in the tech press sensationalizing a story. On servers that story actually does play a significant part in my decision making. Before the CentOS fans out there get all angry at me and start with the inevitable flames let me explain my decisions.

I've decided that RHEL/CentOS just isn't for the typical desktop. The repositories are sparse compared to other distros and I would have had to compile quite a number of apps and dependencies for things I use every day. It was just plain too much work. Yes, I am aware of and tried RPMForge and EPEL and Odiecolon Repo and CentOS Extras. ELRepo proved tremendously useful for firmware and drivers. All of these repos provided useful packages. I tried using yum-priorities to keep all the repos from conflicting with one another and for a time that even seemed to work. With all of those third party repos I still was missing way too many things I use all the time.

I also abhor depending on third party repositories of variable quality. Yes, most of the packages I ended up using were quite good. Some had issues. The fact is that I just do NOT want to rely on multiple sources of packages which I may or may not truly trust. I want the distributor to provide a decent selection of software which they maintain with a decent level of quality assurance. CentOS just doesn't provide that for desktop applications. This was one of my main complaints when I reviewed Slackware 12.1. As much as Slackware fans berated me for this complaint I still don't trust that multiple repos will always play nicely together. They don't. I also do not want to have to build my own packages all the time. I write reviews and write about Linux professionally nowadays. I have to try new things all the time. CentOS is just not well suited for that.

Speaking of new things, I had to go to a third party repo and to compile a webcam app (as there is none worth having in any of the repos) just to make all of my netbook hardware work. My choice of apps was further complicated by the old libraries and tools included with CentOS. Older code makes perfect sense for a stable server environment which, after all, is what CentOS and the upstream Enterprise Linux are designed for. If I want to compile a newer desktop app which depends on newer libraries it may simply mean that the app isn't going to build.

Next comes the performance issue, or more correctly the lack of performance issue. After much tweaking and shutting off of unnecessary services I still found CentOS to be slower than any other distro on the netbook. (This also applies to my aging Toshiba laptop which has hardware fully supported by CentOS.) Even allegedly bloated distros running KDE 4 were faster than CentOS running Xfce. CentOS was and is the only distro I've tried on the netbook that was sluggish at all. Might I have found more stuff to rip out and more performance tuning to do? Sure! I probably could have made it better. The big question is this: Why bother? Was I really getting that much advantage running a business environment on my netbook? I decided the answer was no.

So, between lack of apps, multiple sources and old code CentOS was pretty well doomed on my netbook anyway. The coup de grace came with my last update. After rebooting the system would hang when the ACPI module was loaded. Sure, I could go into single user mode and troubleshoot and fix the problem. I have no doubt about that. I just decided I couldn't be bothered.

OK, so CentOS isn't for netboooks or desktops. You've undoubtedly noted that I said I'm not recommending it on servers either. None of the above really applies to servers, of course. The issue, of late, is the speed, or should I say slowness, of CentOS security patches. This is of vital concern to business and organizational users. When Mozilla released Firefox 3.0.12, a security patch which closed five vulnerabilities identified as "critical", Red Hat had an updated package the very same day. That's what a professional enterprise distro has to do. Downstream Sceintific Linux had a package ready the next day. It took CentOS over a week. This isn't the first such case, either. CentOS has been erratic at best about getting security patches out. The Firefox package was simply the straw that broke the proverbial camel's back.

The net result is that I am now recommending Scientific Linux for people who need a RHEL clone for their business or organization. The story about Lance Davis, the developer that went missing for a time, while overhyped and exaggerated in terms of the impact on the future of CentOS, is relevant here. Scientific Linux is backed by Fermilab, CERN and other major labs and universities. As a result it has a level of funding and stability that an independent project like CentOS does not have. One of the reasons Red Hat does such a good job selling their Enterprise Linux offerings is the support they offer and the strength of the company behind the distro. Scientific Linux may not offer the support or charge for the subscriptions, but the organizational backing insures its future and makes it a stronger choice than CentOS. The recent news forced me to take a long, hard look at Scientific Linux and I decided it was a better choice.

So, for me, CentOS is gone. I do wish the project well. I just hope they find a way to reassure their user community that they can be stable and reliable. The recent bad press has hurt them in that regard.