Wednesday, August 5, 2009

The End of the CentOS Netbook Experiment

I no longer have CentOS running on my netbook. It won't be back. At the moment I am not recommending CentOS for anything, not even servers. On my netbook and on desktops in general it has very little to do with the overhyped and exaggerated claims that miscommunication between the developers would lead to the death of CentOS. I have a story I'm writing forO'Reilly Broadcast about that CentOS misadventure, a combination of self-inflicted pain when the developers aired their dirty laundry in public and some in the tech press sensationalizing a story. On servers that story actually does play a significant part in my decision making. Before the CentOS fans out there get all angry at me and start with the inevitable flames let me explain my decisions.

I've decided that RHEL/CentOS just isn't for the typical desktop. The repositories are sparse compared to other distros and I would have had to compile quite a number of apps and dependencies for things I use every day. It was just plain too much work. Yes, I am aware of and tried RPMForge and EPEL and Odiecolon Repo and CentOS Extras. ELRepo proved tremendously useful for firmware and drivers. All of these repos provided useful packages. I tried using yum-priorities to keep all the repos from conflicting with one another and for a time that even seemed to work. With all of those third party repos I still was missing way too many things I use all the time.

I also abhor depending on third party repositories of variable quality. Yes, most of the packages I ended up using were quite good. Some had issues. The fact is that I just do NOT want to rely on multiple sources of packages which I may or may not truly trust. I want the distributor to provide a decent selection of software which they maintain with a decent level of quality assurance. CentOS just doesn't provide that for desktop applications. This was one of my main complaints when I reviewed Slackware 12.1. As much as Slackware fans berated me for this complaint I still don't trust that multiple repos will always play nicely together. They don't. I also do not want to have to build my own packages all the time. I write reviews and write about Linux professionally nowadays. I have to try new things all the time. CentOS is just not well suited for that.

Speaking of new things, I had to go to a third party repo and to compile a webcam app (as there is none worth having in any of the repos) just to make all of my netbook hardware work. My choice of apps was further complicated by the old libraries and tools included with CentOS. Older code makes perfect sense for a stable server environment which, after all, is what CentOS and the upstream Enterprise Linux are designed for. If I want to compile a newer desktop app which depends on newer libraries it may simply mean that the app isn't going to build.

Next comes the performance issue, or more correctly the lack of performance issue. After much tweaking and shutting off of unnecessary services I still found CentOS to be slower than any other distro on the netbook. (This also applies to my aging Toshiba laptop which has hardware fully supported by CentOS.) Even allegedly bloated distros running KDE 4 were faster than CentOS running Xfce. CentOS was and is the only distro I've tried on the netbook that was sluggish at all. Might I have found more stuff to rip out and more performance tuning to do? Sure! I probably could have made it better. The big question is this: Why bother? Was I really getting that much advantage running a business environment on my netbook? I decided the answer was no.

So, between lack of apps, multiple sources and old code CentOS was pretty well doomed on my netbook anyway. The coup de grace came with my last update. After rebooting the system would hang when the ACPI module was loaded. Sure, I could go into single user mode and troubleshoot and fix the problem. I have no doubt about that. I just decided I couldn't be bothered.

OK, so CentOS isn't for netboooks or desktops. You've undoubtedly noted that I said I'm not recommending it on servers either. None of the above really applies to servers, of course. The issue, of late, is the speed, or should I say slowness, of CentOS security patches. This is of vital concern to business and organizational users. When Mozilla released Firefox 3.0.12, a security patch which closed five vulnerabilities identified as "critical", Red Hat had an updated package the very same day. That's what a professional enterprise distro has to do. Downstream Sceintific Linux had a package ready the next day. It took CentOS over a week. This isn't the first such case, either. CentOS has been erratic at best about getting security patches out. The Firefox package was simply the straw that broke the proverbial camel's back.

The net result is that I am now recommending Scientific Linux for people who need a RHEL clone for their business or organization. The story about Lance Davis, the developer that went missing for a time, while overhyped and exaggerated in terms of the impact on the future of CentOS, is relevant here. Scientific Linux is backed by Fermilab, CERN and other major labs and universities. As a result it has a level of funding and stability that an independent project like CentOS does not have. One of the reasons Red Hat does such a good job selling their Enterprise Linux offerings is the support they offer and the strength of the company behind the distro. Scientific Linux may not offer the support or charge for the subscriptions, but the organizational backing insures its future and makes it a stronger choice than CentOS. The recent news forced me to take a long, hard look at Scientific Linux and I decided it was a better choice.

So, for me, CentOS is gone. I do wish the project well. I just hope they find a way to reassure their user community that they can be stable and reliable. The recent bad press has hurt them in that regard.


roderick tapang said...

"..RHEL/CentOS just isn't for the typical desktop. "

it never was and never will be. :)

Tom said...

So the reason you won't recommend CentOS as a server OS is because they lagged abnormally on a patch for firefox?


Why do you have Firefox installed on your server? I certainly don't -- either on my CentOS systems or my RHEL5 ones.

The truly critical bind updates were released within a day, as are most others.

SciLinux is a great distro, but it does not attempt to "clone" RHEL as CentOS does. They remain close, but that's not good enough for many purposes.

Caitlyn said...

@Tom: If it was just Firefox, rather than a history of erratic patching, then that wouldn't be an issue. Of course Red Hat thought it important to patch promptly because their Enterprise Linux *IS* used on corporate desktops and workstations.

The other issue, which you failed to note, is that CentOS is a small volunteer effort. There is still the issue of what happened to the money Mr. Davis collected. The situation is by no means resolved. Small projects go by the wayside as we saw with Whitebox, another RHEL clone. More businesses would be concerned with that than the small (minimal?) deviations between Scientific Linux and RHEL.

Small volunteer projects are little more than hobbyist distros. They simply are not suitable for business.

Chris said...

What has been your experience with the performance of SL compared to CentOS or others on the server side? I'm curious as I have been looking to use SL or CentOS on my server machines.

Caitlyn said...

@Chris: I don't have enough experience with SL to make a fair comparison yet. My initial observation is that there isn't much if any difference but until I've used SL for a while and done some benchmarking I'm not qualified to make a definitive statement answering your question. Sorry.

Christopher J. said...

I admire your attempt, I can't say I've never tried it myself. I ran CentOS as a desktop for about a week at work, only because my company doesn't allow Linux on the desktop. I was going to argue that it was a server and not being used for desktop purposes. But I knew better deep down that it would suck.

We run CentOS on hundreds of servers....none of which have a GUI. It's really not designed for that sort of robust experience you sound like you were looking for in Xwindows.

I really think this little project was doomed from the start. Really, on a netbook? I mean I could've loaned ya my 1st gen Everex Cloudbook with a VIA processor if you really wanted to put some icing on an "epic fail" cake. That VIA processor is full of suxxorz. I'm sure CentOS would've failed even better on it.

jimmy said...

I am linux consultant and have been using a RHEL clone as my working desktop for at least 5 years. I agree that getting current software to work can be a chore sometimes but it can be done. One of the very nice thing about about open source is that you can DIY the software. The RHEL clone I use is Radix Linux which I compiled myself. So any Red Hat update is just a download the source and compile.
Centos is a good project and I hope that they get back on track.

Caitlyn said...

Anytime you want to throw out the Cloudbook send it to me. There is nothing at all wrong with the Via processor in there. I had one in the original Sylvania g Netbook and it benchmarks surprisingly well. The only issues with the Cloudbook were a horrible gOS implementation that lacked drivers and 512MB of RAM instead of the 1GB most netbooks enjoy.

X was no problem on CentOS if you read my posts. The issue was a lack of packages, a forced dependence on third party repositories of variable quality, and old code that doesn't play nicely with new hardware. There also is a lot of bloat for things that make sense on a server but not the desktop. The issue of slow patching is the only one that applies equally to servers and desktops.

I could have made CentOS work on the netbook. The results would not have been worth the effort expended.

Jan Wildeboer, RHCE said...

You say:

"The net result is that I am now recommending Scientific Linux for people who need a RHEL clone for their business or organization."

And I fail to understand why needing a clone is in some way a viable alternative to taking the real thing. Why not simply take RHEL?


Jan (working at Red Hat)

Lee said...

Hello Caitlyn,

I wouldn't run CentOS on any desktop system, its built for stability so it won't have all the cutting edge software that you wanted. That's just the way it is and how I want it to be for a server.

I had the same problems with software "newness" on my home server, I want to run all sorts of funky things on it, but I'd rather have stability over cutting edge for a server anyday.

I've got CentOS on my laptop, works enough for how often I use it, about once a month. Its for taking on site, we have CentOS servers on customers sites, its so I can swap it out if need be.

You can't really criticize CentOS for not being able to do what it wasn't intended to do.

cement_head said...

I've used Ubuntu for 3 years and I know there are some detractors - but its just so simple to use on a daily basis.

Jonabyte said...

Did you run Centos on any production servers, many large companies do so and works very well.
Centos was never meant to be a desktop distro, let alone a netbook one.
Also, most Linux based servers should be run sans desktop, so your Firefox argument should not even apply.

Don il said...

Go Debian!!!

I have recently installed Debian Lenny on five netbooks for dual boot: 2 Acer One and 3 Toshiba NB105.

The tricky part was putting the Wireless netcard to work. Some googling and driver downloading for the Atheros on Toshiba and it went ok.

All other hardware worked right out of the box.

Caitlyn said...

@Jan: There are individuals, non-profits, and yes, some businesses, which choose not to pay the Red Hat subscription fees. They either can't or won't afford the cost. Many companies run fully licensed Red Hat production servers but use CentOS or SL on non-mission critical systems. I discussed that on my latest O'Reilly article:

Yes, of course, running RHEL is the best choice for businesses and organizations simply because Red Hat offers first class support.

@Lee: Red Hat sells Enterprise Linux for the desktop. Many corporate workstations/desktops do run RHEL. CentOS therefore should offer an acceptable desktop experience. To claim it wasn't designed to work on the desktop simply isn't true.

The netbook part of this article was a follow up to two previous posts (one here, the first on DistroWatch) where I simply wanted to recreate my business environment on what is, after all, nothing more or less than a small laptop.

@cement_head: If you tried Ubuntu Jaunty on my netbook with the current Intel video drivers you would be one of the detractors. Far too many bugs for me in general. Thanks but no thanks.

@Jonabyte: Read the piece about CentOS in business I wrote for O'Reilly at I have used CentOS on production servers extensively for four years. I won't be doing so from here on out due to the two factors I mentioned: the lack of corporate/organizational backup to guarantee the future of the distro and the erratic patching.

The Firefox issue most certainly does apply. This isn't one isolated case but on the the most recent one. There has been slow patch delivery on and off for a long time. Also if Red Hat is intended for the business desktop then it's hard to claim that CentOS isn't as well.

DIE said...

Netbooks generally come installed with operating systems that perform decently. If you go and recomend something different make sure you've throughly tested your hypothesis.

I'd recomend you go with Ubuntu 8.10 Intrepid Ibex should you be looking for a stable platform from which to work. Install the codecs and dvd playback and it's as good as it gets.

I'd also recomend to stop hoping back and forth and stick with something even if you hate it.
Make your columns about fixing the problems you encounter and not exploiting them.

Make your own distribution.
I made my own distribution, if you dig around Slackware's Cdrom you find some documentation about making your own cdrom. I threw in a bunch of packages and gave it a name.

I'd planned on doing the same with Ibex but I haven't had much time. Ultimate Edition kind of did this for me.

There are enough people yelling in the world but few listening and helping.

Caitlyn said...

@DIE: The factory installed netbook OS is Ubuntu Netbook Remix 8.04 LTS. It's been updated but it is still installed and, as you say, it works well. My system is triple boot and I have no intention of removing the factory install or the restore partition.

Considering that I get paid to review distributions it would be silly for me to stick with one thing. If you had read my previous two posts you know why I tried CentOS: I am a Linux/UNIX consultant and I use Red Hat Enterprise Linux extensively in my business. I wanted to see if I could replicate my business environment on my netbook. Nothing ventured, nothing gained.

Create my own Linux distro? What a huge waste of time! Why on earth would I want to do that with over 600 distros out there already? If that's what you like to do with your time then more power to you. I am not a computer hobbyist in that sense. It's not for me.

Vector Linux 6.0 also works perfectly on this netbook. So does the distro I'm testing now for my next review. CentOS didn't but at least I had a good reason to try it. Ubuntu Intrepid Ibex didn't work well without patching and recompiling ALSA and also compiling a webcam driver. Thanks but no thanks. You made a recommendation clearly without knowing the hardware or what I had tried already. Perhaps you should read the blogs you comment on more thoroughly in the future. You won't end up looking silly that way.

Anyway, I've done a fair amount of volunteer work for Vector Linux and I'm now doing work for the Linux Yarok project which seems to be on again, at least for the moment. I think I do my share of helping so please don't judge me by one post as "yelling" without helping... or were you talking about yourself there?

feistyfeline said...

I found that Fedora is optimized for Pentium 4, perhaps Centos is the same. Fedora runs slow as molasses in Siberia on my spare PIII/M; Centos 5 runs a lot better. Slackware is so far the best but you must be ready to work harder one keeping up to date (unless you have slapt-get) and achieving a state of perfect installation. Whatever distribution you select for a netbook be prepared for culling the fat: avoid or get rid of desktop effects, indexing, nautilus (or other desktop-based browser that places icons on the desktop, not the on-demand file manager), prelink at least because they sap memory and processing juice, and thrash the disk. Also browse the system services to disable those that are not useful/critically needed like bluetooth and pcsc smart card stuff.
Firefox is not a deal breaker. The generic tarball download actually runs faster believe it or not. Its what I use installed in /opt/firefox and it updates by itself or manually through the help menu as opposed to using yum.
Otherwise its awesome you are trying out all the distros on your netbook. Keeep your reviews coming.

KevinCoonanMD said...

I too have tried, and bailed on using CentOS and RHEL on the desktop. Like you said, it just isn't worth it. I need a desktop environment so I can do my work--I don't need to work on my desktop environment.

I have tried several other distributions, but have stuck w/ OpenSUSE since 10.1 (currently running 11.1 w/ KDE 4.3). There are no shortages of repositories (I think they have made it a little too easy as there are often a dozen different repos all offering the same version of SW offered on the OpenSUSE main repos). While getting 4.3 working has been a bit of an adventure (and not exactly the fun, exciting, type) for the most part software installs and upgrades are easy.
It would be a mistake for people to think that Ubuntuu is the only end-user friendly desktop Linux option

OpenSUSE is backed by Novell, who offers commercial versions and support. Support for OpenSUSE is also available from IBM and elsewhere.