Friday, August 21, 2009

Circling The Wagons

As anyone who reads my blog regularly knows by now I do a fair amount of writing about Linux and Linux distributions for a number of websites. Over the past year I've been paid to do so, moving me out of the "just another blogger" category and into the tech journalist category. Once upon a time when someone called me a "journalist" I would dispute the label but writing professionally has changed that.

Writing honestly about Linux distributions is not a way to become popular or make friends. When a given distribution, any distribution, has problems and a reporter writes about it there are always fans who will circle the wagons and/or go on the attack. I am very used to that by now. It's no surprise at all that has happened with a few CentOS loyalists. What is surprising is that it is continuing more than two weeks after I last wrote about the subject. Ken Leyba's post on the Cooking With Linux blog gets things seriously wrong on a number of levels.

A netbook is nothing more than a small notebook. Mr. Leyba is completely off base when he claims otherwise. Machines like this have been used in business for about forever. I remember the CEO and other senior execs of a company I supported when I contracted to IBM Global Services using the itty bitty Toshiba Libretto with a 7" screen for travel back in 1999. Nothing has changed all that much other than the fact that you no longer pay a premium for tiny and that has helped make small machines popular. Linux makes them more productive.

Enterprise Linux is marketed for the desktop. To claim that CentOS, an EL clone, is only for servers and that my experiment was somehow invalid because it was done on a desktop/notebook/netbook is patently ridiculous. When I was consulting for Red Hat in 2004-2005 I visited a number of companies that were and undoubtedly still are using Red Hat Enterprise Linux on desktops, workstations, and yes, on laptops. Those enterprise customers would have considered a failure to deliver a Firefox patch that closed a number of critical security vulnerabilities to be a serious problem. The idea that a Firefox patch is irrelevant on an enterprise distribution as some have claimed is simply preposterous.

CentOS has to prove they can get patches out on a timely basis to be taken seriously as an enterprise product. Their track record in that area over the past year has been atrocious. It wasn't one Firefox package. It was a year of things arriving late, sometimes months late.

The response from CentOS developers to the security issue in the comments of my business-centric article for O'Reilly Broadcast on the subject was actually spot on:
"Your point about the security updates is well founded, we try a lot harder to make sure we get things right and we have a much more involved process to establish when the 'right' is. An easy way to work through this would be if Red Hat were to share more info with us. Not sure if that is likely to happen and what the timeframe for that would be, but over the course of the next few months we hope to have a more transparent process in place that lets users track exactly what is going on, where and how."
When patches are a few days late it can, indeed, have something to do with the upstream vendor. When they are two months late that seems entirely unlikely. The fact that a member of the CentOS development team is owning the problem making a commitment to "get things right" in the future is a very positive step.

My complaints about the dependency on third party repositories and the lack of packages in general is an issue that is hardly unique to CentOS. I raised the same issue when I reviewed Slackeare 12.1 last year. It's a significant issue for any desktop/laptop user, not just on netbooks. The need to go to third parties for packages to adequately support newer hardware isn't netbook specific either.

I also feel that the issues around the open letter to Lance Davis called into question how that distro is being managed. Yes, it was blown way out of proportion by the tech media and I said as much in my O'Reilly Broadcast article. That one issue may well have been solved and I certainly bear the CentOS devs no ill will. That doesn't change the fact that Scientific Linux has done a better job with getting patches out on a timely basis. It is not dependent on a few volunteers and has the backing of and funding from major laboratories and universities all over the world. From a business perspective that makes Scientific Linux a safer choice for an Enterprise Linux clone.

The attempt to replicate my business environment on my netbook was a valid experiment. I thought other Linux users, those who think highly of Red Hat Enterprise Linux as I do, might want to do the same. The original DistroWatch Weekly feature article documented the difficulties in doing so. The post here which Mr. Leyba responded to was the fourth of five if you include the original DistroWatch piece and the business-centric O'Reilly piece. Considering that I linked back to the preceding articles should be a very clear message that the one post shouldn't be taken out of context but as part of a larger whole. In any case it wasn't a "rant" against CentOS as Mr. Leyba claims, but rather it tied the proverbial ribbons on the end of my experiment and explained why I decided the whole thing was more trouble than it was worth.

I did make a mistake in putting issues regarding servers and desktop-specific issues in one blog post. While many of the issues involved impact both areas the fact that I did not make a clear enough delineation between the two almost certainly generated misunderstanding. In that sense I did fail to communicate clearly. Mea culpa.

It's important to note that the experiment really wasn't a total failure. In the end I did get the netbook hardware to be 100% functional running CentOS. I also managed to improve performance significantly. What I also did was document the difficulties involved. I don't think there was anything wrong with doing so.

Mr. Leyba has shown integrity by allowing me to respond to him and to his readers directly on the Cooking With Linux blog. I do appreciate that and I will certainly afford him the same opportunity if he wishes to comment.

I thought I had put this issue to bed and moved on a couple of weeks ago. Clearly some people had other ideas. That's perfectly fine. In the final analysis nothing has really changed. I stand by all the pieces I wrote on the subject of CentOS. I'm using Scientific Linux instead and that will continue to be my recommendation for anyone who wants a no cost, no support Enterprise Linux clone either in the server room or on the desktop.


Daniel Fish said...

Just curious... what is your opinion on ubuntu/xubuntu in a small business environment?

Back in 2004, as a voluntary exile from the Microsoft world, I chose to use ubuntu as our Windows replacement because it was what I was most familiar with. Now here we are, almost 2010, and my small business has replaced all but 1 of our Microsoft servers with Ubuntu 9.04 server. We also have 2 roaming users with xubuntu 9.04 laptops. Over the past 6 years, I haven't evaluated any other distributions. Your blog has made me curious whether we would be better served by a different product. Your thoughts?


Unknown said...

This is an unfortunate truth. I can see both sides of the story, though: on one hand, Linux will never really appreciably improve without some honest , constructive criticism; on the other hand, I think the reason why Liinux fans get so defensive in the face of any criticism is that we're used to hearing so much completely unsubstantiated FUD about Linux (a lot of the old myths are still out there, and people still believe them). "Circling the wagons" seem (IMHO) to be a knee-jerk reaction to what Linux fans may perceive (rightly or wrongly) to be yet another round of FUD coming from some detractor.

It's like being punched so many times that, when someone so much as makes a slight move, the victim will sometimes overreact. Just my 2 cents' worth...

Caitlyn Martin said...

@Daniel: Personally I would not use the standard Ubuntu in a business environment but I have no objection to the LTS releases. The standard Ubuntu has too many bugs for my taste and often fixes are delayed until the next release. The six month upgrade cycle is way too fast for most businesses. 18 months of support is also way too short.

Canonical sells the LTS releases, which come out ever two years or so, as their business solution. By the first maintenance release the bugs have generally been squashed. Ubuntu released 8.04.3 (third maintenance release) for Hardy Heron in July.

I still believe that Ubuntu cannot match the reliability of Red Hat. I'll freely admit that I am biased towards Red Hat support because I used to be part of it. Having said that, the Ubuntu LTS releases meet my criteria for business use.

Pilot said...

We have been pleased with Ubuntu server performance. We only use the LTS versions. The following link indicates that Ubuntu server has very good reliability on par with Red Hat and SuSE. Your bias toward Red Hat may be out of date.

Johnny Hughes said...

CentOS has to prove they can get patches out on a timely basis to be taken seriously as an enterprise product.

Really? More than 2 million unique IP addresses do updates against our nearly 300 servers every point release. CentOS is in use at IBM, HP, Dell, and Sun. CentOS is the OS on several of the worlds largest linux machines, including 5 of the top 500 super computers ( Actually CentOS is on more than 5 of those, but some have identified themselves only as Linux.

Can we get better and faster at updates, sure we can. In fact, you will notice a marked improvement moving forward. But lets not pretend that CentOS is not taken seriously in the Enterprise right now.

Caitlyn Martin said...

CentOS used to (past tense) have an excellent track record and that led to a lot of deployments. Over the past year they cannot get security patches out on a timely basis and that is absolutely inexcusable. I suspect the number of deployments will drop very sharply if the CentOS developers can't do a better job getting patches out.

In its present state CentOS should not be seriously considered for new deployments and existing deployments should be as shifted to Scientific Linux as servers are refreshed.

BigAl said...

Well said, Caitlyn.

I used to spend quite a chunk of time writing scripts to tune various desktop distros to my liking (and to the liking of others). Debian is fine, but..... Ubuntu is a fine Debian refinement, but... Lately I am just using Linux Mint - version 5, which is built on Ubuntu 8.04 LTS. It's still not 100% to my liking, but 98% is probably as close as we'll ever get.

You should give it a whirl.

Anonymous said...

I guess you are true Linux journalist since you make the same mistake almost all of them make. You mix very distinct things, many of your readers follow, and huge confusion surfaces.

Currently there are 2 kinds of Linux - those who want to sell themselves to general public and those who don't. For example, Red Hat and Ubuntu evidently fell into the first category. Slackware is commercial, but belongs to the second one, together with Debian.

So, it is very good if you have something you believe is a valid issue (complaint?) or constructive criticism and all Linux fanboys are happy you do, provided you present it correctly.

First, it should be clearly indicated who is making the complaint. A happy Windows user migrating to Linux (what are her reasons to do so, BTW)? A CIO cutting costs? A poor student reviving grandma's PC? A scientist eager to put more nodes into his model? If it is not crystal clear some of the readers may erroneously identify themselves with the source of the complaint and you will get negative feedback.

Second, pay attention to what kind of Linux you are addressing. If Linux wants to sell something to you, it is perfectly valid to say "unless this is fixed, I wont buy", which is equivalent to "it lacks that and this should also be added" or "I did that and it did not work". If Linux does not want to sell, most likely it is a community thing and, you know, the members of a community are normally happy with what they have. If not, they change the thing accordingly or leave. Thus, anything like quoted above is totally inappropriate. "Personally, I do not know how", "it may not be well suited for this purpose", "I expected that but because of this it is done differently" should be used instead.

For example, the dependency on 3-d party repositories is not an issue in Slackware 12.1 since the Slackers prefer it that way.

You may want to meditate on the koan of Slackware. It is a self sustaining business, but is the Slackware Store the main feature of its site? How can community members be active in discussions and make lots of proposals if they are absolutely sure that PV knows better? Why contributors working for free never react like Debian developers reacted when some other Debian developers were compensated?

Let us consider my favorite Linux weakness: sound. Layer upon layer and still not free of conflicts. It is evident how it should be done: an entity must exist (whatever MS wants on Windows, possibly a special Open Sound dialog; a special file on Linux since everything is a file on Unix) that is opened by an application for audio output. All configuration should be done on the system level, not on the application level. Currently sound is done wrong (suboptimal?) in both Windows and Linux (however, I know how to listen to a movie with a USB device while monitoring recording from an analog source through earphones on Linux but not on Windows).

There are many ways to write about it: informative (what the architecture is and how it is evolving), critical (unless fixed, I will not do any sound on a PC), constructive (if done right, that many users will switch to Linux). My guess is that neither way will cause wagon moving. "What a crap of a distro: I opened this program and that another one was mute" surely will cause defensive comments.

Caitlyn Martin said...

@octagon: When I review Slackware, as an example, I am not writing to please Slackers, nor am I writing for people who know and run Slackware already. They know the pros and the cons. I am writing for the vast majority of people out there, Linux users included, who have either never run Slackware or else have not run it in a very long time and want to know what it's like today. That makes an issue like a small official repository and dependence on third party repositories of questionable quality a HUGE issue. You say otherwise because you are NOT the target audience.

Also, I don't meditate about operating systems. I'm not a fan (short for fanatic) of any of them. They are a tool; a means to an end. Those who elevate them to a quasi-religious status are generally unhelpful to the Linux community as a whole. Most people don't look on zealotry in a favorable way. In general I completely disagree when you say fans will accept constructive criticism. That is completely contrary to my experience.

When writing about what is sold as a scalable distribution to be used across a wide variety of hardware and for a variety of uses, whether at home or in business, you can't possibly do a good job writing about it by narrowing the focus. You want me to pigeon hole and target a very narrow audience in order not to offend. Doing so would kind of defeat the purpose of writing for a wide audience, and that is what I am paid to do.

I never called CentOS crap or dismissed it. By your criteria I wrote a constructive if critical piece.

In other words, you are still defending the indefensible (failure to patch security vulnerabilities, poor management) and accusing me of falling into some kind of trap because I didn't write what you wanted to see. Sorry, it doesn't work that way.

Anonymous said...

@Caitlyn:When I review Slackware, as an example, I am not writing to please Slackers, nor am I writing for people who know and run Slackware already. ... I am writing for the vast majority of people out there, Linux users included
I understand that. When I was reading reviews of Ubuntu about a year ago I belonged to the "vast majority". The reviews failed me, all I found, so I had to install Ubuntu and have a look. I guess the problem was that all the reviews were made like your Slackware 12.1 review. "I inserted the distribution CD and the following happened...". That was ideal for those who were going to insert a CD and see what happens. That was acceptable for those who wanted to know if just inserting a Ubuntu CD would automatically save them the MS tax. I wanted to know what Ubuntu was about, what fun all that Ubuntu community had, and how would it be 3 months after the install. If not providing me with that data was not "narrowing the focus", what was?
Since the "travel notes" style did not work for Ubuntu, how could a review of Slackware done without disclosing its design goals avoid community criticism? The "travel notes" part of your 12.1 review is perfect and the community criticism is mostly valid. A koan, right?

Also, I don't meditate about operating systems.
Good for you! I never proposed you should, but why is it wrong for a journalist to meditate on Slackware as a social phenomenon? None of my proposed questions were technical. One revelation I counted for was that there must be some reason for the longevity and integrity of that community. That should provoke deep interest in the above mentioned Slackware design goals. I understand that there is no document that presents all of them in one place, but the mere adoption of the fact that they exist should help a lot.

That makes an issue like a small official repository and dependence on third party repositories of questionable quality a HUGE issue. You say otherwise because you are NOT the target audience.
This cannot ever be true because no community distro can have a HUGE technical issue, provided they burned the right folder to the distribution media. Such distro can have HUGE issues like developers loosing focus or tensions reaching unsafe levels, but not technical ones.
The fact that Slackware has a relatively small official repository is a HUGE achievement. This allows to have a rock solid solution to everyday needs and a foundation for anything else with a limited traffic and without annoying dependencies tracking. The price is that Slackware is not very suitable for installing random programs out of curiosity. Are you sure the community will react negatively if you put it like that? Would it tell less or more to the audience, target or not?
Just in case, an example. If I do not care about Ruby, I use what is in the official Slackware repository. If I care a bit, I compile the latest release using a slackbuild (if there were no slackbuilds, I would gladly write one since it is the easiest way to compile; do not forget I am interested, so I have to compile repeatedly). If I care a lot, I compile from CVS and, possibly, submit patches. Slackware is balanced so that any option is easy, at a price that neither is as easy as possible.

You want me to pigeon hole and target a very narrow audience in order not to offend. Doing so would kind of defeat the purpose of writing for a wide audience, and that is what I am paid to do.
I do not want you to limit the scope. Au contraire, I want you to widen the scope and, whenever some feature that can be misinterpreted as a bug is discovered, explain why things are done that way and not the other and what benefits are obtained.
In general I completely disagree when you say fans will accept constructive criticism. That is completely contrary to my experience.
Let us write the Slackware 13.0 review together and see what happens.

Caitlyn Martin said...

I'm going to keep this short: The Slackware community is too tiny to be considered a "social phenomenon". The community criticism of my review revealed zealotry for the most part, not valid critique. You do NOT get to dictate my writing style or determine what I cover or how I cover it. You also do NOT get to co-write anything with me. I doubt my editor would be pleased if I did that.

Let's just agree to disagree -- about everything -- and accept that you won't like my reviews.

Anonymous said...

@Caitlyn: "Let's just agree to disagree". OK. Agreed. Thanks for your time.

Unknown said...

Let us write the Slackware 13.0 review together and see what happens.

Was that a pickup line?