[geeks] the virtualization project

[Shannon]

Awhile back I started on a project to convert my current server away
from Solaris. My Solaris machine is still running, as things turned out
more complicated and took longer than I expected. That and I got
interrupted by a nor'easter, a hurricane, and a hurricane-induced
shoulder injury.

There were several things that bothered me about Solaris. I don't like
the future with Oracle, I hate how programs, configuration, and data are
scattered all over the place in Solaris, and its always harder to maintain.

Linux packages are bloated, but at least they work and keeping updated
is easy. The big winner here by far is NetBSD: it has to be the cleanest
base and package system in existence.

The following is just notes about what I tried and what I thought about
them. Also a big goal was to convert to a virtual server setup to gain
the benefits of that. However, that turned out to not be as easy a
decision as I thought.

* NetBSD *

This is far and away the clear winner on package management, a clean
UNIX tree and base package, and very easy maintenance. Its SMP is nice
now and it just works. I'm tempted to just forget virtualization and go
back to NetBSD, since it is relatively easy to save whole server
configurations and deploy them fast. Unlike Solaris and to a lesser
extend Linux, NetBSD puts everything in a limited set of sane places.

One big issue with NetBSD is its filesystems are way behind the rest.
Nothing like ZFS or BTFS exists for it, and as time goes on that is more
and more important.

* Xen on NetBSD *

It took awhile to learn how to do this. The documentation is horrible.
However once you learn its not bad, and it works very well. However its
not perfect. Glaring showstoppers for me were no SMP support in either
Dom0 or DomU NetBSD. Boo, hiss. Also there is pretty much no reasonable
way to control CPU power, throttling on NetBSD Xen because the support
is currently scattered too much.

NetBSD 6 is supposed to fix this.

* Xen on Linux (CentOS) *

This works pretty well, and has more support and utilities than with
NetBSD by far, and has better support for CPU power and throttling in
the hypervisor. More filesystems to choose from as well.

However, the Linux systems are clearly moving toward KVM, so not sure if
Xen support will continue at the current pace or not.

Also, I just rather not use Linux. Most of the distributions are very
bloated, even for basic system administration utilities. Its not hard at
all to innocently install a system utility and find it pulling in half
of Gnome. I also hate the politics of GNU, and while you can mostly
ignore it, it always makes me twitch.

Still this does work well, and I've not had stability issues with the
more well made and tested distributions.

Not having ZFS is politically stupid of the Linux camp, but they do have
BRTFS which has a lot of the same features.

* ESX *

I tried ESX. It installs easily, but at that point my impression rapidly
dropped. You can't do much of anything on the actual ESX server console,
and the only well supported tools for managing the VMs is their horribly
ugly Windows only management console. They state emphatically they have
been changing it heavily to move away from any kind of command line
management.

A good thing to learn for getting a job since it is popular in business,
but I really hated how you manage it, and it seemed that in spite of the
GUI front-end, it took me longer to get VMs up and running than NetBSD
Xen's manual method.

I tried a few other GUIs that were supposed to work with it, but it
seemed like all of them were broken in some way, and what if I need to
manage without a GUI?

* Linux KVM *

Didn't look at it much, but it does appear to work, and people report
reliable use cases. It seems like it is not as well engineered as Xen
since it doesn't really use a hypervisor which can isolate bits from one
another, but it is a whole lot easier to work with. I guess there are
pros and cons of both approaches. It also cannot run paravirtualized
operating systems which can be a big negative.

Hardware virtualization is neat, but its also rather heavy. You can make
an OS faster and lighter if it is paravirtualized, while HVM requires
emulating an entire machine. I think I'd rather have Xen.

* Final thoughts *

So in the end, I'm still running Solaris 10. Each solution I looked at
for virtualization had some issues that made me stop and think about it.
NetBSD Xen works nicely but has no SMP support yet, and no really nice
filesystems.

Linux Xen is very well supported, but I really, really didn't want to
run Linux. However I might just yield on that so I can get work done.

Linux KVM looks interesting, but I prefer to run paravirtual guests if I
can so I'm learning toward Xen still.

I guess what I want is: Solaris with ZFS, but without the ugly directory
tree and packages... basically Solaris with ZFS, but with the neat
layout and package system of NetBSD, and first-tier Xen support.

That would be nice.

I could just run a NetBSD server and be done with it. The main reason
for using ZFS is for the data reliability on my server, and I just plain
like it.

In the end, I guess we should be happy there are so many good solutions,
but at the same time I wish feature sets were more uniform, OS like
Solaris would get modernized and cleaned up, and the politics that
clutter and confuse the Linux situation didn't exist.

Anyway... I'm done rambling. Truth is I was playing Ars Magica with a
friend and she had to go to work, so I'm bored out of my mind.

[demime 1.01d removed an attachment of type application/pgp-keys which had a name of 0x5936BC24.asc]