In 2006, I was project manager on a VMware implementation for a health care organization. We virtualized 200 servers in six weeks, after a planning phase of about 2 months. Out of that experience I wondered, "Did virtualization have anything to offer a smaller business?" So I set up a box at home and converted my home "data centre" into a virtualized data centre using VMware's Server product, which was the free product at the time.
After five years it's been an interesting experience and I've learned a lot. At the end of the day, I'm pretty convinced that the small business that has a few servers running in a closet in their office doesn't have a lot to gain from virtualizing within the "closet". (I'm still a big fan of virtualization in a medium or large organization.) I'm going to switch back to running a single server with all the basic services I run (backup, file share, DNS, DHCP, NTP) on a single server image.
I had one experience where the VM approach benefited me: As newer desktops and laptops came into the house, the version of the backup client installed on them by default was newer than the backup master on my backup server (I use Bacula). Rather than play around with installing and updating different versions of the backup client or master, I simply upgraded the backup master VM to a new version of Ubuntu and got the newer version of Bacula. I didn't have to worry about what other parts of my infrastructure I was going to affect by doing the upgrade.
The down side was that I spent a lot of time fooling around with VMware to make it work. Most kernel upgrades require a recompile of the VMware tools on each VM, which was a pain. I spent a fair bit of time working through an issue about timekeeping on the guests versus the VMware host that periodically caused my VMs to slow to a crawl.
Connecting to the web management interface and console plug-in always seemed to be a bit of a black art, and it got worse over time. At the moment, I still don't think modern versions of FireFox can connect to a running VM's console, so I have to keep an old version around when I need to do something with a VM's console (before ssh comes up).
My set-up wasn't very robust in the face of power failures. When the power went off, the VMs would leave their lock files behind. Then, when the power came back, the physical machine would restart but the VMs wouldn't. I would have to go in by hand and clean up the lock files. And often I wouldn't even know there'd been a power failure, so I'd waste a bit of time trying to figure out what was wrong. I should have had a UPS, but that wouldn't solve all the instances where something would crash leaving a lock file behind.
All in all, and even if I had automated some of that, the extra level of complexity didn't buy me anything. In fact, it cost me a lot of time.
Some of these problems would have been solved by using the ESX family of VMware products, but the license fees guarantee that the economics don't work for a small business.
I originally started out planning to give Xen a try, but it turned out not to work with the current (at the time) version of Ubuntu. Today I would try KVM. I played around with it a bit last year and it looked fine for a server VM platform. I needed better USB support, so I switched to VirtualBox. VirtualBox worked fine for me to run the Windows XP VM I used to need to run my accounting program, but it has the free version/enterprise version split that makes me uncomfortable for business use.
So my next home IT project will be to move everything back to a simpler, non-virtualized platform. I'll still keep virtualization around for my sandbox. It's been great to be able to spin up a VM to run, say, an instance of Drupal to test upgrades before rolling out to my web site, for example, or to try out Wordpress, or anything else I need to try.
My blog posts about interesting steps along the virtualization road are here.
Showing posts with label Virtualization. Show all posts
Showing posts with label Virtualization. Show all posts
Friday, 11 November 2011
Sunday, 18 July 2010
Can't Run VMWare Server 2 Management Interface
I filled the disk on my VMWare Server 2 host, which caused all sorts of grief. Part of the grief was that I couldn't get to the management interface at https://vmhost:8333/ui. I solved that problem by killing the VMWare hostd process (after freeing up some space on the disk):
- Look up the process ID: ps -ea | grep hostd
- Kill the process: sudo kill pid
- Remove the old lock file: sudo rm /var/run/vmware/vmware-hostd.PID
- Restart VMWare management: sudo /etc/init.d/vmware-mgmt restart
Tuesday, 8 April 2008
Copying VMs
I tried copying my tiny Ubuntu VM, and it ran, except eth0 wouldn't come up, and of course the host name was wrong.
To fix eth0, you have to update /etc/iftab with the new VMWare-generated MAC address for the Ethernet interface. I added a script to the base VM in /usr/local/sbin/changemac to make it easier:
sudo vi /usr/local/sbin/changemac
And add:
#!/bin/sh
mac=`ifconfig -a | grep "HWaddr" | cut -d " " -f 11`
echo "eth0 mac $mac arp 1" > /etc/iftab
Then do:
sudo chmod u+x /usr/local/sbin/changemac
Note that you're adding the script to the "template" VM, so you'll only have create the script once for each template you create, not each time you create a new VM.
Now you can copy the "template" VM. Make sure the "template" VM isn't running. Log in to the VMWare host, change to the directory where you have the VMs, and copy the VM:
cd /usr/local/vmware/Virtual\ Machines
sudo cp -R --preserve=permissions,owner old_VM_directory new_VM_directory
Now in the VMWare console:
If you forget to change the host name in /etc/dhcp3/dhcient.conf the first time around:
To fix eth0, you have to update /etc/iftab with the new VMWare-generated MAC address for the Ethernet interface. I added a script to the base VM in /usr/local/sbin/changemac to make it easier:
sudo vi /usr/local/sbin/changemac
And add:
#!/bin/sh
mac=`ifconfig -a | grep "HWaddr" | cut -d " " -f 11`
echo "eth0 mac $mac arp 1" > /etc/iftab
Then do:
sudo chmod u+x /usr/local/sbin/changemac
Note that you're adding the script to the "template" VM, so you'll only have create the script once for each template you create, not each time you create a new VM.
Now you can copy the "template" VM. Make sure the "template" VM isn't running. Log in to the VMWare host, change to the directory where you have the VMs, and copy the VM:
cd /usr/local/vmware/Virtual\ Machines
sudo cp -R --preserve=permissions,owner old_VM_directory new_VM_directory
Now in the VMWare console:
- Import the new VM and start it.
- Log in at the console and run /usr/local/sbin/changemac.
- Change /etc/hostname, /etc/dhcp3/dhclient.conf, and /etc/hosts to have the host name you want for the new machine.
- Reboot.
If you forget to change the host name in /etc/dhcp3/dhcient.conf the first time around:
- Change it
- Type sudo date and then enter your password. This is just to make sure that sudo isn't going to prompt you for passwords
- Type sudo ifdown eth0 && sudo ifup eth0
Monday, 7 April 2008
Firewall on the VM Quick Reference
Here's how to set up the firewall. Here's my /etc/iptables.rules:
*filter
:INPUT ACCEPT [273:55355]
:FORWARD ACCEPT [0:0]
:LOGNDROP - [0:0]
:OUTPUT ACCEPT [92376:20668252]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
# Accept SSH so we can manage the VM
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i lo -j ACCEPT
# Allow ping (Zenoss uses it to see if you're up).
-A INPUT -p icmp --icmp-type echo-request -j ACCEPT
# Allow SNMP.
-A INPUT -p udp -s 0/0 --sport 1024:65535 --dport 161:162 -j ACCEPT
# Silently block NetBIOS because we don't want to hear about Windows
-A INPUT -p udp --dport 137:139 -j DROP
-A INPUT -j LOGNDROP
# Drop and log the rest.
-A LOGNDROP -p tcp -m limit --limit 5/min -j LOG --log-prefix "Denied TCP: " --log-level 7
-A LOGNDROP -p udp -m limit --limit 5/min -j LOG --log-prefix "Denied UDP: " --log-level 7
-A LOGNDROP -p icmp -m limit --limit 5/min -j LOG --log-prefix "Denied ICMP: " --log-level 7
-A LOGNDROP -j DROP
COMMIT
More on this later.
*filter
:INPUT ACCEPT [273:55355]
:FORWARD ACCEPT [0:0]
:LOGNDROP - [0:0]
:OUTPUT ACCEPT [92376:20668252]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
# Accept SSH so we can manage the VM
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i lo -j ACCEPT
# Allow ping (Zenoss uses it to see if you're up).
-A INPUT -p icmp --icmp-type echo-request -j ACCEPT
# Allow SNMP.
-A INPUT -p udp -s 0/0 --sport 1024:65535 --dport 161:162 -j ACCEPT
# Silently block NetBIOS because we don't want to hear about Windows
-A INPUT -p udp --dport 137:139 -j DROP
-A INPUT -j LOGNDROP
# Drop and log the rest.
-A LOGNDROP -p tcp -m limit --limit 5/min -j LOG --log-prefix "Denied TCP: " --log-level 7
-A LOGNDROP -p udp -m limit --limit 5/min -j LOG --log-prefix "Denied UDP: " --log-level 7
-A LOGNDROP -p icmp -m limit --limit 5/min -j LOG --log-prefix "Denied ICMP: " --log-level 7
-A LOGNDROP -j DROP
COMMIT
ntp on the VM
Bringing up the firewall on the "template" VM, I noticed that I was getting more ntp traffic than I expected. I discovered that in my ignorance, I had set my local ntp server to broadcast, which I don't need. I commented the broadcast line, and everything's still working.
I also found a good post on ntp that answered one of my long-time questions: What should I look at to see if the ntp client was actually working. Do ntpq -p. On the resulting listing, "the delay and offset values should be non-zero and the jitter value should be under 100." (The post is Red Hat based, but the information specifically about ntp is distro-agnostic.)
I also found a good post on ntp that answered one of my long-time questions: What should I look at to see if the ntp client was actually working. Do ntpq -p. On the resulting listing, "the delay and offset values should be non-zero and the jitter value should be under 100." (The post is Red Hat based, but the information specifically about ntp is distro-agnostic.)
Sunday, 30 March 2008
SNMP on the VM
Setting up SNMP on a machine so it can be monitored by Zenoss seems to mess me up every time. This time the problem was the -i option of snmpconf. It's advertised to put the configuration file where the SNMP programs will find it, but it doesn't put it at the front of the list of paths where the programs look, at least not on Ubuntu 6.06.
The solution: don't use snmpconf -i. Run snmpconf to set the access. Make sure it matches what you've set up in Zenoss, particularly the version of SNMP and therefore the access model. When you're done, do sudo mv snmpd.conf /etc/snmp/.
The solution: don't use snmpconf -i. Run snmpconf to set the access. Make sure it matches what you've set up in Zenoss, particularly the version of SNMP and therefore the access model. When you're done, do sudo mv snmpd.conf /etc/snmp/.
Friday, 28 March 2008
SNMP
The basic VM needs to have SNMP running on it, because there's no point having a server if you're not monitoring it. I had Zenoss set up a year ago monitoring some of my computers, but I was getting "bad oid" messages on the new VM template I was setting up.
The solution: Zenoss had a default SNMP version of 1 for Linux systems. I had set up SNMP on the new VM for version 2c. In Zenoss 2.0 I navigated to /Devices/Server/Linux page and selected the zProperties tab, then scrolled down to zSnmpVer and set it to v2c.
The solution: Zenoss had a default SNMP version of 1 for Linux systems. I had set up SNMP on the new VM for version 2c. In Zenoss 2.0 I navigated to /Devices/Server/Linux page and selected the zProperties tab, then scrolled down to zSnmpVer and set it to v2c.
Tuesday, 25 March 2008
Basic Tiny VM Part 1
The basic tiny VM needs:
mount /dev/cdrom
sudo dd if=/dev/cdrom0 of=/usr/local/vmware/ISOs/Ubuntu-6.06.1.iso
The VMTools ISOs are in the /tmp/vmware-server-distrib/lib/isoimages:
sudo cp /tmp/vmware-server-distrib/lib/isoimages/*.iso /usr/local/vmware/ISOs
Install VMTools. Here are some good instructions.
sudo apt-get install ssh ntp-simple snmpd snmp
(snmp is the package that contains snmpconf, which you need to set up snmp, and snmpwalk, which is useful for debugging.)
Configure the ntp server. I've set up an ntp server in the DNS, so I set the "server" line in /etc/ntp.conf to the following:
server ntp
And then restart ntp:
/etc/init.d/ntp-server restart
Run snmpconf to set up SNMP. That's probably a whole post in itself.
I'll do the firewall later. I've ignored my family for too long tonight.
- Ubuntu 6.06.1 Server (the basic install, not LAMP)
- VMTools
- SNMP so you can monitor it (I'm using Zenoss)
- ssh so you can administer it
- ntp as a client so it keeps time. For now I'll sync to my existing ntp server
- basic firewall rules that allow the above
mount /dev/cdrom
sudo dd if=/dev/cdrom0 of=/usr/local/vmware/ISOs/Ubuntu-6.06.1.iso
The VMTools ISOs are in the /tmp/vmware-server-distrib/lib/isoimages:
sudo cp /tmp/vmware-server-distrib/lib/isoimages/*.iso /usr/local/vmware/ISOs
Install VMTools. Here are some good instructions.
sudo apt-get install ssh ntp-simple snmpd snmp
(snmp is the package that contains snmpconf, which you need to set up snmp, and snmpwalk, which is useful for debugging.)
Configure the ntp server. I've set up an ntp server in the DNS, so I set the "server" line in /etc/ntp.conf to the following:
server ntp
And then restart ntp:
/etc/init.d/ntp-server restart
Run snmpconf to set up SNMP. That's probably a whole post in itself.
I'll do the firewall later. I've ignored my family for too long tonight.
Can't Connect to Console of VMs
I had everything built and running VMWare Server. Good. So I copied all the VMs I'd built when I was running VMWare on my desktop over to the new server. I started a few, and they were running fine. I could connect to the Zenoss console on one of them, and could ping both. However, all I got was a black screen when I tried to look at the console of the VM using VMWare Console.
The VMWare documentation recommended using the version of VMWare Console program specific to the server you're running. I grumbled a bit and re-installed (which was actually quite easy), then tried viewing the console of my VMs again. I still got a black screen, but I also got an error message saying that the .vmx file had to have execute permission for the user running the VMWare Console. I checked the .vmx files and sure enough, because of the way I copied them everything had 0644 permissions.
So I cd'd to the directory where all the VM directories were and typed:
find . -name "*.vmx" -exec chmod u+x \{} \;
That worked because the user connecting with VMWare Console is the same one that owned all the files. You'll have to do something slightly different if that's not the case.
Now they work fine.
The VMWare documentation recommended using the version of VMWare Console program specific to the server you're running. I grumbled a bit and re-installed (which was actually quite easy), then tried viewing the console of my VMs again. I still got a black screen, but I also got an error message saying that the .vmx file had to have execute permission for the user running the VMWare Console. I checked the .vmx files and sure enough, because of the way I copied them everything had 0644 permissions.
So I cd'd to the directory where all the VM directories were and typed:
find . -name "*.vmx" -exec chmod u+x \{} \;
That worked because the user connecting with VMWare Console is the same one that owned all the files. You'll have to do something slightly different if that's not the case.
Now they work fine.
Monday, 24 March 2008
VMWare Server on Ubuntu 6.06.1
The install went smoothly. I created a user "vmware" and added it to the admin group. Then I had to:
sudo apt-get install xinetd
sudo apt-get install libx11-6 libx11-dev libxtst6 xlibs-dev
The last line was thanks to this post. Without it, it wouldn't validate my serial number (and I'm sure I would have run into other problems).
The only default I changed was to put my virtual machines in /usr/local/vmware/Virtual Machines, because /usr/local is the big partition I made for VMs.
sudo apt-get install xinetd
sudo apt-get install libx11-6 libx11-dev libxtst6 xlibs-dev
The last line was thanks to this post. Without it, it wouldn't validate my serial number (and I'm sure I would have run into other problems).
The only default I changed was to put my virtual machines in /usr/local/vmware/Virtual Machines, because /usr/local is the big partition I made for VMs.
Sunday, 23 March 2008
Virtualization So Far
As should be obvious from my recent posts, I've been trying to set up a host for virtual machines. I need to be able to try things out easily, and virtual machines are great for that. I'd also like to get rid of my old boxes that are running core network infrastructure. It's not so much that I want to get rid of them, but the risk of continuing to use them is a problem. I have an 11-year-old Macintosh Performa that's my DHCP and DNS server for my whole network. If it breaks, I'm scrambling to replace it unless I get something new built. Obviously if it runs on a computer with a 1 GB hard drive and 32 MB of memory, I should be able to run it on a VM.
Anyway, being cheap I wasn't sure I wanted to pay for VMWare. They have a free version of course, but XenSource's free version is a license-key upgrade, whereas VMWare Server to Virtual Infrastructure (AKA ESX) is a complete software upgrade. So I thought I'd try XenSource, especially since they seemed to be saying that they could run any OS if you bought a CPU with virtualization support.
So I carefully researched the chips I was looking for and bought a Dell SC440 with an Intel Xeon 3050. A low-price server but with the right parts, or so I thought.
The install of XenSource was easy, as was the install of XenCenter, the control program on Windows. Unfortunately, there was a problem with the shortcut to install XenCenter. I posted a question in the Xen community boards and got no help. I found the solution myself a few days later, but not after noticing that there was very, very little activity on the community boards. I wonder if anyone is using Xen, or at least is anyone using it without paying Citrix for support?
Also, it turns out you can't run anything you want as a VM. I tried to run Ubuntu Server 6.06.1 and it gets disk errors. This is a known problem, apparently. Okay, I know it's hard to support every Linux distro, but Ubuntu should be one you support. Look at the numbers.
Anyway, worse than not supporting Ubuntu is that the answer from Citrix seemed to be, "use one of our supported distros." They'll always be niche if that's their approach. The market for virtualization is the world of heterogenous data centres that need to shrink their power and A/C footprint. You're not going to get into that market unless you can run anything that an off-the-shelf PC can run. So, I decided to try VMWare.
Installing a 60-day evaluation copy of ESX 3i didn't work. Neither did installing an evaluation copy of ESX 3.5, but at least it told me that the network card wasn't supported. So I tried Ubuntu 6.06.1, and the network card wasn't supported there, either. Broadcom, what are you doing releasing a NIC that doesn't work with older drivers? I found how to get Ubuntu installed, and so I'll continue with installing the free version of VMWare Server. This is not what I wanted to be doing.
I guess the lesson is you really have to check the hardware compatibility list, but I didn't even know I was going to go this path. I'm interested in how many other problems I'm going to have.
Even though I'm not up with VMWare Server, I have to say that it's the preferable approach. At least you have an underlying OS you can work with, and my experience with VMWare elsewhere says it's going to run whatever I try to put on it. Too bad the thinner versions (ESX) don't work on my hardware.
Anyway, being cheap I wasn't sure I wanted to pay for VMWare. They have a free version of course, but XenSource's free version is a license-key upgrade, whereas VMWare Server to Virtual Infrastructure (AKA ESX) is a complete software upgrade. So I thought I'd try XenSource, especially since they seemed to be saying that they could run any OS if you bought a CPU with virtualization support.
So I carefully researched the chips I was looking for and bought a Dell SC440 with an Intel Xeon 3050. A low-price server but with the right parts, or so I thought.
The install of XenSource was easy, as was the install of XenCenter, the control program on Windows. Unfortunately, there was a problem with the shortcut to install XenCenter. I posted a question in the Xen community boards and got no help. I found the solution myself a few days later, but not after noticing that there was very, very little activity on the community boards. I wonder if anyone is using Xen, or at least is anyone using it without paying Citrix for support?
Also, it turns out you can't run anything you want as a VM. I tried to run Ubuntu Server 6.06.1 and it gets disk errors. This is a known problem, apparently. Okay, I know it's hard to support every Linux distro, but Ubuntu should be one you support. Look at the numbers.
Anyway, worse than not supporting Ubuntu is that the answer from Citrix seemed to be, "use one of our supported distros." They'll always be niche if that's their approach. The market for virtualization is the world of heterogenous data centres that need to shrink their power and A/C footprint. You're not going to get into that market unless you can run anything that an off-the-shelf PC can run. So, I decided to try VMWare.
Installing a 60-day evaluation copy of ESX 3i didn't work. Neither did installing an evaluation copy of ESX 3.5, but at least it told me that the network card wasn't supported. So I tried Ubuntu 6.06.1, and the network card wasn't supported there, either. Broadcom, what are you doing releasing a NIC that doesn't work with older drivers? I found how to get Ubuntu installed, and so I'll continue with installing the free version of VMWare Server. This is not what I wanted to be doing.
I guess the lesson is you really have to check the hardware compatibility list, but I didn't even know I was going to go this path. I'm interested in how many other problems I'm going to have.
Even though I'm not up with VMWare Server, I have to say that it's the preferable approach. At least you have an underlying OS you can work with, and my experience with VMWare elsewhere says it's going to run whatever I try to put on it. Too bad the thinner versions (ESX) don't work on my hardware.
Tuesday, 18 March 2008
Installing the Xen Guest Agent on an Ubuntu VM
To install the guest agent on an Ubuntu VM, I tried:
If you just type sudo /mnt/Linux/Install.sh, it tells you you're running an unsupported distribution. I thought I'd be clever and try the force it to use the Debian 3 tools:
- Start the Ubuntu VM if it isn't already running
- In XenCenter, select the VM in the left panel, and select the Console tab in the right panel
- Select "xs-tools.iso" from the drop down list just above the console window
- Click on the console window, log in if necessary, and type sudo mount /dev/hdd /mnt
If you just type sudo /mnt/Linux/Install.sh, it tells you you're running an unsupported distribution. I thought I'd be clever and try the force it to use the Debian 3 tools:
- Type: sudo /mnt/Linux/install.sh -d debian -m 3
- Reboot: sudo shutdown -r now
Monday, 17 March 2008
Installing Ubuntu on Xen
I was able to install Ubuntu 6.06.1 on Xen by putting the Ubuntu install CD in the CD drive of the Xen server. I got lost for a while because I didn't create the storage when initially setting up the VM. Once I actually made a disk, the install went okay. I've found a post saying that Ubuntu will start having errors and switch the disk to read-only, so I'll watch for that.
I still need to install the Xen tools...
I still need to install the Xen tools...
Sunday, 16 March 2008
Ubuntu on Xen
I wanted to continue to use Ubuntu Server in my VMs. I like their approach to long-term support, the server install, and the way they provide a pre-configured LAMP server.
As I read about how to build one, I stumbled across a post from someone else who was doing what I wanted to do, and had problems. That scared me off just blindly stumbling ahead, so now I have more research to do.
I'm pretty sure it should be relatively easy to have a kernel that works on top of Xen, and on which I can run an otherwise standard Ubuntu 6.06 install, but it's going to take some digging... This is harder than I wanted it to be.
As I read about how to build one, I stumbled across a post from someone else who was doing what I wanted to do, and had problems. That scared me off just blindly stumbling ahead, so now I have more research to do.
I'm pretty sure it should be relatively easy to have a kernel that works on top of Xen, and on which I can run an otherwise standard Ubuntu 6.06 install, but it's going to take some digging... This is harder than I wanted it to be.
First Guest on Xen
I created a Debian 3.1 VM from the templates using XenCenter. I just followed the XenCenter wizard. It was trivial, although before I created it I spent almost 1 1/2 hours reading documentation and the Internet.
The only thing I had to do was set the time zone. Being old fashioned, I used the command line console and tzconfig(1).
The only thing I had to do was set the time zone. Being old fashioned, I used the command line console and tzconfig(1).
Saturday, 15 March 2008
XenServer Install
I bought a Dell SC440 for Jade Systems (that's me). I'm setting it up as a virtual host so I can create virtual computers at will. I've decided to try XenServer, I think mostly because the free version is just a license-key upgrade to the full version. Besides, I already have some experience with VMWare, albeit second hand.
I had to F2 into the BIOS on initial boot of the SC440 to turn on virtualization in the CPU. It was relatively easy to find on the menus, but not right at the top level (My apologies for the vagueness here. The servers are in another room, so I wasn't blogging while I was installing).
The install of XenServer 4.0.1 went exactly according to the instructions. I read through the four pages of documentation first, and so I had all the answers I needed -- the usual network set-up, which time servers to use, etc.
I just finished installing the security patch. First challenge: actually getting the file to the XenServer host. I put a DVD in the DVD drive, but there's no entry for CD or DVD in /etc/fstab. The installation instructions recommend using a USB key. I flailed on that for a while, Googling for Linux-for-dummies level help on USB, before I said to myself, "Right. I guess I still have to be a Linux sysadmin," and found the appropriate dev to mount the dvd. From there it was maybe a minute to install the patch.
Total time: About half an hour to install XenServer the other night, mostly unattended after the usual initial Linux questions, plus 45 minutes now, mostly Googling.
I had to F2 into the BIOS on initial boot of the SC440 to turn on virtualization in the CPU. It was relatively easy to find on the menus, but not right at the top level (My apologies for the vagueness here. The servers are in another room, so I wasn't blogging while I was installing).
The install of XenServer 4.0.1 went exactly according to the instructions. I read through the four pages of documentation first, and so I had all the answers I needed -- the usual network set-up, which time servers to use, etc.
I just finished installing the security patch. First challenge: actually getting the file to the XenServer host. I put a DVD in the DVD drive, but there's no entry for CD or DVD in /etc/fstab. The installation instructions recommend using a USB key. I flailed on that for a while, Googling for Linux-for-dummies level help on USB, before I said to myself, "Right. I guess I still have to be a Linux sysadmin," and found the appropriate dev to mount the dvd. From there it was maybe a minute to install the patch.
Total time: About half an hour to install XenServer the other night, mostly unattended after the usual initial Linux questions, plus 45 minutes now, mostly Googling.
Friday, 31 August 2007
iSCSI vs. Fibre Channel - You're Both Right
Reading another article from an expert who provides less than useful information has finally prompted me to try to provide useful guidance for IT managers of 50 to 1,000 diverse servers running a variety of applications.
iSCSI vs. fibre channel (FC) is a classic technology debate with two camps bombarding each other mercilessly with claims that one or the other is right. The reason the debate is so heated and long lived is because there isn't a right answer: there are different situations in which each one is better than the other. Here's how to figure out what's best for you:
Start with the assumption that you'll use iSCSI. It's less expensive, so if it does what you need, it should be your choice. It's less expensive at all levels: The switches and cables enjoy the ecnomy of scale of the massive market for IP networking. You already have staff who know how to manage IP networks. You already have a stock of Cat 6 cables hanging in your server rooms or network closets.
If you have mostly commodity servers, they transfer data to and from direct-attached storage at less than gigabit speeds. Gigabit iSCSI is fine. If you have a lot of servers, you have to size the switches correctly, but you have to do that with FC as well, and the FC switch will be more expensive. Implement jumbo frames so backups go quickly.
Just because you're using iSCSI doesn't mean you're running your storage network over the same cables and switches as your data centre LAN. In fact, you probably aren't. The cost saving doesn't come from sharing the existing LAN, it comes from the lower cost per port and the reduced people cost (skill sets, training, availability of administrators in the labour market) of using the same technology. As long as your storage and general-purpose networks are not sharing the same physical network, a lot of the criticisms of iSCSI evaporate.
If you have large, specialized servers that can and do need to sustain high data transfer rates, then definitely look at FC. Be sure you're measuring (not just guessing) that you need the data transfer rates.
If you have a large farm of physical servers running a huge number of virtual machines (VMs), look at FC. My experience is that virtual machine infrastructures tend to be limited by RAM on the physical servers, but your environment may be different. You may especially want to think about how you back up your VMs. You may not need the FC performance during the day, but when backups start, watch out. It's often the only time of day when your IT infrastructure actually breaks a sweat.
You might look at a FC network between your backup media servers and backup devices, especially if you already have an FC network for one of the reasons above.
Yes, FC will give you higher data transfer rates, but only if your servers and storage devices can handle it, and few today go much beyond one gigabit. FC will guarantee low latency so your servers won't do their equivalent of "Device not ready, Abort, Retry, Ignore?"
The challenge for an IT manager, even (or especially) those like me who have a strong technical background, is that it's easy to get talked into spending too much money because you might need the performance or low latency. The problem with that thinking is that you spend too much money on your storage network, and you don't have the money left over to, for example, mirror your storage, which may be far more valuable to your business.
A final warning: neither technology is as easy to deal with as the vendor would have you believe (no really?). Both will give you headaches for some reason along the way. If it wasn't hard, we wouldn't get the big bucks, would we?
iSCSI vs. fibre channel (FC) is a classic technology debate with two camps bombarding each other mercilessly with claims that one or the other is right. The reason the debate is so heated and long lived is because there isn't a right answer: there are different situations in which each one is better than the other. Here's how to figure out what's best for you:
Start with the assumption that you'll use iSCSI. It's less expensive, so if it does what you need, it should be your choice. It's less expensive at all levels: The switches and cables enjoy the ecnomy of scale of the massive market for IP networking. You already have staff who know how to manage IP networks. You already have a stock of Cat 6 cables hanging in your server rooms or network closets.
If you have mostly commodity servers, they transfer data to and from direct-attached storage at less than gigabit speeds. Gigabit iSCSI is fine. If you have a lot of servers, you have to size the switches correctly, but you have to do that with FC as well, and the FC switch will be more expensive. Implement jumbo frames so backups go quickly.
Just because you're using iSCSI doesn't mean you're running your storage network over the same cables and switches as your data centre LAN. In fact, you probably aren't. The cost saving doesn't come from sharing the existing LAN, it comes from the lower cost per port and the reduced people cost (skill sets, training, availability of administrators in the labour market) of using the same technology. As long as your storage and general-purpose networks are not sharing the same physical network, a lot of the criticisms of iSCSI evaporate.
If you have large, specialized servers that can and do need to sustain high data transfer rates, then definitely look at FC. Be sure you're measuring (not just guessing) that you need the data transfer rates.
If you have a large farm of physical servers running a huge number of virtual machines (VMs), look at FC. My experience is that virtual machine infrastructures tend to be limited by RAM on the physical servers, but your environment may be different. You may especially want to think about how you back up your VMs. You may not need the FC performance during the day, but when backups start, watch out. It's often the only time of day when your IT infrastructure actually breaks a sweat.
You might look at a FC network between your backup media servers and backup devices, especially if you already have an FC network for one of the reasons above.
Yes, FC will give you higher data transfer rates, but only if your servers and storage devices can handle it, and few today go much beyond one gigabit. FC will guarantee low latency so your servers won't do their equivalent of "Device not ready, Abort, Retry, Ignore?"
The challenge for an IT manager, even (or especially) those like me who have a strong technical background, is that it's easy to get talked into spending too much money because you might need the performance or low latency. The problem with that thinking is that you spend too much money on your storage network, and you don't have the money left over to, for example, mirror your storage, which may be far more valuable to your business.
A final warning: neither technology is as easy to deal with as the vendor would have you believe (no really?). Both will give you headaches for some reason along the way. If it wasn't hard, we wouldn't get the big bucks, would we?
Friday, 27 April 2007
Virtualization: There's Gotta be a Catch
Virtualization solves lots of problems for many if not most organizations that have more than a rack of servers. On an earlier assignment I calculated a worst-case saving of C$380 per month for a virtual server over a physical server (using ESX 2.5 and 3.0 from VMWare). But there's a catch to virtualization, and that catch is backups.
Virtualization introduces wrinkles on your backup approach. Fortunately, to start off you're probably okay doing your backups the same way you always have been. The backup wrinkles are not enough to stop you from embarking on virtualization.
Here are some of the things you need to watch for as you add virtual machines (VMs) to your virtualization platform:
As you add VMs to your infrastructure, you may run into decreasing backup performance. The reason: many servers today are at their busiest during their backup. You may be able to run 20 VMs comfortably on one physical server, but if you try to back up all those VMs at once you'll run into bottlenecks because the physical server has a certain number of network interfaces, and all the data is coming from the same storage device, or at least through the same storage interface. Again, the solution is to watch backup performance as you virtualize and make adjustments.
Be aware that you might have to make changes to your backup infrastructure to deal with real challenges of backup performance introduced by virtualization. If your backups are already a problem, you might want to look into this in more detail. (The problems and solutions are beyond the scope of this post.)
How long do you have to rebuild servers after a data centre fire? You may not have even thought of the problem (don't be embarrassed. Many of us haven't). With virtualization you have to think about it because an equivalent event is more likely to happen: The storage device that holds all your VMs may lose its data, and you're faced with rebuilding all your VMs. I have second-hand experience (e.g. the guys down the street) with storage devices eating all the VMs, but I've never directly known anyone who had a serious fire in the data centre.
If the backups on your physical servers can be restored to bare metal, then you don't have to worry about your storage device eating the VMs. You may have to make some changes to your bare-metal backup -- I have no experience with that topic so I don't know for sure -- but once you do you should be able to restore your VMs relatively quickly.
If you can't or don't have backups that can be restored to bare metal, then you have a challenge. I doubt that most general purpose data centres are full of identically configured servers, with detailed rebuild procedures and air-tight configuration management so every server can be rebuilt exactly like the one that was running before. If you had to rebuild 200 VMs from installation disks, you'd probably be working a lot of long nights.
If most of your servers that you plan to virtualize have database-like data (large files that change every day), I'd recommend looking at changing your backup approach for those servers to a product like ESX Ranger, or look for some of the user-built solutions on the Internet. These products will back up the entire virtual machine every time they run, and may not allow individual file (within the VM) restores. However, for a database server you're probably backing up the whole server every night anyway, so that won't be a significant change to your backup workload.
If you want to virtualize file server-like servers, there isn't really a good solution that I or anyone I know has found at this time. If your backup infrastructure has enough room to take the additional load, simply back up with ESX Ranger or one of the other solutions once a week (or however frequently you do a full backup), along with your current full and incremental backup schedule. If you have to rebuild the VM, you restore the most recent ESX Ranger backup first. If you just have to restore files on the server, because a user deleted an important document, for example, just use the regular backups.
If you have the budget to change backup infrastructures, ESX Ranger can probably provide a pretty good overall solution. However, you have to provide backup and restore for physical servers as well, so the staff who do restores have to be able to deal with two backup systems.
One final gotcha that I've run across: There are some great devices out there from companies like Data Domain that provide excellent compression of exactly the type of data you're backing up when you back up an entire VM. Unfortunately, ESX Ranger compresses the data too, which messes up the storage device's compression. Whatever solution you put together, make sure your vendor commits to performance targets based on the entire solution, not on individual products.
As with so much of what we do in IT, it's really hard to summarize everything in a way that makes sense in a blog post. Comment on this post if you'd like more details or reasons why I make the recommendations I make.
Virtualization introduces wrinkles on your backup approach. Fortunately, to start off you're probably okay doing your backups the same way you always have been. The backup wrinkles are not enough to stop you from embarking on virtualization.
Here are some of the things you need to watch for as you add virtual machines (VMs) to your virtualization platform:
- Do you have highly-tuned start and stop times for your backup jobs, for example when you have inter-dependencies between external events and your backup jobs?
- Do the servers you plan to virtualize have file server-like data, in other words, does it consist of a lot of small files that mostly don't change?
- If you had a fire in your data centre today, before virtualizing, how soon would you have to have all the servers rebuilt?
- Is your backup infrastructure really only being used to half capacity or less?
As you add VMs to your infrastructure, you may run into decreasing backup performance. The reason: many servers today are at their busiest during their backup. You may be able to run 20 VMs comfortably on one physical server, but if you try to back up all those VMs at once you'll run into bottlenecks because the physical server has a certain number of network interfaces, and all the data is coming from the same storage device, or at least through the same storage interface. Again, the solution is to watch backup performance as you virtualize and make adjustments.
Be aware that you might have to make changes to your backup infrastructure to deal with real challenges of backup performance introduced by virtualization. If your backups are already a problem, you might want to look into this in more detail. (The problems and solutions are beyond the scope of this post.)
How long do you have to rebuild servers after a data centre fire? You may not have even thought of the problem (don't be embarrassed. Many of us haven't). With virtualization you have to think about it because an equivalent event is more likely to happen: The storage device that holds all your VMs may lose its data, and you're faced with rebuilding all your VMs. I have second-hand experience (e.g. the guys down the street) with storage devices eating all the VMs, but I've never directly known anyone who had a serious fire in the data centre.
If the backups on your physical servers can be restored to bare metal, then you don't have to worry about your storage device eating the VMs. You may have to make some changes to your bare-metal backup -- I have no experience with that topic so I don't know for sure -- but once you do you should be able to restore your VMs relatively quickly.
If you can't or don't have backups that can be restored to bare metal, then you have a challenge. I doubt that most general purpose data centres are full of identically configured servers, with detailed rebuild procedures and air-tight configuration management so every server can be rebuilt exactly like the one that was running before. If you had to rebuild 200 VMs from installation disks, you'd probably be working a lot of long nights.
If most of your servers that you plan to virtualize have database-like data (large files that change every day), I'd recommend looking at changing your backup approach for those servers to a product like ESX Ranger, or look for some of the user-built solutions on the Internet. These products will back up the entire virtual machine every time they run, and may not allow individual file (within the VM) restores. However, for a database server you're probably backing up the whole server every night anyway, so that won't be a significant change to your backup workload.
If you want to virtualize file server-like servers, there isn't really a good solution that I or anyone I know has found at this time. If your backup infrastructure has enough room to take the additional load, simply back up with ESX Ranger or one of the other solutions once a week (or however frequently you do a full backup), along with your current full and incremental backup schedule. If you have to rebuild the VM, you restore the most recent ESX Ranger backup first. If you just have to restore files on the server, because a user deleted an important document, for example, just use the regular backups.
If you have the budget to change backup infrastructures, ESX Ranger can probably provide a pretty good overall solution. However, you have to provide backup and restore for physical servers as well, so the staff who do restores have to be able to deal with two backup systems.
One final gotcha that I've run across: There are some great devices out there from companies like Data Domain that provide excellent compression of exactly the type of data you're backing up when you back up an entire VM. Unfortunately, ESX Ranger compresses the data too, which messes up the storage device's compression. Whatever solution you put together, make sure your vendor commits to performance targets based on the entire solution, not on individual products.
As with so much of what we do in IT, it's really hard to summarize everything in a way that makes sense in a blog post. Comment on this post if you'd like more details or reasons why I make the recommendations I make.
Subscribe to:
Posts (Atom)