Is This the Beginning of the End for Microsoft...

...or... The end of the... or whatever.

Here you can read about Microsoft explaining their Health Vault product (sorry, I think you might need to register). "We're coining a new term, the Unified Intelligence system." I think experience in IT shows that if you have to "coin a new term" you're selling something that no one wants. Is Tim Bray right, that the options to Microsoft are here "right now" and we could be seeing the start of a big change?

NetBeans Out of Memory Updating Ruby Gems

I got a message "Exception in thread "main" java.lang.OutOfMemoryError: Java heap space" while updating Ruby Gems in NetBeans 6.0.1 on Windows XP. It took a little longer than usual for me to find reports of this problem using Google, but when I did I found that it is a known problem.

Someone suggested changing the command line arguments to NetBeans to increase the size of the memory for the JVM, but that didn't work for me. Instead, I figured out how to load gems from the command line, and once I did that one time, I was able to use the gem manager in NetBeans.

So, in more detail: Originally I had installed NetBeans from a privileged account using all the default options. To get new gems, in a non-privileged account, I had to create two environment variables with the following values:

set JAVA_HOME=C:\Program Files\Java\jdk1.6.0_05
set JRUBY_BASE=C:\Program Files\NetBeans 6.0.1\ruby1\jruby-1.0.2

Your values may differ, particularly the version numbers. I wrote it here as you'd do it in the command prompt window, but I actually did it using the System control panel.

I opened a comment prompt window and did:

%JRUBY_BASE%\bin\gem install login_generator

After a couple of minutes, everything seemed to end normally. I did Tools->Ruby Gems from a running NetBeans instance and didn't get the desired results, so I restarted NetBeans and then Tools->Ruby Gems got me a list of gems, including login_generator as an installed gem.

MySQL From Remote Host

I couldn't get MySQL Administrator on a Windows XP desktop to connect to a MySQL instance I had running on an Ubuntu 6.06 server that I built as a LAMP server. I was getting:

Could not connect to the specified instance.

MySQL Error Number 2003
Can't connect to MySQL server on 'server' (10061)

I had to edit /etc/my.cnf (or /etc/mysql/my.cnf depending on where yours is stored) on the Ubuntu server to comment out the "bind-address" line, then restart the server. I also had to add a non-root user with all privileges. In fact, you have to add two users as described here:

use mysql
grant all privileges on *.* to 'user'@'localhost' identified by 'password' with grant option;
grant all privileges on *.* to 'user'@'%' identified by 'password' with grant option;
flush privileges;

There are some posts that show how to enable remote logins by the MySQL "root" user, but I prefer not to do it that way.

Tape Rotation with Bacula

I love the topic of backups. I say that because it's IT's dirty secret. No one should keep data in one place only, yet it's very difficult to set up a backup solution. Different organizations have different needs, and so backup software has to provide a lot of options. But the need for options means when you just want to get basic backup running quickly, it's a challenge.

This post is part of a series about rolling your own backup solution. There are other ways to do it, but I wanted to do my own solution one more time...

I'm backing up a Windows XP desktop and a Windows XP laptop, a Dell SC440 which is the VMWare host, plus a number of Linux VMs that provide my basic infrastructure: DNS, DHCP, file server, Subversion server, test platforms for software development, and the backup server itself.

I chose tape in part because I can take the backup off-site. I'll take a tape off-site once a week. That means I might lose a week's worth of work if my house burns down, but I'm not ready to invest in the time and effort to swap tapes every day, either.

The Bacula documentation has a good section on backup strategies, but none of them include mine. I'll have to figure it out myself.

Bacula manages tapes in a tape pool. A pool is just a group of tapes. (Bacula calls tapes "volumes".) I want to let Bacula fill up one tape per week before it uses another, which is the default behaviour. At the end of the week, I want to eject the tape and use another. I'll let Bacula automatically recycle the tapes, meaning that after a week (in my case), Bacula will reuse a tape, overwriting the old backups on it.

Anyway, I started with a rotation to do a full backup Sunday night, incremental backups all week, and then eject the tape Saturday night after the last incremental. With three tapes I would always have last week's tape off site, except on Sunday.

I really only got started when I realized that that's a lot of tape wear given that the off-site happens once a week and that I have a fair bit of disk space on my main server. So my next idea is:

Take a full backup Monday night to disk, and incrementals up to Sunday night. Then, Monday morning write the whole disk volume to tape and take it off-site. That way I only run the tape once a week, and hopefully in a scenario that minimizes the chance of shoe-shining. I'll write the data to disk without compression, and let hardware compression compress the data to tape.

This also has the nice property that last week's backups are also on the disk (if I have enough disk space), so if I need a file I can get it from disk rather than retrieving the tape.

Securing DNS/bind/named

This is another late posting of some notes when I built some new infrastructure servers on VMs to replace my aging PowerPC Macs that ran my network.

The security info I got when my ISP told me I had a badly configured name server requires that you create a /var/named directory:

sudo mkdir /var/named 
sudo chgrp bind /var/named 
sudo chmod 770 /var/named 
sudo chmod g+s /var/named 
sudo mkdir /var/log/named 
sudo chmod 770 /var/log/named 
sudo chmod g+s /var/log/named

Building A DHCP/DNS Server

Months ago I built a DHCP/DNS server from scratch. Most of these notes I made at the time I was building it, meaning to fix them up within a day or two and post them. Of course, I kept doing other things before finishing the documentation, so here are my rather raw notes. This was for Ubuntu 6.06 running on VMWare Server.

1. Create a new VM with a 2 GB disk, don't preallocate and make sure all disks are less than 2 GB. Only give it 64 MB of RAM
2. Attach the Ubuntu .iso to the CD and start the VM
3. Build with the options you want
4. Do the following:
   



sudo apt-get update
sudo apt-get upgrade
sudo apt-get install ssh ntp-simple snmpd snmp bacula-client build-essential linux-headers-$(uname -r)



5. Install VMTools

cd /tmp
sudo mount /cdrom
sudo tar xvfz /cdrom/VMwareTools-1.0.0-27828.tar.gz
cd vmware-tools-distrib
sudo ./vmware-install.pl



6. Edit /etc/dhcp3/dhclient.conf to send host-name "netres01";
   
7. Restart the network to get into DNS and DHCP (if you already have one)
   
8. Install DHCP and DNS and stop the services:

sudo apt-get install dhcp3-server bind9 sudo
/etc/init.d/bind9 stop sudo /etc/init.d/dhcp3 stop



9. Since this is a DNS server, I'll allow it to use a fixed IP address. Edit /etc/network/interfaces. Edit the forward and reverse zone files.
   

auto eth0
iface eth0 inet static
address 10.3.3.2
netmask 255.255.255.0
network 10.3.3.0
broadcast 10.3.3.255
gateway 10.3.3.3
pre-up iptables-restore < /etc/iptables.rules post-down iptables-save -c > /etc/iptables.rules

You have to kill the existing dhclient process because ifdown/ifup doesn't (it wouldn't know how, really).

Change the key for the DNS server before starting it, or you'll have to manually look up the pids and kill the named processes. rndc stops working because the key has changed since named started.

If you had a name server from DNS before, it will still be in /etc/resolv.conf.

The biggest thing is to get the permissions right on the /etc/bind directories and files.

nsupdate

Set up the new DNS first and get it working.

Challenge # 42 of Healthcare IT

Many who've worked in healthcare IT believe it's more difficult than IT in other contexts. Everyone has their reasons. I'd like to add mine here.

Mistakes in healthcare are really bad. They literally lead to people's health being compromised, or in the worst case, people dying. Projects are about doing something new. Doing something new is about making mistakes and learning from them, or at least trying out new ideas, some of which will turn out to be wrong.

Sometimes these two things are in direct contradiction. More often it leads to all sorts of misunderstandings between the healthcare team and the external project team that are hard for either side to recognize, let alone overcome.

For example, it's pretty standard practice on a project to do a design and put it in front of a group of people for review. While it can be hard to listen to others criticize your design after all the work you've done on it, we all get used to it.

Now imagine you're a nurse, doctor or pharmacist. All your life you've been terrified of making a mistake because someone might die because of it. Everyone around you is also terrified of making a mistake, and in fact the best way for them to feel good is to catch you making a mistake. It's pretty easy to fall into a pattern of avoiding mistakes at all costs, avoiding blame for mistakes when they do occur, and catching others' mistakes in order to appear to be a better nurse, doctor or pharmacist than the others.

You're not likely even to be able to understand a consultant who suggest you put up a proposed design and let others criticize it. And if you understand, you're not likely to want to go along with it. Every fibre in your being is about avoiding mistakes. And everyone you work with considers making a mistake to be the worst thing anyone can do. No consultant is going to convince you that you should publicly set yourself up to "make a mistake".

If you're running a project in a healthcare environment, you need to understand the depth of fear of making mistakes. To move the project forward in spite of this fear, try some of these ideas:

• Let the people you're working with tell you what makes them comfortable. They won't necessarily tell you just because you ask them. You have to listen to how they want to do the project
• Bring groups together and facilitate group decision making, rather than expecting one person to tell you an answer. It will take longer than if you could find one person to make the decision, but the reality is, you aren't going to find that one person
• Use project staff if you can. Just let them know they're going to take a beating. The passion with which many people expose other people's mistakes in healthcare is unnerving
  

By the way, I'm really glad that healthcare providers have a phobia about mistakes. If I'm ever in the hospital I want to know that everyone there is doing everything they can to avoid mistakes. It's only difficult when you're trying to run a project.

Bacula Catalog Job and MySQL

To make the Bacula catalog job work:

1. Edit /etc/bacula/bacula-dir.conf on the backup server
2. Change where it says -u -p to -u bacula
3. Edit ~bacula/.my.cnf and put this:

[client]
password=your_secret_password

4. chmod 400 ~bacula/.my.cnf ; chown bacula:bacula ~bacula/.my.cnf

Now it should work. Don't forget to do "reload" in bconsole.

Bacula Notes

The Bacula documentation is good, but given the complex and interdependent nature of the backup problem, it's pretty overwhelming at first.

One thing that's not immediately obvious is where the configuration files are. The bacula-fd.conf file for Windows XP clients is at C:\Documents and Settings\All Users\bacula\bacula-fd.conf. On Ubuntu using the packages installed from universe, the configuration files are in /etc/bacula.

If you get errors that the server can't connect to the client, make sure the director definition in the client's bacula-fd.conf allows the director to connect, and that the client's password matches the server's password in the client resource of /etc/bacula/bacula-dir.conf. There's a helpful picture of what you need to do in the Bacula documentation.

Accessing a SCSI Tape Drive from a VM

I ordered my Dell SC440 with an internal DAT tape drive. lsscsi reports it as a Seagate DAT72-052. I'm pretty sure that the Ubuntu 6.06 installation picked it up automatically -- I flailed around a bit to get this working but I don't think at the end of the day that I did anything on the host to get the tape drive working.

I'm creating a VM to run my backup. For large installations you won't want to do this, but for me I see no reason not to. And a big part of the reason I'm doing this is to see what's possible, so onward.

To enable the tape on a VM, you have to shut down the VM. Then, in the VMWare Console select VM > Settings > Hardware > Generic SCSI, and specify the physical device to connect to. In my case it was /dev/sg0. You also have to specify the controller and target for the tape drive.

I had no idea what the controller and target were, so on the VMWare host, I did:

sudo apt-get install lsscsi
lsscsi -c

and got:

Attached devices:
Host: scsi1 Channel: 00 Target: 06 Lun: 00
Vendor: SEAGATE Model: DAT DAT72-052 Rev: A16E
Type: Sequential-Access ANSI SCSI revision: 03
Host: scsi2 Channel: 00 Target: 00 Lun: 00
Vendor: ATA Model: WDC WD1600YS-18S Rev: 20.0
Type: Direct-Access ANSI SCSI revision: 05

I took the channel as the controller: 0, and the target: 6. I entered all that into the VMWare Console and clicked enough okays to get out of the configuration. (I couldn't find the link in VMWare's on-line documentation for configuring generic SCSI devices, but if you type "SCSI" in the "Index" tab of the VMWare Console's help window you can find slightly more detailed instructions.)

When I started the VM, I got a message that said, among other things: "Insufficient permissions to access the file." Since it looked like everything else was right, I did ls -l /dev/sg0 on the VMWare host (not the VM) and got:

crw-rw---- 1 root tape 21, 0 2008-03-23 17:23 /dev/sg0

Since VMWare was running as user vmware, I added the vmware user to the tape group:

sudo adduser vmware tape

Then I restarted the VM and it worked fine. It pays to read the error message closely.

Another SCSI Package

This is useful:

sudo apt-get install lsscsi

It shows you what SCSI devices you have attached to a machine and some important values.

Installing Bacula

To install bacula with MySQL (after you do this):

sudo apt-get install mysql-server bacula-director-mysql

Then you have to set up exim4, the mail system. Choose:

mail sent by smarthost; no local mail

After you install the MySQL version of the bacula director, you can install the rest of bacula this way, and also install some recommended packages:

sudo apt-get install bacula
sudo apt-get install dds2tar scsitools sg3-utils

I had these notes from an earlier set-up of exim4:

Look into setting up /etc/aliases later to redirect mail to more useful places. Also, make sure the domain of the outgoing address is one known to the outside world (e.g. jadesystems.ca) or the SMTP server will probably reject the message.

Bacula: Backups

To install bacula on Ubuntu, you need to add the universe repositories to /etc/apt/sources.list. It's just a matter of uncommenting four lines:

deb http://ca.archive.ubuntu.com/ubuntu/ dapper universe 
deb-src http://ca.archive.ubuntu.com/ubuntu/ dapper universe 
... 
deb http://security.ubuntu.com/ubuntu dapper-security universe 
deb-src http://security.ubuntu.com/ubuntu dapper-security universe

Then:

sudo apt-get update

The standard install of bacula uses sqllite, which the bacula guy reports as having problems...

Copying VMs

I tried copying my tiny Ubuntu VM, and it ran, except eth0 wouldn't come up, and of course the host name was wrong.

To fix eth0, you have to update /etc/iftab with the new VMWare-generated MAC address for the Ethernet interface. I added a script to the base VM in /usr/local/sbin/changemac to make it easier:

sudo vi /usr/local/sbin/changemac

And add:

#!/bin/sh
mac=`ifconfig -a | grep "HWaddr" | cut -d " " -f 11`
echo "eth0 mac $mac arp 1" > /etc/iftab

Then do:

sudo chmod u+x /usr/local/sbin/changemac

Note that you're adding the script to the "template" VM, so you'll only have create the script once for each template you create, not each time you create a new VM.

Now you can copy the "template" VM. Make sure the "template" VM isn't running. Log in to the VMWare host, change to the directory where you have the VMs, and copy the VM:

cd /usr/local/vmware/Virtual\ Machines
sudo cp -R --preserve=permissions,owner old_VM_directory new_VM_directory

Now in the VMWare console:

1. Import the new VM and start it.
   
2. Log in at the console and run /usr/local/sbin/changemac.
   
3. Change /etc/hostname, /etc/dhcp3/dhclient.conf, and /etc/hosts to have the host name you want for the new machine.
   
4. Reboot.

I'm sure you should be able to do this without a reboot, but I don't know which startup scripts do what needs to be done. Also, I had some problem with sudo not working after changing /etc/hosts.

If you forget to change the host name in /etc/dhcp3/dhcient.conf the first time around:

1. Change it
2. Type sudo date and then enter your password. This is just to make sure that sudo isn't going to prompt you for passwords
3. Type sudo ifdown eth0 && sudo ifup eth0

The above process will work even if you're on a remote ssh session (e.g. Putty), because the network will go down and up before your terminal times out.

Firewall on the VM Quick Reference

Here's how to set up the firewall. Here's my /etc/iptables.rules:

*filter
:INPUT ACCEPT [273:55355]
:FORWARD ACCEPT [0:0]
:LOGNDROP - [0:0]
:OUTPUT ACCEPT [92376:20668252]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
# Accept SSH so we can manage the VM
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i lo -j ACCEPT
# Allow ping (Zenoss uses it to see if you're up).
-A INPUT -p icmp --icmp-type echo-request -j ACCEPT
# Allow SNMP.
-A INPUT -p udp -s 0/0 --sport 1024:65535 --dport 161:162 -j ACCEPT
# Silently block NetBIOS because we don't want to hear about Windows
-A INPUT -p udp --dport 137:139 -j DROP
-A INPUT -j LOGNDROP
# Drop and log the rest.
-A LOGNDROP -p tcp -m limit --limit 5/min -j LOG --log-prefix "Denied TCP: " --log-level 7
-A LOGNDROP -p udp -m limit --limit 5/min -j LOG --log-prefix "Denied UDP: " --log-level 7
-A LOGNDROP -p icmp -m limit --limit 5/min -j LOG --log-prefix "Denied ICMP: " --log-level 7
-A LOGNDROP -j DROP
COMMIT


More on this later.

ntp on the VM

Bringing up the firewall on the "template" VM, I noticed that I was getting more ntp traffic than I expected. I discovered that in my ignorance, I had set my local ntp server to broadcast, which I don't need. I commented the broadcast line, and everything's still working.

I also found a good post on ntp that answered one of my long-time questions: What should I look at to see if the ntp client was actually working. Do ntpq -p. On the resulting listing, "the delay and offset values should be non-zero and the jitter value should be under 100." (The post is Red Hat based, but the information specifically about ntp is distro-agnostic.)