Wednesday 9 May 2007

Test Everything - Yeah, Right

I just read a short article that said the first thing to do to solve a lot of our IT infrastructure problems is to, "Create and maintain a test environment that mimics the production environment. Also commit to testing every change." (Emphasis from the original article.)

No problem. Since my 2,000 square foot data centre is already full of the 450+ physical and virtual servers I have in it, I'll just build another data centre and buy another 450 servers with all the associated software support and licensing costs. The lease costs alone on 2,000 square feet around here are about C$40,000 per month. I'm sure most IT managers have enough spare change in their budget to pull off that one -- not.

On top of that, you can never test everything. When I led the virtualization of 135 physical servers, the only problems we had after going live with any of the virtual machines were caused by changing IPs (which we had to do because of the enterprise network design). We wouldn't have caught those with a test data centre. There are lots of problems similar to that one lurking in a typical IT infrastructure.

You've got to build flexible infrastructures. Make sure nothing uses hard-coded IPs. Make sure your DNS doesn't have legacy pointers hiding in corners that you forget to update. Limit the partitioning of networks by physical location as much as possible in the first place. Build commodity server infrastructures.

(P.S. We did have other problems virtualizing the 135, but we found them before we turned the VMs over to production. The only problems that were seen by users were IP-change-related.)

No comments: