Sunday, 20 November 2011

Data Centre Relocation Gotchas

Here are a couple of gotchas we ran into while relocating close a medium-size data centre:

  • When restarting a server in its new location, it decided to do a chkdsk. Unfortunately, the volume was a 10 TB SAN LUN. Fortunately, we had a long weekend to move that particular server, so we could wait the almost two days it took for the chkdsk to run. (I don't know why the server decided to do chkdsk. Rumour has it we didn't shut down the server cleanly because a service wouldn't stop.) 
  • A website tells me to run "fsutil dirty query c:" to see if chkdsk is going to run on the C: drive the next time the system boots
  • On Linux, here are a couple of ways to make sure you won't have an fsck when you restart the server
  • We were frequently burned by the Windows "feature" to automatically add a server to DNS when the server starts up. Either we'd get DNS changes when we weren't ready for them, or we'd get the wrong changes put into DNS. For example, servers that have multiple IPs on one NIC, where only one of the IPs should have been in DNS
Here's a short checklist for turning off and moving a server:

  • Check to see if the server is going to check file system consistency on the next startup (chkdsk or fsck)
  • Shut the server down cleanly
  • If it's a physical server, shut it down and then restart it. Rumour has it that the hard drive can freeze up if the server hasn't been stopped in a long while. Better to find that out before you move it than after. This has never happened to me
  • Do a host or nslookup after starting the server to make sure your DNS entries are correct. Make sure the entry is correct and that you have the right number of entries (usually one)

No comments: