Network Team for Data Centre Relocations

I had a real "Doh" moment this week. We're about 80 percent of the way through relocating a 500 server data centre. Things have been going pretty well, but right from the start we've found we were under-staffed on the network side. We have a pretty good process in place, with what I think is just the right amount of documentation. The individuals working on the network team are excellent. We even brought in one more person than we had planned for, but we're still struggling with burnout.

The light bulb went on for me a few days ago: Here, like other IT organizations of similar size and purpose, we have about ten times as many server admins as we do network admins. We have a bigger pool of people on the server side to draw from when we organize the relocations, which typically happen over night or on the weekend. While we rotate the network guys for the nights and weekends, it's a smaller pool. They do shifts more often, and they more often have to come in the next day, because they have to plan for the next relocation on their list and it's coming soon.

Another factor is that, in our case, the network team started intense work earlier than everyone else. We're occupying a brand new cage in a data centre, so the network team had to build the whole data centre network first. They did three weeks of intense work before we moved any servers at all.

As we hit six months of intense work for the network team, the strain is showing. We're going to try to rearrange some work and delay what we can. Other than that, we'll probably have to suck up the remaining 20 percent of the project somehow.

In the future, I'm not sure what I'd do. One approach, if I had enough budget, would be to hire a couple of contract network admins well in advance. Tell them they're going to work Wednesday to Sunday, and often at night. Train them up ahead of time so they're as effective as your in-house people. Then give most of the nasty shifts to the contractors.

What would you do?

(If you're looking for numbers, we have three full-time network engineers, and we're drawing on the operational pool from time to time.)