I had a real "Doh" moment this week. We're about 80 percent of the way through relocating a 500 server data centre. Things have been going pretty well, but right from the start we've found we were under-staffed on the network side. We have a pretty good process in place, with what I think is just the right amount of documentation. The individuals working on the network team are excellent. We even brought in one more person than we had planned for, but we're still struggling with burnout.
The light bulb went on for me a few days ago: Here, like other IT organizations of similar size and purpose, we have about ten times as many server admins as we do network admins. We have a bigger pool of people on the server side to draw from when we organize the relocations, which typically happen over night or on the weekend. While we rotate the network guys for the nights and weekends, it's a smaller pool. They do shifts more often, and they more often have to come in the next day, because they have to plan for the next relocation on their list and it's coming soon.
Another factor is that, in our case, the network team started intense work earlier than everyone else. We're occupying a brand new cage in a data centre, so the network team had to build the whole data centre network first. They did three weeks of intense work before we moved any servers at all.
As we hit six months of intense work for the network team, the strain is showing. We're going to try to rearrange some work and delay what we can. Other than that, we'll probably have to suck up the remaining 20 percent of the project somehow.
In the future, I'm not sure what I'd do. One approach, if I had enough budget, would be to hire a couple of contract network admins well in advance. Tell them they're going to work Wednesday to Sunday, and often at night. Train them up ahead of time so they're as effective as your in-house people. Then give most of the nasty shifts to the contractors.
What would you do?
(If you're looking for numbers, we have three full-time network engineers, and we're drawing on the operational pool from time to time.)
Friday 30 September 2011
Saturday 24 September 2011
The Data Centre Relocation Calendar
I'm past the half-way point in relocating a 500-server data centre. The servers are a real variety -- a typical medium-scale business data centre. We're using mostly internal resources, supplemented by some contractors like myself with data centre relocation experience.
I chose not to come in and impose a relocation methodology on everyone. There are a lot of reasons for that, some of which were out of my control. Rather than using a methodology out of the can, I tried to foster an environment where the team members would build the methodology that worked for them.
This turned out to be quite successful. One of the items that emerged fairly late for us, but was very successful, was a shared calendar in Microsoft Outlook/Exchange. (The tool wasn't important. It could have been done in Google Calendar. Use whatever your organization is already using.)
The shared calendar contained an event for every relocation of some unit of work -- typically some service or application visible to some part of the business or the public. Within the calendar event we put a high-level description of what was moving, the name of the coordinator for that unit of work, and hyperlinks to the detailed planning documents in our project shared folders. The calendar was readable by anyone in the corporation, but only my team members could modify it.
What struck me about the calendar was how it organically became a focal point for all sorts of meetings, including our weekly status meeting. Without having to make any pronouncements, people just began to put the calendar up on the big screen at the start of most of our meetings. We could see at a glance when we might be stretching our resources too thin. The chatter within the corporation that we weren't communicating enough diminished noticeably.
Based on my experience, I'd push for a calendar like this much earlier in the project. We built it late in our project because I had a lot of people on my team who were reluctant to talk about dates until they had all the information possible. We got so much value in having the calendar that I think it's worth it to make a calendar early in the planning stage, even if the dates are going to change.
I chose not to come in and impose a relocation methodology on everyone. There are a lot of reasons for that, some of which were out of my control. Rather than using a methodology out of the can, I tried to foster an environment where the team members would build the methodology that worked for them.
This turned out to be quite successful. One of the items that emerged fairly late for us, but was very successful, was a shared calendar in Microsoft Outlook/Exchange. (The tool wasn't important. It could have been done in Google Calendar. Use whatever your organization is already using.)
The shared calendar contained an event for every relocation of some unit of work -- typically some service or application visible to some part of the business or the public. Within the calendar event we put a high-level description of what was moving, the name of the coordinator for that unit of work, and hyperlinks to the detailed planning documents in our project shared folders. The calendar was readable by anyone in the corporation, but only my team members could modify it.
What struck me about the calendar was how it organically became a focal point for all sorts of meetings, including our weekly status meeting. Without having to make any pronouncements, people just began to put the calendar up on the big screen at the start of most of our meetings. We could see at a glance when we might be stretching our resources too thin. The chatter within the corporation that we weren't communicating enough diminished noticeably.
Based on my experience, I'd push for a calendar like this much earlier in the project. We built it late in our project because I had a lot of people on my team who were reluctant to talk about dates until they had all the information possible. We got so much value in having the calendar that I think it's worth it to make a calendar early in the planning stage, even if the dates are going to change.
Subscribe to:
Posts (Atom)